Correlation and Causation
Misuse and Misconception of Statistical Facts
A quote on statistical lack of interest in Wired magazine on the part of an American Schoolteacher, that, perhaps innocently, misled me, serves as a warning how easily we may be duped with scientifically sounding facts and numbers. Given the amount of statistical facts and their wide spread exploitation by the mass media, it surely pays to be at least aware of the dangers. Below I collected excerpts from three books that (in part) are concerned with this phenomenon.
Following is an excerpt from
J. A. Paulos,
Children with bigger feet spell better. In areas of the South those counties with higher divorce rates generally have lower death rates. Nations that add fluoride to their water have a higher cancer rate than those that don't. Should we be stretching our children's feet? Are more hedonist articles in Penthouse and Cosmopolitan on the way? Is fluoridation a plot?
Although studies do exist which establish all of these findings, the above responses to them only make sense if one does not appreciate the difference between correlation and causation. (Interestingly, the philosopher David Hume maintained that in principle there is no difference between the two. Despite some superficial similarity, however, the issues he was getting at are quite different from the present ones.) There are various kinds and various measures of statistical correlation, but all of them indicate that two or more quantities are related in some way and to some degree, not necessarily that one causes the other. Often the changes in the two correlated quantities are both the result of a third factor.
The odd results above are easily explained in this way. Children with bigger feet spell better because they're older, their greater age bringing about bigger feet and, not quite so certainly, better spelling. Age is a factor in the next example as well since those couples who are older are less likely to divorce and more likely to die than are those from counties with younger demographic profiles. And those nations that add fluoride to their water are generally wealthier and more health-conscious, and thus a greater percentage of their citizens live long enough to develop cancer, which is, to a large extent, a disease of old age.
Following is an excerpt from
Marylin vos Savant,
The Power of Logical Thinking
My own favorite "bad example" of limited logic comes from an article about cats that appeared in The New York Times' weekly science supplement called "Science Times" on August 22, 1989. It stated, "The experts have also developed startling evidence of the cat's renowned ability to survive, this time in the particular setting of New York City, where cats are prone at this time of year to fall from open windows in tall buildings. Researchers call the phenomenon feline high-rise syndrome."
But the sidebar about how they managed to survive the falls was what interested me the most. It elaborated, ". . . From June 4 through November 4, 1984, for instance, 132 such victims were admitted to the Animal Medical Center.... Most of the cats landed on concrete. Most survived. Experts believe they were able to do so because of the laws of physics, superior balance, and what might be called the flying-squirrel tactic....
... [Veterinarians] recorded the distance of the fall for 129 of the 132 cats. The falls ranged from 2 to 32 stories.... 17 of the cats were put to sleep by their owners, in most cases not because of life-threatening injuries but because the owners said they could not afford medical treatment. Of the remaining 115, 8 died from shock and chest injuries.
"Even more surprising, the longer the fall, the greater the chance of survival. Only one of 22 cats that plunged from above 7 stories died, and there was only one fracture among the 13 that fell more than 9 stories. The cat that fell 32 stories on concrete, Sabrina, suffered [only] a mild lung puncture and a chipped tooth....
". . . Why did cats from higher floors fare better than those on lower ones? One explanation is that the speed of the fall does not increase beyond a certain point, [the veterinarians] said.... This point, terminal velocity, is reached relatively quickly in the case of cats. Terminal velocity for a cat is 60 miles per hour; for an adult human, 120 m.p.h. Until a cat reaches terminal velocity, the two speculated, the cat reacts to acceleration by reflexively extending its legs, making it more prone to injury. But after terminal velocity is reached, they said, the cat might relax and stretch its legs out like a flying squirrel, increasing air resistance and helping to distribute the impact more evenly." That seemed to make sense, so I filed the article for future reference.
Some time later, a reader wrote to ask me to "please explain why a cat will land on its feet when it falls from a great height," and I obliged by citing the study in my column, adding the following as an additional point of interest: "Amazingly, the cats that fell longer distances fared better than the others. Of the 22 cats that fell more than 7 stories, 21 survived; of the 13 cats that fell more than 9 stories, all survived. Sabrina, who fell 32 stories onto concrete, suffered only a minor lung puncture and a chipped tooth; I'll bet she was treated to a whole bowlful of tuna that day."
Later, reading these statistics in my published column bothered me, but I didn't know why. It never occurred to me to scrutinize the statements from the original article further. So it wasn't until my assistant dropped a handful of letters about the subject on my desk that I finally took notice. The first was from Pamela Marx in Brooklyn, New York, who wrote, "I have had two cats fall from terraces in two separate instances, and both, unfortunately, died. One was a tenth-floor terrace, and the other was on the fourteenth floor. I never reported these incidents to any medical center and believe that other people probably don't report their cats' deaths, either. You can add my two cats to your list and report that at least two cats died in fifteen falls over nine stories." At that point, the error seemed so obvious that I didn't know how I had missed it in the first place.
Following is an excerpt from
200% of Nothing
Testing the Waters
In early February 1990, thousands of cases of Perrier water were pulled from store shelves all across North America when it was reported that a quantity of benzene had accidentally gotten into a batch of the famous water in the French bottling plant. Benzene is a known carcinogen. Newspapers were somewhat vague about the risks involved, but the fact that some stores were removing Perrier from their shelves as a precaution became the main story, and what amounted to a public ban on the water ensued.
What would be the actual risk to health of drinking benzene-laced water? How do medical authorities calculate risk from a carcinogen? First, they formulate an idealized model that involves a person of a certain weight who inhales or ingests a certain amount of the carcinogen-carrying substance on a daily basis over a fixed period of time. From this risk model, to which a specific risk or probability of developing cancer is attached, authorities extrapolate by scaling in various ways. For example, a real person who has twice the weight of the idealized model person may be assigned only half the dose since there is twice as much tissue to take up the toxin. Scaling up or down from the model dosage will also alter the risk. Someone who ingests half the dose of the substance that the model does will have a reduced probability of developing cancer over the same period of time.
For example, the Environmental Protection Agency (EPA) provides cancer risk figures based on a 70-kilogram (154-pound) person consuming a carcinogen for 70 years. Putting its figures together, the EPA calculates that the risk of drinking a 12 parts per billion (ppb) concentration of benzene in drinking water at the rate of two liters a day amounts to a lifetime risk of 1 in 100,000 of developing cancer.
In other words, if you weighed about 154 pounds and drank two liters of water every day at this level of benzene concentration, the probability that you would contract cancer as a direct result sometime during the rest of your life would be about 0.00001. The Perrier that found its way to North American shelves had an estimated concentration of 15 ppb, a little more than the 12 ppb used in the model. This represents an increase of 25 percent. Scaling up the probability by the same amount produces 0.000013, certainly more than the actual lifetime risk of a 154-pound person drinking two liters of tainted water every day. This represents the risk in the long run. As economist John Maynard Keynes once remarked, 'in the long run we are all dead.' Abuse detectives are just the sort of people who would scoff at such a small probability. Just to be on the safe side, of course, they might work out the short-run probability, say, for a year. The resulting risk figure would still be many times greater than the risk from drinking tainted water for a few weeks.
To calculate the risk of drinking the water for a year, you have to work backwards (see Chapter 12). If the dedicated drinker faces a risk of .000013 over a 70-year period, he or she obviously faces a much reduced probability over a one-year period: about 0.0000002 as it turns out. The six zeros mean that someone who drinks 2 x 365 = 730 liters of the deadly water over a year runs about the same chance of developing cancer sooner or later in his or her life as a direct result of drinking the water as he or she has of winning a 6-49 lottery after buying just three tickets. Someone who drinks the mild benzene cocktail for just a few weeks faces an even smaller lifetime risk, infinitesimal in fact.
If you think that weighing cancer risks against lottery chances amounts to comparing apples and oranges, you'd be right. But there's nothing wrong with comparing a very small apple with a very large one, especially if it helps to put things into perspective. So why not compare the risk of ultimately contracting cancer after drinking benzenated water for a year with the overall risk of developing cancer in any event? According to my trusty 1992 Houghton-Mifflin Almanac, the death rate from all forms of cancer in the year 1990 was 202.1 per 100,000. This figure can be translated directly into a probability simply by dividing the 202.1 by 100,000. The risk of someone dying from cancer in the year 1990 was, therefore, about .002. Suppose, for the moment, that this annual risk does not change much from one year to the next. Over a 70-year period, however, the total risk escalates, just as it did in the breast cancer example (see Chapter 2), into something larger, in this case, about .13. If this represents the probability of someone developing cancer sometime during a 70-year period, how much more should he or she worry when the risk of drinking the bad water is added to the overall lifetime risk? The figures speak for themselves:
Worry drinking tainted water: 0.1200002
The media, of course, rarely delve into risk for fear of discovering something that is less than astonishing. Anyway, they have other ways of dramatizing things. Enter the simple word swap. Raising a risk by 100 percent is not at all the same thing as raising a risk to 100 percent.
(Note: the following has been written in, I think, 1999-2000. In January 2002, I was unpleasantly surprised to discover that the link to the cited document at the NEA site became broken - the document has disappeared. Much later, in March 2007, Hollis Hudetz found a copy of the document in the archive.org. Below this is the document I link to.)
I have discovered an example of abuse of statistics on Internet. To my chagrin, the document I refer to is promoted by the National Education Association that has otherwise an exemplary site. The document ARE TEACHER UNIONS HURTING AMERICAN EDUCATION? combines results of several published investigations to assert that Teachers Unions are good for education. I can't question the factual results that apparently SAT scores in states with unionized teachers are higher than in other states. Be that as it may, the document gives no clue as to how and in what way teachers unions engendered an improvement of SAT scores. I would presume an intelligence level of an average (of course) teacher that would preclude any reliance on statements like:
For the NAEP reading scores, low income, large class size, high absence rates, minority enrollment, and the level of private school attendance are correlated
with low performance levels at the state level.
I'd be happy to apologize, but to me the statement says that higher private schools attendance somehow is a cause (perhaps, one of many) of the low NAEP reading scores. As a parent, I would definitely do my best to enroll my kid in a private school wherever he can't get decent public schooling. It shows that even where there exists correlation between random variables it would be imprudent to claim any causal effect. I actually impute an interpretation inverse to that implicit in the document.
You may be interested to know that there is an excellent explanation of the pitfalls of statistical correlation (oddly enough)in the Help function of a statistical software package that we recently purchased at work, called Statistica. The explanation of apparently absurd correlations is quite simple: If one takes a certain error level, or confidence level as the working level for accepting the possiblity of error, and if one does enough statistical correlation studies, then JUST BY CHANCE, you will produce statistical correlations that are in the error range: i.e. have no real-world meaning. The explanation in the Help function puts mine to shame, but it is a more mathematical explanation than any other I have seen, and for the first time, I was completely comfortable abiding by the principle that statistical correlation PROVES nothing. Experimental correlation MUST be used to show cause and effect.
Thank you for your time. I hope you get the chance to take a look at this package- it's HELP function is better than some texts on probablity and statistics.
Copyright © 1996-2017 Alexander Bogomolny