Bad Stats

Statistics can be misleading. Marketers have become adept at using numbers to make their case, but we still need to be careful about how we interpret what they are telling us. In 2007, Colgate toothpaste was all over the media telling us, “80% of dentists recommend Colgate.” That was what the ads said. The number was correct.

But what they didn’t tell you is how they got that data. It sounds like what they are saying is that 80% of dentists recommend Colgate over other brands, and that is what they want you to believe. They got that data by asking dentists to list several brands of toothpaste that they would recommend. So what they are actually saying is that Colgate is one of many brands that lots of dentists (80%) would include on their recommended list. That means that there are other brands that may have been recommended just as frequently, and there is no indication that Colgate is better than any of the other recommended brands. That puts a different spin on it, huh? 

Graphs

Some statistics are presented to us in graphical form, and that can be a little misleading, too. Many of the pundits who say that climate change is a hoax point to this graph: 

It looks like the average annual temperature has been relatively consistent over the last decade. And except for that one year when the temperature dropped a little, that’s true. If you use a different range of dates, the data tells a completely different story. 

Now we can see that the general trend is upward, and dramatically upward. We have to be really careful to think hard about the data that is presented. Are they cherry picking, only showing us what they want us to see? Or is it a relatively complete story? 

Choosing only the best data can be found in other situations, too. When drug companies show the work done in clinical trials, how do we know that they aren’t showing us only the good clinical trials? It is important to have independent researchers do follow-up studies to determine that the results of a good trial can be replicated. Internal studies have too many ways that they can be influenced. 

Ethics

Of course, there are times when the real question is not a statistical issue at all, but rather an ethical one. The FTC won a settlement against Reebok shoes when they ran an ad that said, “Reebok EasyTone shoes not only look fantastic, they’ll help make your legs and butt look great, too. It’s the shoe proven to work your hamstrings and calves up to 11% harder and tones your butt up to 28% more than regular sneakers. Just by walking.” 11% harder? Tones 28% more? It turns out that they just got these numbers the old fashioned way — they made them up. And they ended up paying a fine of $25 million. 

Figures can’t lie, but liars can figure.

Politicians are constantly making up numbers, some more outrageous than others. In 2013, Michelle Bachmann (R-Minn) said, “70 cents of every dollar spent on food stamps goes to bureaucrats.” The real number is $0.0033 (less than a penny). 

During the current election season, it will be more important than ever for us to look carefully at the claims that are being made with numbers. Is crime up or down? By how much? Is the economy better or worse? What about inflation? All these things depend greatly on how you look at them, and a person’s individual circumstances. The economy can be doing great, but if I am out of a job, or my spouse is ill, things will not seem so wonderful. 

Politicians use statistics in the same way that a drunk uses lamp posts—for support rather than illumination. – Andrew Lang

The other common mistake we all tend to make is assuming that correlation equals causation. If a new president is elected and the economy starts to improve, we assume that it is as a result of new policies and great leadership. In reality, it is likely that the economy is doing well all on its own. A giraffe could have been elected and the economy would still be doing well. Think of a situation where there is a new invention (say, the automobile) that makes a huge impact on the economy. The president benefits, but really had nothing to do with the positive change. This is not to say that political leaders don’t make a difference — they do. But their direct impact is often overestimated. 

Weird Statistics

Finally, let me show you a strange occurrence which makes me skeptical of many statistical arguments. 

There are two hospitals, Alpha General Hospital and Beta General Hospital. Everyone says Alpha General is the best, because they show a survival rate of 90% in their patients. Beta General only has a survival rate of 80%. Where should we take grandma?

Here’s the data: 

The complication arrives when you split the incoming patients into two groups: those who arrived in poor health, and those who were in good health, and only needed minor procedures. Even though Alpha General has the higher survival rate overall, it is Beta General who has better survival rates in each of the two groups. So whether you are admitted in good health or poor, it would be better for you to go to Beta General. Weird. This is called Simpson’s Paradox, when subgroup percentages can show a different result than a combined group. 

There are lies, damned lies, and statistics. — Mark Twain

The moral of the story is that we need to be very diligent when we are evaluating statistics. Be warned.