Lying with Statistics and Misleading Data
Misrepresenting statistics is a common way for companies to sell products and programs. These are some of the most common deceptions you should watch out for.
Misleading Charts, Creative Graphing and Pictographs
An ad run by Chevy said, “More than 98% of all Chevy trucks sold in the last 10 years are still on the road.” It was accompanied by a bold graphic that we’ve recreated below.
At first glance, it looks like Chevy is dramatically better than the competition. But if you look closer, you’ll realize they’re only showing the numbers between 95% and 100%. In the advertisement, the numbers were shaded to blend into the background.
The real difference between Chevy and Ford is less than 1%. Toyota is only a 2% difference. But that’s not how the ad makes things appear. By showing only the top part of the chart, they’re deliberately trying to mislead the reader. Charts and graphs need to show the complete data set, not just the cherry-picked sections.
The type of graph should also be appropriate for the data you’re trying to visualize. You wouldn’t use a pie chart to demonstrate differences between groups; it should compare parts of a whole.
Be especially careful when pictographs (pictures in graphs) are used. When graphic designers increase the size of an image, they often keep the scale the same as they increase the size. In chart 1 below it’s supposed to show that the second column is twice as much as the first. However, the image is four times larger in area, making the second column SEEM significantly more. If you must use pictures, chart 2 is a better example with two bags stacked on top of each other.
Statistical Cherry Picking and Missing Figures
Using only a piece of the data can make something look much different than it really is. It should never be used in scientific experiments, because it does not present all the results of an experiment.
Unfortunately, people who market information don’t care about credibility. Most only care about sales over the next week, month or quarter. If twelve tests were run and a product was only successful in one of the tests, unscrupulous companies ignore the 11 failures and trumpet the single success.
Ask for all the data and determine how many times the experiment was run. If tests were conducted 12 times, but you only see the results of 1, you’re being lied to. If they say it’s proprietary, they’re hiding something and lying to you.
Who you include in a survey matters. For example, Jeff Bezos, the founder of Amazon.com, graduated from Princeton in 1986. Let’s say there were 1,000 people in his graduating class.
You could say that the average earning in 2020 for people who graduated from Princeton in 1986 is $24 million a year. Of course, that’s ridiculous; those 1,000 graduates probably didn’t earn anything close to that.
It’s estimated that Jeff Bezos saw his fortune grow by $24 BILLION in 2020. Assuming everyone else in his graduating class made NOTHING, and by just dividing that $24 billion by 1,000, you end up with that average of $24 million per person.
Who you survey matters. Including Mr. Bezos in with the total, skews the results. A proper statistical analysis would consider who was polled and ensure outliers are noted and potentially removed from the averages.
What if I told you that 95% of people surveyed said they would be interested in using their cell phones to manage their retirement? How credible would that number be if I told you I only asked people who had a cell phone? What if I only asked people already using a banking app on their cell phone?
You would immediately realize that’s not representative of the world at large. Only calling people on landlines would skew the average age older, while only surveying people in a playground would skew the age younger. Who’s being surveyed and how the survey is conducted matters.
How many people you poll can be manipulated to achieve your desired results. Let’s say you’re testing a headache remedy. If you use very small samples, normal variations in how people respond can make the end result look dramatic, either for or against the medication you’re testing. Keep running the test with small groups until you get the results you want.
When looking at studies, bigger is usually better. Data from 100 people is good, 1,000 people is better, but 10,000 people can give you real insight and start to reveal significant trends.
Vague Terms, Misleading Definitions and Ambiguous Meanings
Let’s say I claim there hasn’t been a “terrorist attack” in the United States since 9/11/2001. Are mass shootings terrorist attacks? Does it only count as a terrorist attack if the perpetrator wasn’t an American citizen? Is mass murder a terrorist attack if the attacker is mentally ill?
The definitions people use when polling, analyzing and reporting information matter. Sometimes you have to agree to common terms before objectively gathering, tabulating, and analyzing the data.
Unlimited Extrapolation or Plotting Results Ad Infinitum
Extrapolation is a statistical method that tries to predict future trends based on historical data. However, you can’t assume things will continue on the same way forever. A chart showing sales taking off and then continuing to double far into the future is a fantasy.
Knowing how things trend up is important, but you have to recognize when they might level off or even decline. Statistics should always consider where peaks will hit and when declines might start.
With the rise of social media, it’s easy for people to make numbers up. Here’s an example from Psychology Today in the 2011 article “How to Lie with Statistics” by Peter Corning.
The social critic Ann Coulter, who is often loose with the facts, was caught in this famous example. In one of her books she claimed that President Ronald Reagan, despite various scandals during 1987, only saw a five-point drop in his approval rating, from 80% to 75%. Actually, it was a 16 point decline, from 63% to 47%, more than a trivial difference.
Always ask for the source of the data. Honest people are happy to provide details. Liars can’t.
These are a few of the more common ways statistics and data can be twisted to lie. Below you’ll find videos and articles to help you understand what else to look for and what to avoid.
How statistics can be misleading - Mark Liddell
How to spot a misleading graph - Lea Gaslowitz
How We’re Fooled By Statistics - Regression to the Mean
WeBeFit Articles on how to spot lies, misleading data and manipulated statistics in the health and fitness industry.
Call for a FREE Consultation (305) 296-3434
CAUTION: Check with your doctor before
beginning any diet or exercise program.