Thursday, December 2, 2010

Statistics Make Fools Of Us All

The phrase "lies, damn lies, and statistics" succinctly expresses the power that numbers have to obfuscate the truth.  It always amazes me how easy it is for statistics to be misinterpreted by even the best of us.  One of my favorite personal finance blogs is Free Money Finance.  While I do not know the author personally, he always strikes me as a person who is logical, level-headed, and generally well versed in subjects related to personal finance.  That is why I am dismayed to report that over the past week, he has not only misinterpreted statistics once, but twice!

Let me state for the record that it is not my intent to rake the author over the coals, so to speak.  I am willing to give him the benefit of the doubt and assume that his mistakes were not done purposefully or to further some hidden agenda.  The point of bringing this to light is to show how even somebody who seems to be an authority can misinterpret statistics and to illustrate that you cannot take statistics at face value, no matter how authoritative the source appears to be.

The first statistical lapse came on Sunday when he posted the article Given Grows Your Net Worth.  In his article he cites several "studies" which show that people who give to charity end up richer, not poorer:

We've talked about this before, but a series of studies by BYU professor Arthur C. Brooks found the following:

“My conclusion was, sure enough, that when people get richer, they tend to give more money away. But I also came up with the following counterintuitive finding—that when people give more money away, they tend to prosper.”

“Specifically, here’s what I found: Say you have two identical families—same religion, same race, same number of kids, same town, same level of education—everything’s the same, except that one family gives $100 more to charity than the second family. Then the giving family will earn on average $375 more in income than the non-giving family—and that’s statistically attributable to the gift.”

So, if you give $100 you're not worse off by $100. You're actually better off by $375.  Think of the implications of this at growing levels of giving. For instance:
•If you give $100, you're better off by $375.
•If you give $1,000, you're better off by $3,750.
•If you give $10,000, you're better off by $37,500.
•If you give $100,000, you're better off by $375,000.
•If you give $1,000,000, you're better off by $3,750,000.

Certainly makes a compelling case for giving as much as you can, huh?

He is citing a study which shows that families who give more earn more on average.  However, he immediately jumps to the conclusion that the giving more leads to a higher income.  There is no way that you can conclude that based upon the statistics.  There are, in fact, three possible conclusions:

1. More giving leads to a higher income.
2. Higher income leads to more giving.
3. Giving and income are both the result of some third factor and are not directly related to one another.

Immediately the author jumps to the conclusion that #1 is true, and he engages in some crazy extrapolation which makes his case look even more ridiculous.  At first when I read this, I thought that he was engaging in sarcasm, but there is no indication in his article that sarcasm is intended.  He even concludes by saying that the research "seems pretty solid".

This is a common mistake that most people make.  Somebody shows that two statistics are correlated and rather than trying to understand whether they are correlated because of 1, 2, or 3, they pick the choice which best agrees with their point of view and ignore the other possibilities.

A few years ago, there were a number of "studies" showing that children who had listened to classic music when they were young did better in school than those that didn't.  Immediately, people jumped to the conclusion that classic music was the key to raising a genius, and a new cottage industry was formed to sell overpriced CD's to crazed parents.  Nobody stopped to consider that maybe parents who would expose their children to classic music were more educated themselves and thus more likely to have smarter than average kids (option 3 above).

The second misleading statistic that was quoted by the author of Free Money Finance was the following from his article Interesting Facts on Cars and Wealth.  Note that the author of Free Money Finance is quoting another blogger, Dr. Thomas Stanley.  The words in quotes are Dr. Stanley's.

And here's a thought from his follow-up post titled Frugal Millionaire with a Mercedes? II:

"Too many Americans may believe that by driving a new car they are emulating economically successful people. But only 8.6% of those driving this year's model motor vehicle are millionaires [those with $1M in investments]. Don't ever feel degraded if you are riding around in a used motor vehicle."

The implication here is that because only 8.6% of people driving new cars are millionaires, millionaires are not likely to be driving new cars.  Again, this lone statistic doesn't prove or disprove this proposition.  Dr. Stanley, who is being quoted, is jumping to a conclusion that isn't supported by this one fact.

The fact is that millionaires make up a small fraction of the total population, so it makes total sense that they would represent a small fraction of new car buyers.  Just because they only represent 8.6% of new car buyers doesn't mean that only 8.6% of millionaires drive new cars.  However, Dr. Stanley concludes exactly that.  The statistic which would prove his case would be the percentage of millionaires who drive new cars versus used cars.  It is possible that even though 8.6% of new cars are driven by millionaires, 75% of all millionaires might drive new cars.  We just don't know.

The lesson here is that numbers carry so much authority that people are just blinded by them.  If somebody makes a claim and happen to throw in a number, people automatically accept the claim as fact regardless of whether the number is actually relevant to the claim!  Even though Free Money Finance's author is just quoting the statistic, he isn't totally off the hook.  Rather than just accepting the claim as fact because an irrelevant number was presented along with the claim, he should have used his initiative and questioned the number and the claim.

Like I said at the outset, it wasn't my intent to through Free Money Finance under the bus.  It is to show how even the best of us can be made to look foolish by misleading statistics.  The bottom line is that when presented with statistics, you have to think about what is being said and whether it makes logical sense.

No comments:

Post a Comment