Is it statistically significant?

Written by Gary Silverman, CFP®

A few weeks back I wrote an article regarding what the research says about whether or not we save more as we get older. Another study in the same area looked at whether people save more once the kids leave the house. Figuring these two were a little too similar I skipped writing an article on the kid angle. (Spoiler alert: Yes. People do save more when the kids leave the house.)

But you shouldn’t care.

I have both taken and taught statistics. While I’m not sure I was all that good at either, I did learn a thing or two. It’s come in handy as I read a lot of research in my field that provides some sort of statistical analysis. Part of the analysis is to determine if differences discovered in the study are “statistically significant.” I won’t bore you with the full definition of the term (mainly because I don’t remember it), but it measures the likelihood that the differences are due to a true difference or just luck.

In the case of the study involving kids leaving the home, the researchers measured that indeed “empty nesters” saved a bit more than people with kids still at home. The analysis of those measurements showed that it was likely a true measure of what was going on rather than the researchers accidentally picking the wrong folks to measure. Thus it was deemed statistically significant.

However, while the phenomena is significant and therefore a real event likely happening to families, it didn’t matter a whole heck of a lot. The differences measured (in three different ways) all showed less than a 1% change. In other words, yes, something was going on, but it didn’t make much of a difference.

There are a lot of studies that show statistical significance but when looked at as how it might actually affect your life there’s not much there. That’s why when a study comes out saying that X causes more Y you should ask a couple questions. The first, was it statistically significant? If yes, then ask, how much of a difference was there?

Imagine if I told you that the stock market tended to be weak one particular month but then goes up more than average the next one. You’d be interested. You might even design a strategy where you’d get out before the bad month and get back in as the next month began. But what if the effect was so small (though statistically significant) that your portfolio might eke out an extra $10 a year.  To add insult to injury, transaction fees and taxes end up costing you $25 to enact your strategy. This would be an example of something being significant in a rather worthless way.

So look deeper before you embark on some new investment strategy, diet, exercise, or other endeavour because you read that it caused good things to happen. Make sure it’s worth your time, effort, and money given the gain you’ll actually see.

This article was published in the Wichita Falls Times Record News on January 3, 2016.