Spotting Junk Science

No one can argue that we are living in the information age. But with all of this information we are buried in daily, how can we tell what is worth a look and what is, well, junk? Often we are faced with contradictory sources of information on one subject, each claiming science as its basis. Just because information is presented in a news source by someone with a "Dr." in front of his or her name, doesn't mean it is accurate. Bad science exists, and it can be spotted.

You don't have to be a statistician to sort out the good science from the junk science. All it takes is a little knowledge of some basic statistical concepts. This will make you a more selective consumer of information.

Correlation is not causation

This is the very first thing I learned in my very first statistics course, and it is the number one way to determine if something is junk science.

What does this concept mean? Simply because two events happen to coincide, it doesn't mean that one caused the other. Here's an example:

You read an article that says a study was conducted on shark attacks and ice cream sales on a particular stretch of beach. The more ice cream that is sold, the more shark bites occur. The author has determined that ice cream attracts sharks.

This is the classic example of correlation vs. causation. These two happenings are strongly correlated, but the author has attributed one as the cause of another without any real proof. If we stop to think about this, we'll find that both events are actually due to the fact that both shark attacks and ice cream sales increase during the summer, when there are more beach visitors. There is a separate, outside source of both increases.

Consider this the next time you read an article which claims a relationship where there is only a correlation. It is probably junk science.

Sample size

We've all heard the phrase "there are three types of lies: lies, damn lies, and statistics." Sample size is one of the main ways in which statistics can lie.

When reading a study, pay close attention to how many people participated. Appropriate sample size can vary depending on the type of study and the effectiveness of the experimental treatment, but pay close attention to samples that seem unusually small or unusually large.

For example, a sample size of 5 should raise a huge red flag. If someone is claiming to see significant results from a sample this small, there is more than likely a flaw somewhere in the research.

If a sample size is abnormally large, then the author may be trying to artificially magnify the results. A sample of 5,000 will show even the tiniest of differences between individuals to be significant. Purveyors of junk science will assume that the general public isn't aware of these statistical tricks.

One study wonders

You see these pop up in news stories all the time. One single study will report something outrageous and a news agency will blow it out of proportion. Often news agencies do not check for junk science at all.

If you see a study that makes an outrageous claim, see if it is the only one. Good research can be replicated time and again, with similar results.

Not every one-off study is junk science, but this should be a red flag.

Assumptions

Much of the junk science out there tries to play on what the audience assumes to be true. Often these articles will back up "common sense" or play on familiar stereotypes. Just because a study fits nicely into your worldview doesn't mean it's good science.

Take a look at what assumptions the author is feeding off of. Does the researcher have other studies to back up what he or she is saying? If not, it may be junk science. More often than not common sense can play tricks on us, and what we "just know" to be true isn't.

Good science won't fall into this trap, and neither should you.

Hopefully these tips will help you become a more discerning reader of science, or at least help you make sense of the daily information overload. Be a critical reader. Junk science is out there, but if you learn to spot it, you won't fall prey.