How to Evaluate Medical Research

Medical research influences the decisions of healthcare providers and patients. Scientific research informs decisions as common as which toothbrush to use and as important as when to treat prostate cancer. How trustworthy is medical research? How can health care providers and consumers evaluate studies? Epidemiologist John Ioannidis of the Harvard School of Public Health in Boston published an influential essay on this subject in Plos Medicine in 2005.

Ioannidis states that in "modern research, false findings may be the majority or even the vast majority of published research claims." His essay systematically explains why. He offers guidelines to aid in determining the value and validity of research. Lay people and healthcare providers should consider these rules of thumb when relying on scientific research to guide decisions. Here are suggestions on how to apply Dr Ioannidis' recommendations to every day decision making.

More reliable studies include thousands of subjects. A study of 20 subjects is very unreliable. It may not be possible to perform a large study on a particular subject. That does not mean that the small study is more trustworthy.

Try this experiment. Ask your best friend what their favorite baseball team is. Chances are you and your best friend like the same team. The result of your survey is that 100% of the subjects in this study choose one baseball team. Now, imagine asking all of the people in the United States the same question. This new study will include subjects who prefer every team in the country. This second study provides a much more accurate picture of baseball team preference.

The exact same relationship occurs in scientific research. A small study of a few people is unlikely to have the same result as a large study; the large study will be more accurate.

Studies that illustrate a very small difference between outcomes are more unreliable. For example, if eating a particular breakfast cereal is likely to lower blood pressure that is great. However, research may demonstrate a very small difference in blood pressure for people who eat that cereal compared to people who do not. The manufacturer may promote their product with a simple tag line, "Lowers Blood Pressure". They might include a graph that shows that the difference is less than one percent, but the range of the graph can make the difference appear impressive (Figure 1). When making decisions based on research, choose reports that demonstrate differences of five percent or more. Large differences are more reliable.

When results are open to interpretation, or when rules of a study allow researchers too much flexibility when choosing subjects, the results are more questionable. How do the researchers measure their results? Is there a machine or a computer that measures the results? Do the researchers interpret readings by looking at a measuring device? If researchers receive training to obtain measurements the same way, the study will report that the researchers are "calibrated".

Always consider who benefits from the findings of the study? Does the researcher have a financial or a professional interest in the results? Is the researcher paid by a manufacturer in some way? When choosing products, medications and even treatments, it is not only important to understand how the study is done. It is important to understand who performed the study, who pays the salaries of the authors, and who pays for the study.

When the manufacturer of an anti-plaque mouthwash pays for a study, they hope for the study to demonstrate that their product is effective. They will publish a study that demonstrates the effectiveness of their product. Studies that show that their product is ineffective will not be published. Researchers understand the politics of their work. Flexible study rules, measurements that are open to interpretation and creative use of statistics allow researchers to deliver studies that benefit their sponsors.

Before accepting statements such as "recent studies suggest that coffee (regular and decaffeinated) may offer a variety of health benefits against diseases such as cancer and diabetes." A study paid for by CoSIC is not likely to be reliable. The Coffee Science Initiative, a trade organization is "funded by the Institute for Scientific Information on Coffee, a group of senior executives from major European Union food manufacturers. (Kraft, Nestle, Sara Lee, Paulig, Tchibo, Illycaffe & Lavazza)" (www.CoSIC.org). Alternatively, the Agency for Healthcare Research and Quality's (AHRQ) is a government agency whose "mission is to improve the quality, safety, efficiency, and effectiveness of health care for all Americans." (www.AHRQ.gov) Funding from the AHRQ is less influenced by industry concerns.

According to Ioannidis, "The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true." Just think of how many times research demonstrates that a food or drink is unhealthy and later articles tout benefits of the same. The article "Cup of Confusion, Is Coffee Healthy or Not?" on MSNBC.MSN.Com (http://www.msnbc.msn.com/id/15823127/ns/health-diet_and_nutrition/) illustrates this issue.

Ioannides reasons that researchers are competitive. They will try to best each other by proving earlier research wrong. It takes many years to learn the true answer to any scientific question.

A final test of the value of research is the publication itself. Many scientific journals submit studies to a panel of experts for evaluation. This is called "peer review". Peer reviewed articles are more trustworthy than articles that are not subject to peer review. Unfortunately, some peer reviewed publications may be subject to influence from advertisers. Consider articles on toothpaste in peer reviewed dental journals that contain advertisements from toothpaste manufacturers.

The best information comes from large studies or studies that compare the results of many smaller studies (meta-analysis). Understand who the researchers are and who pays for their studies. Only trust research published in peer reviewed scientific or professional journals. Never trust reports of research on exciting new technology, medications or techniques. It will take more research and a long time to find the truth. Finally, ask knowledgeable professionals such as your dentist or physician to explain the importance of published studies.

Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124