Top Secrets of the Scientific Method

A hot debate in the healthcare and research worlds has followed Claudia Wallis' accusations of irresponsibility by Michael Waldman and his colleagues. The economics professors pointed out a statistical correlation between "…reported autism cases, cable TV subscription data and weather reports. Yes, weather reports…" as Wallis sneers in her Time.com article, "A Bizarre Study Suggests that Watching TV causes Autism."

Obviously, she assumes that the reader will see immediately the ridiculous nature of this claim. In fact, you're probably thinking, "Sure, that's seems ridiculous to me." It's true, as Wallis excitedly points out, that there is currently no known scientific mechanism for TV watching to cause autism. That's where research comes in - to determine whether or not such a claim is, in fact, plausible.

"Wait," you say, "didn't he already do the research?" We all learned the Scientific Method in grade school. Repeat it with me: "Observation, Hypothesis, Prediction, Experimentation." So in order to suggest that autism and television watching are connected, Waldman followed this simple formula and reached his conclusions, right?

Not so fast: We're getting WAY ahead of ourselves. The fact is statistics are the end of the process in fields such as biology or engineering. But we are looking at a social factor - time spent watching television as a child. This research has now entered the world of social science. Waldman is, incidentally, incorrect when he states, as quoted by Wallis, that correlations are typically considered to be cause-and-effect relationships. Technically, there is more work to be done before you can claim that two statistics that happen to go together are related. In social science, statistics are not the answer to the question, but rather the catalyst for researching the question. Now, as professors of economics, Michael Waldman and his colleagues probably recognize their study as social science research, but I can't speak for them. However, Waldman has said himself (WebMD's Medscape.com) that this study is only a preliminary result, but rather a challenge to the health professions to take a look at the possibility. It's true, as Wallis excitedly points out, that there is currently no known scientific mechanism for TV watching to cause autism.

This is the real world, baby. The Scientific Method from grade school is only half the story. In reality, the whole process happens over and over again. Gregg Easterbrook, writing for Slate.com, defends the actions of Waldman and his colleagues as making crucial first steps in the process of scientific investigation. Here's a real-world look into how it happens:

1)Develop a question. Michael Waldman said to himself one day, hey, I wonder what other factors could be influencing the growing rate of autism besides genes and toxins. Perhaps, as in ADHD, television watching may have an effect. This step is often intuitive, based on the researcher's previous knowledge and experience of the subject.

2)Look at the literature. This step includes such activities as reviewing previous experiments and their results, as well as finding statistical data related to your question. In social science this phase is often based on previously collected data - hospital records, DMV records, and other public information - such as, yes Ms. Wallis, weather reports.

3)Form a hypothesis. The hypothesis presents an explanation for the statistical correlation found in step 2, with the purpose of showing it to be EITHER right or wrong.

The hypothesis for Waldman et al. may be something like: "Spending 60%-80% of waking time as 1-3 year olds watching television is one among many risk factors for developing autism."

NOTE: It is equally desirable to show a hypothesis as right OR wrong, because it adds to the available knowledge about a subject either way.

4)Get funding. This step is a project in itself. Questions as yet unanswered in the current literature are pointed out, and the researcher shows how testing her hypothesis will, of course, make the world a better place. Funders are most often persuaded by statistics. But there is no sense in proposing an experiment or scientific study that has already been statistically verified. The point of the statistics is to show that there is a relationship between, say, television watching and autism, and that it is worth someone's while (and money) to find out, what exactly is that relationship?

5)Perform the experiment or other type of study. In medical and research involving possible social causes or explanations, this is often done by questionnaires and interviews about, for instance, family and child television watching habits. This is called a cross-sectional study (see the glossary section of the Inter-University Consortium for Political and Social Research at http://webapp.icpsr.umich.edu). A more time-consuming and expensive approach would be to perform a longitudinal study in which the researcher monitors the television-watching habits of a large group of children over time and documenting which of them eventually develops autism. You can see why cross-sectional studies are done more often.

6)Replicate the experiment. The question has been answered for one group of people, at one time in history, perhaps in a certain city or region. Now it's time to do the same study again in a different region, nationwide, or even worldwide, to determine whether the results are most always true, or only true under certain conditions.

Wallis in her article on Time.com makes the mistake of assuming that Waldman and his buds were working on step 5 of the scientific method, when in reality they were most likely on step 2. Making public preliminary research and findings is one of the natural processes of peer review in science, and should never be taken for making hard-and-fast conclusions. Understanding the scientific method as it is practiced in scientific circles today will help Wallis and others observe and correctly identify the purpose and scope, and therefore the value, of scientific research.