Evidence and Alternative Medicine: Assessing Therapy Claims

If we are buying a car and the salesman tells us it's a good runner, we are cautious and perhaps a little distrustful. After all, they want to sell us the car and they may tell us things that aren't true, just to make the sale. As a consumer, we want some evidence. We test drive it, we check the service record, we scrutinise it all over, and check its general condition. We want to be sure that there is some independent evidence that what the salesman says is right.

When we are consuming alternative medicine, we are in a very similar situation. Someone is telling us that their therapy will do us some good and they are selling us sessions which are supposed to address some ailment or condition. We are quite right to expect some evidence that what they claim is in fact true.

Do anecdotes counts as evidence?
One of the most common forms of everyday evidence is the recommendation, the account from someone who has experienced the therapy and thinks that you should try it too. This is what we mean by an anecdotal account, a first hand impressionistic account. It worked for me, so why not try it?

Unfortunately, this is the most unreliable form of evidence. It is subject to the patient's feelings, impressions, beliefs, and misunderstandings, and it is inevitably biased. Whatever they say happened to them, it may not be correct. Having spent their money on the treatment, they will be predisposed to defend the purchase so their description will inevitably be biased. It needs some independent corroboration, some independent evidence based on more objective measures.

So if an alternative therapist is relying on anecdotal accounts and recommendations as evidence, then that evidence is not evidence at all.

Controlled, double-blind clinical trials
When clinical medicine investigates new treatments, the bias of anecdotal accounts is removed by organising the clinical trial so that it is controlled, and double-blind. That means that there is always some group of patients that doesn't get the treatment: the effects can then be compared between the treated group and the non-treated group. If the treatment works, there will be a measurable difference between the group that had the treatment, and the one that didn't.

But it is also double-blind. This means that even the patients and practitioners don't know which is the real and which is the fake treatment. That stops either party from skewing their results based on their expectations.

Peer review and journals
Clinical research is published in peer-review journals where other medical practitioners are invited to criticise and challenge the results, including how the study was organised, how it was measured, the size of the groups, and how trustworthy the results are. This provides a high standard of clinical evidence. Of course, there are mistakes and sometimes papers are published which make claims that turn out to be false, but these mistakes are likely to be found.

The publication of the controlled trials also means that the results are cumulative so that other practitioners can learn from the research.

Research in Alternative Medicine
Over the years, there has been a growth in journals publishing research into alternative medicine. Often these journals are owned by associations or societies which promote their therapies, and also represent the practitioners. They are not independent scientific research journals as they have an interest in promoting the techniques and practises rather than challenging them. Nevertheless they do publish case studies and accounts of trials.

When we are faced with such research, we have to adopt a sceptical stance, in just the same way as we approach all scientific research. We have to ask the same sort of questions. Was the trial controlled and double-blind? Was the sample size sufficiently large to draw scientific conclusions? Did the paper depend on anecdotal accounts? Were the researchers trying to prove a result they had already decided on, or were they researching a phenomenon with an open mind?

In the UK at the moment, there is a libel case proceeding which concerns a claim by chiropractors to be able to treat childhood colic. So the question inevitably arises as to whether or not that claim is justified by the available evidence. The evidence provided by the chiropractors was assessed by the British Medical Journal and was found to be inadequate to support the claims. Dr Edzard Ernst evaluated each of the papers offered as evidence and gave a detailed assessment of why it did not constitute adequate evidence. Independent, high quality, reproducible evidence is critical if the claims of alternative medicine practitioners are to be taken seriously.

Traditional Therapies
Some alternative therapies, such as Acupuncture, Ayurvedic Medicine, and Reiki, claim that because they have a long history of belief behind them, that this constitutes evidence that they work. Unfortunately, it only means people believed that they worked. When they have been tested, they have been shown not to work.

We could equally well claim that bleeding patients by opening their veins when they have a fever, is also a traditional technique practised for a long time in Europe and the US. Indeed, it led to the death of President George Washington who was bled to treat pneumonia. Just because people believe it, it doesn't mean that it works. Any treatment needs to be tested thoroughly before we can claim that it works. The bleeding technique was tested, and it didn't work. In fact, it killed people.

Some questions for alternative practitioners?
We need to ask practitioners about the evidence for their therapies because they are selling us goods just like the car salesman. They want to sell us a consultation or a course of treatment so we are entitled to ask for the evidence that it will do us good.

We have to dismiss the anecdotal accounts because these cannot provide the kind of unbiased objective evidence we need. We also need to check that any academic papers cited are controlled and double-blind. If they are not, they are poor quality and almost certainly misleading. But there are other signs that we should check for as well.

Does anything they say claim properties for forces, energies, or influences which cannot be detected and demonstrated? If such forces and energies cannot be detected and demonstrated, then clearly they cannot be harnessed for use by a practitioner. If changes in these forces and energies are claimed, we need to understand how they detect and measure such a change. And a simple claim that they feel it, would be no more convincing than the car salesman who says "trust me". If they can demonstrate some effect, however slight, we need to understand how this relates to changes in the human body.

It is very worthwhile keeping an open mind when people propose new and interesting theories of how the healing process works but the theory alone is not enough. We always have to have some evidence to show that the theory is correct, and appealing to undetectable entities means that these theories won't fly. Not only will there be no evidence, but there won't be any reason to believe what the practitioner says.

Having the idea is only the first part of the work. It has to be followed by serious investigation, study, experiment, measurement, and peer review, to provide evidence which stands up to scrutiny. Only then, are we justified in making the sort of claims made by alternative medicine practitioners.