Another Misuse of EBM

We agree with Roy Poses' comment on the recent JAMA article about EBM, particularly regarding some of its unfortunate "misuses." In the article that stands as the defining credo of Health Care Renewal, Roy wrote:

Most physicians believe that medicine should be based on science. Evidence based medicine, (EBM) ‘‘the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients’’ [74], is a refinement of this view. Meanwhile, the scientific basis of medicine is under increasingly severe attack.

We'd like to take this opportunity to identify a misuse of EBM that, ironically, contributes to the attack on the scientific basis of medicine, and that authors Montori and Guyatt didn't consider--probably because they are unaware of it. We explain it in the following letter, submitted to JAMA in response to their article. We discuss it at more length on the other blog to which we contribute, Science-Based Medicine, referenced from the letter:

To the editor:

In their recent article, “Progress in Evidence-Based Medicine,” Drs. Montori and Guyatt include important caveats about misuses of EBM. They mention trials funded by industry. They list several, documented problems with such trials, and correctly warn that these can result in erroneous interpretations by “unsophisticated users of the medical literature.” They correctly worry about “abandoning appropriate skepticism.”

There is a related misuse of EBM that leads to erroneous interpretations even by sophisticated authors, and that calls attention to systematic problems in EBM: trials of highly implausible hypotheses. Thus, regarding the homeopathic preparation Oscillococcinum, the Cochrane Collaboration writes:

Though promising, the data were not strong enough to make a general recommendation to use Oscillococcinum for first-line treatment of influenza and influenza-like syndromes. Further research is warranted…[1]

The data, however, are not promising, and further research is not warranted. The reason is that for highly implausible treatments, existing knowledge easily trumps modestly “positive” trial data. In the case of homeopathy, each of its tenets has been more than adequately disproved, and its claim that the potency of a “remedy” increases with each dilution of the original substance, continuing well past the point at which not a molecule of that substance remains in the preparation, contradicts physical reality.[2]

Such considerations reveal at least two, related problems with the evidence of EBM as it pertains to clinical trials. The first is explicit: by relegating “physiology, bench research or ‘first principles’” to Level 5 of its “Levels of Evidence” hierarchy, EBM fails to acknowledge that consistency with firmly established knowledge is a necessary, even if insufficient, basis for accepting a hypothetical treatment as effective.[2]

The second problem is part of the culture, if not the letter of EBM: by eschewing Bayesian in favor of “frequentist” statistics, its practitioners can excuse themselves from justifying estimates of prior probability, and erroneously conclude that trial data tell us more about reality than they do.[3-5]

In January, 2008, several colleagues began the blog Science-Based Medicine, referenced below.* The name, which could just as well have been “knowledge-based medicine,” is a frank recognition of the EBM weaknesses described above. Because of our interests, Science-Based Medicine’s content is predominantly, though not entirely, concerned with implausible health claims. Its larger message, however, is that these claims call attention to fundamental ways in which EBM can be improved, such as to justify its own claim to being an arbiter of real evidence.

Kimball C. Atwood, IV

Wallace I. Sampson

10/19/2008

* Steven Novella, David Gorski, Harriet Hall, Wallace Sampson, Kimball Atwood

[1] Vickers A, Smith C. Homoeopathic Oscillococcinum for preventing and treating influenza and influenza-like syndromes. Cochrane Database of Systematic Reviews 2006, Issue 3. Art. No.: CD001957. DOI: 10.1002/14651858.CD001957.pub3.

[2] Atwood K. Homeopathy and Evidence-Based Medicine: Back to the Future Part V. Science-Based Medicine 2008 (Feb. 8). http://www.sciencebasedmedicine.org/?p=42. Accessed October 19, 2008

[3] Atwood K. Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine.” Science-Based Medicine 2008 (Feb. 15). http://www.sciencebasedmedicine.org/?p=48. Accessed October 19, 2008

[4] Goodman SN. Toward evidence-based medical statistics. 1: The P value fallacy. Ann Intern Med. 1999;130:995-1004.

[5] Goodman SN. Toward evidence-based medical statistics. 2: The Bayes factor. Ann Intern Med. 1999;130:1005-13.