The Truth Wears Off, or Some Science Sucks

Jonah Lehrer has a very interesting article in the latest New Yorker about a scientific effect that has yet to be named, causing apparently robust results to wane over time. Most of the famous cases concern drugs -- when they first came out, anti-depressants seemed to have very pronounced effects on people who took them, but more recent studies find much smaller effects. Lehrer put some excerpts from the article on his blog, like this one on the new anti-psychotics:

But the data presented at the Brussels meeting made it clear that something strange was happening: the therapeutic power of the drugs appeared to be steadily waning. A recent study showed an effect that was less than half of that documented in the first trials, in the early nineteen-nineties. Many researchers began to argue that the expensive pharmaceuticals weren’t any better than first-generation antipsychotics, which have been in use since the fifties. “In fact, sometimes they now look even worse,” John Davis, a professor of psychiatry at the University of Illinois at Chicago, told me.

Before the effectiveness of a drug can be confirmed, it must be tested and tested again. Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.

For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe? Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.

Some of the causes of this problem are well known. It is easy to imagine how drug trials end up weighted, in many conscious and unconscious ways, toward strongly positive results. But the effect has hit results in fields where the pressure to get the "right" result is much less prominent. Lehrer interviewed scientists who seem very conscientious and are mystified by the way effects they identified have declined over time, declines in some cases measured by their own experiments.

I am not surprised by any of this, because I am deeply skeptical of much current science. These statistical studies of subtle effects are just not very good science: they are too subject to manipulation, too dependent on the attitudes of the experimenters, too complex for anyone to understand the whole experimental design. The effects Lehrer describes prove this to me. After all, the effect of gravity is not declining, and matter is not getting less atomic. Real science doesn't go out of date. So if the effectiveness of medical treatments seems to decline, it is not necessary to speculate some squishiness in the world that renders the scientific method invalid. We simply need to realize that science, like everything else, comes in good and bad forms.