Understanding Scientific Findings

William J. Sutherland, David Spiegelhalter, and Mark Burgman have an editorial in Nature offering Twenty Tips for Interpreting Scientific Claims. They are:

• Differences and chance cause variation.
• No measurement is exact.
• Bias is rife.
• Bigger is usually better for sample size.
• Correlation does not imply causation.
• Regression to the mean can mislead.
• Extrapolating beyond the data is risky.
• Beware the base-rate fallacy.
• Controls are important.
• Randomization avoids bias.
• Seek replication, not pseudoreplication.
• Scientists are human.
• Significance is significant.
• Separate no effect from non-significance.
• Study relevance limits generalizations.
• Feelings influence risk perception
• Dependencies change the risks.
• Data can be dredged or cherry picked.
• Extreme measurements may mislead.

The authors offer a little paragraph on what they mean by each of these, should you be curious.

Most of these cautions relate to understanding statistics, which I suppose underscores the importance of statistics in science these days. I am not sure what "feelings influence risk perception" is doing on the list, since this is more a problem of non-scientific thinking than a problem judging scientific findings. I could quibble with a lot of the others, too. But the bigger problem is not that people don't understand these things, it is that scientists themselves work to undermine understanding of these principles. It sometimes happens that scientists will publish an article that clearly says their findings are preliminary and uncertain, only to have the press blow it out of proportion. But more often it is the scientists themselves who blow their own work out of proportion. It is scientists, not reporters or the public, who draw regression lines through messy data sets and act like their lines mean something. And it is scientists who structure their own experiments around picking up small and quite likely irrelevant statistical anomalies, to which they then apply "significance" tests so they can publish and crow about their results.

Really understanding science requires more than statistical techniques. It requires the judgment that comes from familiarity with science broadly speaking, not just a tiny area of study; is that really a likely causal connection? how would it work? have I heard similar claims before that didn't pan out? does this fit with other research, or with findings in other, related disciplines? what are the broad problems with this kind of analysis?

When I apply this sort of thinking to new scientific claims, I end up writing off whole fields of research. Diet is first among them; are their any claims about diet science, beyond the most obvious (eat a balanced diet and don't eat too much) that have stood the test of time? On the other hand I end up with renewed confidence in certain other fields, especially evolution. For me climate science hangs in between, very likely to be true in its outlines but rife with unjustified precision and bogus claims.

Here is my tip for interpreting scientific claims: science is hard, especially the science of complex, interrelated systems like the human body or the earth's climate. When it comes to subjects like these, pay no attention to any single study. Only a massive weight of evidence derived from many different approaches should change your mind.