The NIH Worries about Bad Science

Francis Collins, head of NIH, and Lawrence Tabak, one of his top deputies, have published an editorial in Nature complaining of “a troubling frequency of published reports that claim a significant result, but fail to be reproducible.” It is interesting to see that the heads of big science funding bodies are aware of these problems and starting to think about ways to combat them:

Factors include poor training of researchers in experimental design; increased emphasis on making provocative statements rather than presenting technical details; and publications that do not report basic elements of experimental design. Crucial experimental design elements that are all too frequently ignored include blinding, randomization, replication, sample-size calculation and the effect of sex differences. And some scientists reputedly use a ‘secret sauce’ to make their experiments work — and withhold details from publication or describe them only vaguely to retain a competitive edge. What hope is there that other scientists will be able to build on such work to further biomedical progress? . . .

Exactly. But as I have argued, the biggest obstacle to reform is the meritocratic nature of the scientific world and of our society more broadly. In a world where more people want to be research scientists than there are jobs for them, there has to be some way of deciding who gets the good slots in the system, and the measure we use is journal publication.

Perhaps the most vexed issue is the academic incentive system. It currently over-emphasizes publishing in high-profile journals. No doubt worsened by current budgetary woes, this encourages rapid submission of research findings to the detriment of careful replication. To address this, the NIH is contemplating modifying the format of its 'biographical sketch' form, which grant applicants are required to complete, to emphasize the significance of advances resulting from work in which the applicant participated, and to delineate the part played by the applicant. Other organizations such as the Howard Hughes Medical Institute have used this format and found it more revealing of actual contributions to science than the traditional list of unannotated publications. The NIH is also considering providing greater stability for investigators at certain, discrete career stages, utilizing grant mechanisms that allow more flexibility and a longer period than the current average of approximately four years of support per project.

This is an interesting change; perhaps if researchers in their 30s can get seven or eight years of funding rather than four they will do more careful work and not rush their publications. But they will still face the "publish or perish" test eventually, so we'll have to wait and see if these measures make any difference.