Data, Noise, and the Pressure of Big Money Science

Last winter, Nature published a paper by a team at Johns Hopkins University, based on a method they say reveals interactions between genes. In July the lead author of the paper, Yu-yi Lin, committed suicide in his office in Taiwan. Poking around, Post reporter Peter Whoriskey discovered that the work behind that Nature paper had long been criticized by the team's own statistician, Daniel Yuan. The impressive thing about the article Whoriskey wrote is that even though Hopkins and the other authors of the paper refused to speak with him, making Yuan his only inside source, he managed to write a pretty substantial, non-scandal-mongering piece.

This is not a scandal in the sense that somebody stole money to spend on his mistress. It is a scientific dispute. On one side are Lin and laboratory director Jef Boeke, who have been very successful in getting NIH grants to study gene interactions, first in fruit flies and then in humans. On the other, Yuan, who wrote in a response to the Nature article,

The overwhelming noise in the . . . data and the overstated strength of the genetic interactions together make it difficult to reconstruct any scientific process by which the authors could have inferred valid results from these data.

This is the sad reality of American science today. Things get published that may or not be true, based on the reputations of the authors and the trendiness of the claims. Sometimes the work is fraudulent or incompetent; other times it is merely premature, rushed into publication by scientists who need to justify their research grants and apply for more funding. Based on emails that Yuan provided to Whoriskey, it seems that the Hopkins team was feeling the pressure:

During Yuan’s time there, the lab received millions in NIH funding, and according to internal e-mails, the people in the lab were under pressure to show results. . . .

As far back as 2007, as the group was developing the methodology that would eventually form the basis of the Nature paper, Yuan wrote an anguished e-mail to another senior member of the lab, Pamela Meluh. “I continue to be in a state of chronic alarm,” he wrote in August 2007. “The denial that I am hearing from almost everyone in the group as a consensus is troubling to me.”

Meluh quickly wrote back: “I have the same level of concern as you in terms of data quality, but I have less basis to think it can be better. . . . I’m always torn between addressing your and my own concerns and being ‘productive.’ ”

Then Boeke weighed in, telling Yuan that if he could improve the data analysis, he should, but that “the clock is ticking. . . . NIH has already given us way more time than we thought we needed and at some point we’ve got to suck it up and run with what we have.”

Ultimately this sort of thing results from our competitive, meritocratic social structure. Top science jobs are hard to get, and they go to people who produce publishable results. This makes the pressure to get those results extreme. Suppose you devote three or four years of your life, at a crucial stage of your career, to research that ultimately doesn't pan out. What then? Under our system you are screwed, and it may well be that the only way for you to continue as a scientist is to doctor up your results enough to get them published somewhere. This, it seems to me, is the likely background to Lin's suicide.

In the 1960s, nearly 60 percent of applications for NIH grants were successful; now, less than 20 percent are. The intensity of the competition for that money, and for jobs in the labs that have grants, will only get worse. For scientists, it means that besides their other problems they regularly have to negotiate ethical minefields, wondering when to admit defeat, risking their funding, and when to dress it up as some sort of success; for the rest of us it means that the claims coming from cutting-edge labs have to be treated with extreme skepticism, and that not even publication in the top tier of journals is a guarantee that the science is any good.