Metascience, or, a Crisis is a Terrible Thing to Waste

In The Structure of Scientific Revolutions, Thomas Kuhn called what scientists do most of the time "normal science." They work in approved ways on approved problems within established institutions. They generate data that is meaningful within the established paradigm. It's a nice life. But what if those approved ways of working are faulty and produce bad results? Whose job would it be to fix that? In "A Vision of Metascience," a sort of long think piece, Michael Nielsen and Kanjun Qiu explore exactly this question, asking how science can correct its own methods when so much is invested in them.

Their analysis centers on the "replication crisis" that overwhelmed social psychology in 2011-2015. That crisis had many components, including a paper by Daryl Bem that used conventional statistical tweaks to show causality moving backward in time – thus, that those tweaks could make the data show anything– but the key was a 2015 paper from the Open Science Collaboration. That paper reported the results of attempts to replicate 133 famous experiments in social psychology, of which only 36 produced statistically significant positive results. (Since then a few of those findings have been replicated in other studies, but most have not.) One response to this crisis was the creation, by the Center for Open Science, of "Registered Reports." In this system scientists pre-register their experimental protocols and other methods and get them peer reviewed by journals before the experiment is performed; if the plan passes peer review, then the result is automatically published, whatever it happens to be.

The founding of the Center for Open Science is quite a story in itself:

Many people have played important roles in instigating the replication crisis. But perhaps no single person has done more than Brian Nosek. Nosek is a social psychologist who until 2013 was a professor at the University of Virginia. In 2013, Nosek took leave from his tenured position to co-found the Center for Open Science (COS) as an independent not-for-profit (jointly with Jeff Spies, then a graduate student in his lab). Nosek and the COS were key co-organizers of the 2015 replication paper by the Open Science Collaboration. Nosek and the COS have also been (along with Daniël Lakens, Chris Chambers, and many others) central in developing Registered Reports. In particular, they founded and operate the OSF website, which is the key infrastructure supporting Registered Reports. That's not all OSF does, it's also a general platform for scientists to share papers, code, and data, and is designed to make it easy for other labs to replicate work. Finally, Nosek's been a frequent public advocate for replication, doing on-the-ground work to change how scientists think about the subject, which has required both strong scientific argument and also good marketing and brand-building. In short, Nosek and the COS are key figures in driving a massive, systemic change in social psychology. They're helping change the culture of science.

The origin story of the COS is interesting. In 2007 and 2008, Nosek submitted several grant proposals to the NSF and NIH, suggesting many of the ideas that would eventually mature into the COS. All these proposals were turned down. Between 2008 and 2012 he gave up applying for grants for metascience. Instead, he mostly self-funded his lab, using speaker's fees from talks about his prior professional work. A graduate student of Nosek's named Jeff Spies did some preliminary work developing the site that would become OSF. In 2012 this got some media attention, and as a result was noticed by several private foundations, including a foundation begun by a billionaire hedge fund operator, John Arnold, and his wife, Laura Arnold. The Arnold Foundation reached out and rapidly agreed to provide some funding, ultimately in the form of a $5.25 million grant.

Buoyed by the funding, in 2013 Nosek left the University of Virginia to start the Center for Open Science. This may seem strange: why not keep it at the university? But as we've seen the work of the COS was not social psychology in the conventional sense. Rather, Nosek was something else, a metascience entrepreneur, working to achieve a scalable change in the social processes of science. Setting the COS up independently gave them freedom to operate in ways difficult in a conventional academic environment. For instance, in many universities it would be difficult and slow to hire the designers and programmers needed to develop infrastructure such as OSF and Registered Reports. Nosek estimated to us that roughly 1-in-5 of the COS staff could be considered researchers in anything like the conventional sense. The repeated objection when attempting to make such hires in an academic environment is "that's-not-really-science". It's ironic, in retrospect: Nosek and the COS are having tremendous impact on psychology, as a consequence of placing metascience at the core of their practice. It's a more expansive view of what a scientist can be.

This story concretely reflects many of the inhibiting factors we discussed earlier. Consider the "that's-not-really-science" problem: change in the social processes of science is no-one's job, certainly not the job of working scientists. Or the challenge of raising funds via conventional channels: it seems to us not an accident that the COS ultimately raised money from an unconventional source. And then the structural obstacle of hostility from important peers. Nosek reports a journalist telling him that a "big shot" peer had said "Nosek is John Arnold's willing idiot". Tage Rai, who became an editor at Science magazine after the 2015 article, repeatedly attacked work on replication, claiming for instance that there "are powerful private and gov't interests hoping to co-opt the replication crisis to gain leverage over deciding what kind of research you can produce", and directly attacking Nosek. So the replication crisis is a story of perseverance, not just by Nosek, but by everyone involved.

The good news here is that science is working to correct itself, which more than you can say about any other major part of our culture. The bad news is that such work is difficult and unending.