Replication Studies, or, Not Trying Very Hard to Confirm

Matthew Lieberman raises some interesting questions about the movement to replicate published studies in psychology:

Now there's "the replication crisis", which is a replication crisis in science, if it's even a crisis. It's just that we need to be reminded sometimes that when you see the first flashy study published in science or psych science, it's just an anecdote. It's a scientific anecdote, and we should go collect some more. It can be a really exciting one that you want to tell all your friends about, but it's one little tiny piece of data. We've perhaps taken to assuming those things were facts, and then we're shocked when those things don't replicate in study number two.

There's a lot of stuff going on where there's now people making their careers out of trying to take down other people's careers. The replication isn't necessarily an unbiased process, as it's presented. There are camps, and suddenly now failing to replicate someone else is really seen as an indictment, in many people's eyes, of the person who did the original research, rather than saying there's expectancy effects. If I expect not to replicate someone's work, that's going to influence how I design my study, the measures I look at, it's going to influence how I interact with my participants. I've heard stories about participants saying that they've been told, "Oh, you're just in a replication effort, so it doesn't matter if you know more than you should." There are things going on, and it is troubling to me.

I can see this happening in two ways. When people dislike a finding, they may try to make their experiment fail. Or, if they are trying to replicate ten studies in a year, they may not try very hard to get them right. As Lieberman says, the way people approach a study is especially important when it involves human subjects, and people who want to reject a finding can probably make that happen. Lieberman says this is being done most aggressively to studies that bother people because their findings don't fit with the general idea of  how the mind works, and if true that would be troubling.