How did you know when to stop making corrections?

In his memoirs physicist Richard Feynman said that when graduate students presented their results in the departmental seminar, explaining that they had corrected the data for this problem and that one, he often asked them, "How did you know when to stop making corrections?"

I was reminded of this by a study that came out this week on the effect of strict voter ID laws on minority voter turnout. Some liberal web sites have touted the findings, but the Times and the rest of the mainstream press have ignored them. I wondered why, but Keven Drum dove into the details of the study and he will explain it to us:

Their regression suggests that black turnout was up in states with strict photo ID laws. For some reason, though, the result isn't statistically significant, so they ignore it. Conversely, their result for primaries shows black turnout down. But even though it's a weaker result, it is statistically significant, so they report it.

And there are other things that make no sense. Not only do the authors report numbers for depressed turnout that are far larger than anyone has gotten before, but they suggest that photo ID laws cause black turnout to rise while mixed-race turnout declines. That's pretty hard to fathom.

There are other problems. Their charts are incomprehensible. They rely on data collected over the internet. And the results in this paper are precisely the opposite of what one of the authors reported just a year ago in a paper using the same methodology. . . .

And, what reminded me of Feynman, they offer this list of all the ways they corrected their data:

We also control for individual demographic characteristics... age... education level... family income... nativity... gender, marital status... having children, being a union member, owning a home, being unemployed, and religion... and whether the respondent was registered to vote in the pre-election survey... We also have to incorporate other state level electoral laws... early voting... all-mail elections...no excuse absentee voting... the limit on the number of days before the election that residents can register to vote.... Finally, to help identify the independent effect of ID laws, our analysis has to include the electoral context surrounding each particular election... political competitiveness of each state... the presence of different electoral contests... whether the Senatorial and Gubernatorial contests are open seats or not, whether the Senatorial and Gubernatorial contests are uncontested or not, and finally the region (South or not).

Maybe that isn't fair to them; maybe all of those things matter. But the result is a study that will never persuade anyone of anything and probably shouldn't. Besides, as I have said before, if it takes tons of fancy statistics to identify an effect, it can't be very big.

I think strict voter ID laws are wrong, because they discourage participation in elections, and we should be doing all we can to encourage participation. If one person who has never voted decided for once to try it but got turned away, I think that is sad. But honestly these laws probably have little effect on outcomes because the people they discourage weren't likely to vote in the first place.