How to Distinguish the Effect of Schooling from the Effect of Everything Else

Kevin Drum highlights some research that tries to measure the quality of school systems by separating out two variables: how well students are doing in the third grade, vs. how much they learn from the third grade to the eighth:

Third-grade scores are (probably) strongly influenced by poverty and home life, while growth from third to eighth grade is (probably) more influenced by the quality of schooling. They have little to do with each other.

I find it fascinating that the performance of students in the third grade is so weakly correlated with how much they learn over the next five years. Most of what I have read makes it seem as if genes, home life and neighborhood count for almost everything and the quality of schooling for little, but this measure makes it look like the quality of schools and teachers matters a lot. Schools vary a lot in terms of how much they teach third through eighth graders.

The study's authors write:

Grade 3 average scores are likely much more strongly influenced by early childhood experiences than the growth rates….Some caution is warranted in interpreting the average growth rates as pure measures of school effectiveness. Nonetheless, relative to average test scores (at grade 3 or any grade), the growth rates are closer to a measure of school effectiveness.

If we take the growth rates, then, as rough measures of school effectiveness, then neither socioeconomic conditions nor average test scores are very informative about school district effectiveness. Many districts with high average test scores have low growth rates, and vice versa. And many low-income districts have above average growth rates. This finding calls into question the use of average test scores as an accountability tool or a way of evaluating schools.

If this finding holds up, it very much does.