GARRISON KEILLOR HAD IT RIGHT. One of the opportunities a long graduate level examination offers is to catch up on some reading, in this case the Fall 2002 Journal of Economic Perspectives. Thomas J. Kane and Douglas O. Staiger write "The Promise and Pitfalls of Using Imprecise School Accountability Measures." (The article is available online through J-Stor, not to be confused with JSTARS, but that is configured university by university.) The tease, from the abstract: "[W]e describe the statistical properties of school test scores, which are less reliable than is commonly recognized, and explore the implications for school incentives. Many accountability systems that appear reasonable at first glance perform in perverse ways when test score measures are imprecise." What's at the heart of the imprecision? Good ol' small sample effects. (A quick check at Number 2 Pencil reveals a number of other difficulties with testing; if I've missed coverage of small sample effects there, please advise.) Thus, small schools are more likely to turn up as "underperforming" on the basis of a drop in their test scores, or to go from "underperforming" to "excellent" in a year, because of the presence of a few low- or high- scorers in a small class of test-takers. The paper is also scornful of a California assessment of minority achievement: "California's rules are analogous to a system that makes every [c.q.] school flip a coin once for each [numerically significant] minority subgroup and then gives awards only to schools that get 'heads' on every [c.q.] flip." Moreover, the rules do not provide for the kinds of reallocations that achieve greater efficiencies, which would involve shifting students to more effective schools and closing down less effective schools.