On occasion, students and reporters ask me what makes me trust or distrust folks who claim to be education researchers, and it’s a harder question to answer than one might think. As an historian with some quantitative training, I am eclectic on methods–I have no purity test other than “the evidence and reasoning have to fit the conclusion.” It’s not the existence of error: even great researchers make occasional errors, and it’s a good thing in the long run for researchers to take intellectual risks (which imply likely error/failure). Further, we all have the various myside biases cognitive psychologists write about.

But when I come across something like the following produced by the Cato Institute’s Andrew Coulson and displayed by Matthew Ladner twice on Jay Greene’s blog (including on Thursday), I start to wonder. Here’s Coulson’s chart:

Look at both vertical edges, and you will see that this is a two-vertical-axis chart, with the per-pupil costs on the left and some (unstated) measure of achievement that is labeled “[subject] scores” in percentage terms. Because one can often manipulate units and axes to leave almost any impression one might wish, I wondered if I could use the same underlying data to leave the opposite impression.

First, once I looked at Table 182 from the 2009 Digest of Educational Statistics, it became clear that the cost figures (supposedly the total cost of a K-12 education taken by multiplying per-pupil costs by 13) is false. If you look at the columns in the linked data (Table 182), the per-pupil costs when adjusted for inflation approximately double rather than triple as asserted in this figure. Second, there is no possible source for the approximate “0%” line from NAEP long-term trends data, unless there is an additional calculation unexplained by Coulson.

But let’s look at the real data and see if you can manipulate that to leave an opposite impression:

Whoa, Nelly! It looks at first glance from this figure that Coulson’s dead wrong: when comparing the trend lines for per-pupil costs after inflation (the green line: like that, green = money?) to reading (blue) and math (brown) trends, it looks like reading trends may not be great, but math looks to have had a pretty good return on the total investment in all K-12 education.

How did I manipulate the data to get this result? First, I chose a measure of per-pupil expenditure change that was both acceptable academically and also would shrink the apparent change: the natural log of the ratio of current per-pupil expenditures to 1971-72 per-pupil expenditures. Then I put average NAEP long-term scale scores on another academically-acceptable measure, using the starting scale score for the interval as 100 for each subject and age (1971 mean scale = 100 in reading, 1978 mean scale = 100 in math). Then I made sure the vertical axes had the “right” low-high range to contrast the greatest increases in NAEP trends (math for 9- and 13-year-olds) with a visually-shrunken per-pupil trend line: the vertical scale on the left included nothing more than the total range of the re-calculated trend scores, while the vertical scale on the right was just a little more than twice the range of the natural log of the expenditure ratios. Voila!: a figure that looks like it shows the exact opposite of what Coulson’s figure looks like it shows. Addendum: Two commenters misunderstood my point even with the phrase “manipulate the data” in the first sentence of this paragraph. So maybe I should make clear that, yes, the figure I prepared is also a demonstration of what not to do, except that in contrast with Andrew Coulson, I am telling you exactly how I am making the pretty colors dance to my tune.

This manipulation of data and presentation by Coulson is the type of behavior that makes me distrust not only the piece in which something like this appears but the broader work of an individual.