I am firmly of the belief that you lay out grading expectations for students with as much notice as possible, and so here is my grading scale for reporters and newspapers in handling the value-added scores that the city Department of Education is releasing today, after courts have mandated their release:

A: Newspaper reporting on the NYC value-added scores that earns an “A” contains accessible descriptions of the political controversy over value-added scores and methodological issues surrounding the value-added scores and does not name individual teachers. In addition,coverage includes several dimensions of depth such as some of the following:¹

• A detailed discussion of the controversies over the value-added pilot in NYC.
• A discussion of implementation issues in using value-added measures in evaluation in other jurisdictions. To be considered as credit towards an “A” grade, the discussion must have a substantial sidebar on one other jurisdiction or include more than one jurisdiction in the discussion.
• A discussion of key differences between the calculation of value-added scores released by the NYC DOE and other calculations, such as the NYS DOE calculations that are anticipated. To be considered as credit towards an “A” grade, the discussion must include at least two important design decisions, such as whether to include demographic variables (and which to include) or the difference between model residuals as “effect scores,” simple gain scores, and “whether students gained” measures.
• Putting the scores released by NYC DOE into the broader discussions of value-added calculations, including direct or indirect quoting of well-known researchers in the topic such as Doug Harris, as well as the “usual suspects” quoted in the NYC context. To be considered as credit towards an “A” grade, this context must directly or indirectly quote more than two well-known VAM researchers.
• Reporting on the reporting: Putting the score release and response into the broader controversy over reporting of teachers raised by the L.A. Times display. To be considered as credit towards an “A” grade, this analysis must include a discussion of the range of journalist responses in New York City.

B: Newspaper reporting on the NYC value-added scores that earns a “B” contains accessible descriptions of the political controversy over value-added scores and methodological issues surrounding the value-added scores and does not name individual teachers, but does not have any of the additional qualities indicated in an “A” grade for coverage.

C: Newspaper reporting on the NYC value-added scores that earns a “C” contains accessible descriptions of the political controversy over value-added scores and methodological issues surrounding the value-added scores and does not name individual teachers, but “C” coverage has significant errors of commission or omission in reporting the story,  such as one of the following:

• One or more format of display of scores (print or online) may contain standard errors for individual scores of anonymized teachers, but anonymized scores are displayed without critical information about the limits on accuracy. This includes displays of value-added measures in a nonquantitative way (such as the pointer on a color-coded dial from green to red) that imply a highly-accurate point estimate.
• Indulges in some sensationalizing scores by focusing a significant passage of text on one or more individual unnamed teachers with very low scores, possibly supplemented with anecdotal or out-of-context reporting about the teacher, classroom, school, or students of the teacher.

D: Newspaper reporting on the NYC value-added scores that earns a “D” may have some context and discussion of methodological issues surrounding the value-added scores but nonetheless contains very serious weaknesses in handling the material, such as one of the following:

• Reprinting the score database with individual scores attached to named teachers, even with context and standard errors or other indication of the limits of accuracy.
• Indulges in sensationalizing of scores by focusing a significant passage of text on one or more individual named teachers with very low scores, possibly supplemented with anecdotal or out-of-context reporting about the teacher, classroom, school, or students of the teacher.

F: Newspaper reporting on the NYC value-added scores that earns an “F” fails the essential obligations of reporting the story fairly and in a non-sensationalist manner, most likely in one or more of the following:

• Reprinting the scores attached to named teachers with no standard errors for individual scores or otherwise displaying value-added data in ways that a reader would assume the point estimate is accurate to a great degree. This includes displays of value-added measures in a nonquantitative way (such as the pointer on a color-coded dial from green to red) and consistent displays on webpages (not just print editions) that imply a highly-accurate point estimate.
• Sensationalizing scores by focusing a sidebar or major article on one or more individual named teachers with very low scores, possibly supplemented with anecdotal or out-of-context reporting about the teacher, classroom, school, or students of the teacher.
• Not covering the story and not covering the release of the data in any way. (This is specific to New York-area papers.)

Notes

1. You may note that “drawing substantive conclusions about teaching quality in NYC” is not in the list of examples, in part because of the questions raised about the Teacher Data Reports. That doesn’t mean that doing so is beyond the bounds of good journalism, but I think doing so with integrity would be much more difficult than any of the items in the “A” list of activities. [↩]