It has been said that "numbers tell the tale." But at the Education Writers Association forum on School Improvement Grants in Chicago Saturday, I saw the same numbers being used to tell some very different stories -- raising serious questions about efforts to improve schools by closing them or giving them "turnaround" treatments.

Chicago has long been the epicenter for school "turnarounds." Going way back to 1997, this city has had strong mayoral control of its schools, and has experimented with school closures, reconstitutions, and the latest version, turnarounds. Former Chicago education CEO Arne Duncan brought this approach to the federal level through Race to the Top and an ambitious $4.5 billion School Improvement Grant program, targeting the lowest 5% of schools across the country.

Early in the day, we heard from Jason Snyder, the man in charge of turnarounds at the Department of Education. He shared numbers that we heard from Secretary Duncan last week.

In year one under the new SIG:

▪ Nearly one in four schools saw double digit increases in math proficiency.

▪ Roughly one in five schools had double-digit increases in reading proficiency.

In nearly 60 percent of SIG schools, the percent of students who were proficient in math or reading went up in the first year.

So here is our first chance to play the numbers game. The actual data was not released, so we have to make some inferences here - but can we tell if these are "positive results" as Secretary Duncan suggested?

It appears that regarding math, more than 75% of SIG schools did not increase by more than 10%. In reading, roughly 80% of schools failed to increase by 10% or more. Note that we are not told how many schools decreased - which is certainly possible.

But what does it really mean to say that nearly 60 percent of SIG schools went up in the first year? That sounds pretty good - but does it stand up to scrutiny?

Imagine a sort of Punnett square of possible outcomes for SIG schools. (see the image at right) We have four squares. One indicates math and reading are both up. One shows math is up but reading is down, one shows math is down but reading is up, and one shows math and reading are both down.

I assume that the nearly 60% number describes the schools that fall into any of the three boxes other than "math and reading both down." 

If the data shows that nearly 60% of the schools have seen increases in reading or math, then we must have more than 40% in the red box, indicating reading and math are both down.

If we take a sort of null hypothesis, that these scores are moving randomly, then we would expect 25% in each box. In that case a full 75% of the schools should show improvement in math, reading, or both. What we seem to be seeing is that SIG schools are improving at a pace less than random chance would provide. The real question is not "how big is the boost these schools are getting?" The question is are they really improving at all? Or is this just a game of spinning numbers to try to make things appear better than they are? The claim has been made that this is a success story -- but I am struggling to see that. It is hard to tell with the very limited data we have been given. If I have made any erroneous assumptions, please help me out here! (note: I sent a request for clarification on these numbers yesterday but have not heard a response yet. If one comes, I will share it in an update to this post.)

Next, I want to take a look at two dueling reports that were shared at the meeting.

The initial report that was shared and referred to often in the day is a glossy, well-funded study by the Consortium on Chicago School Research. The report summary states: 

This report finds that four years after undergoing dramatic reform efforts such as turnaround, very low-performing elementary schools in Chicago closed the gap in test scores with the system average by almost half in reading and two-thirds in math. The improvements took time to develop; test scores were not significantly better in the first year of reform, but grew larger over time.

This report chose to compare the turnaround schools to a comparison group of schools with similar performance issues at the time of the turnaround.

Within the report, we find that in the year prior to the start of the turnaround process, the turnaround schools were 0.46 of a standard deviation below the system average in reading. After four years, they had risen to 0.26 of a standard deviation below the average. Math achievement went from 0.53 below to 0.2 below average.

The report found that the seven high school turnarounds had no effect at all - the number of students on track to graduate was similar to non-turnarounds.

Another report was also mentioned at the conference, though copies were not distributed. This one was done by a group called Designs for Change, and is entitled Chicago's Democratically-Led Elementary Schools Far Out-Perform Chicago's "Turnaround Schools."

This report compared Chicago's 198 non-turnaround elementary schools to the 12 that have been "turned around" under the guidance of the Academy for Urban School Leadership (AUSL), and highlighted the 33 that had outperformed the system-wide average. (These 210 schools are all neighborhood schools that are more than 95% free and reduced lunch).

Designs for Change director Don Moore explained the rationale for this comparison group:

The reason that we emphasized the superior achievement of the top 33 democratically-led-led schools, as compared with the 12 turnaround schools. is that the democratically-led schools were, by and large, never given special resources or flexibility and were almost never acknowledged publicly for their accomplishments or studied to see what made them so successful in student achievement and retaining teachers. Several had major facilities problems, compared with the complete makeover given to turnaround schools. Doing so much better with so much less in the way of money and other advantages indicates that democratically-led schools have much greater potential for wide-spread expansion and impact.

In contrast, the turnaround schools were repeatedly featured by the school system and the press as being the models for other schools to follow--even though the top-achieving turnaround school (Howe) was out-scored by 52 democratically-led schools and even though the turnaround schools receive $7 million in extras resources over their initial five-year AUSL involvement.

We wanted to document the potential that the top democratically-led had demonstrated, without substantial extra resources, but by following the practices and exhibiting the social cohesion that our previous research had indicated was the key to creating a high-poverty high-achieving school (see Designs for Change, "The Big Picture").

According to the Designs for Change report, the schools that have approached improvement with patience, working to build community support and teacher stability, following principles of democracy and inclusion, have made gains far beyond those being seen at the mayor's "Turnaround schools," which have been showered with resources.

One hundred-ninety-eight of these extremely high poverty Chicago elementary schools on which the study focuses are led by elected Local School Councils, consisting of six parents, two teachers, one non-teaching staff member, two community members, the principal, and (in high schools) a student.

...the vast majority of Local School Councils quietly oversee school policy and carry out their official duties of evaluating the principal, approving the budget, and monitoring the School Improvement Plan ... and are active in building school and community partnerships.

The report offers us portraits of some of these schools. Here is one:

Gallistel Language Academy (1,444 students in three buildings, 93% Latino, 96% low income). 70% of students Meet or Exceed ISAT Standards in Reading. 79% Meet or Exceed ISAT Standards in Math (83% following previous state policy about when English Language Learners had to begin taking the ISAT). Gallistel's LSC hired a new principal in spring 2000, who has unified the school. Gallistel is intensely over-crowded. 400 to 600 Gallistel parents, teachers, and students have testified at school system facilities hearings each year over the past several years, asking for major repairs. The main building is plagued by electrical outages, leaks, and widely varying temperatures. Despite these obstacles, 75% of Gallistel teachers have remained at the school for at least four years.

We also get a picture of another world for the "Turnaround Schools."

Through Chicago's Turnaround School strategy, all of the current staff of a Turnaround School are fired, and almost all aspects of the school's operation are tightly controlled by either an independent "Turnaround Specialist" that contracts with the Chicago Board (thus far in Chicago, the Academy for Urban School Leadership or "AUSL") or by a department of the school system administration called the Office of School Improvement. These leadership groups have near-total authority to select staff, define the school's learning program and oversee other important aspects of students' learning experiences (such as discipline), based on a contract or series of contracts that was supposed to last five years. Turnaround teachers remain unionized.

These turnarounds have been in operation for as long as five years, so we now have some data to see what has worked, and what has not.

This study focused on three areas. It looked at student reading scores, teacher turnover, and the resources provided to the different schools. We get a detailed comparison of the standings of all 210 schools, and some facts emerge.

• 33 schools were above the city-wide average for all of Chicago's 480 elementary schools. All 33 high-achieving schools are led by elected Local School Councils who chose their principals and have unionized teachers, typically teachers with substantial experience.
• 14 of the 33 highest-scoring schools were more than 90% African American.
• 16 of the 33 highest-scoring schools were more than 85% Latino.

No Turnaround School scored above the city-wide average.

There are only three Turnaround Schools among the top hundred schools.

Heavy investment in the turnarounds was part of the plan. These schools got a fresh start, with staff that was specially trained. The report looks at the promises made at the outset, and compares those to the results we are seeing.

AUSL has repeatedly stated that if they were given complete control of a school for five years, they would "reset" or completely transform the culture of a Turnaround School.

As a result, they argued, a new cadre of "turnaround-ready teachers" would be in place to sustain improvement long-term and that AUSL staff could substantially transition out of the school after five years.

The "Turnaround schools" were low-performing schools, in need of this "dramatic" intervention. What happened to similar schools not lucky enough to get this?

...13 non-Turnaround Schools started at a lower average score than the Turnaround Schools, but ended with a higher average score. And unlike the Turnaround Schools, these 13 schools had characteristically not received any substantial extra resources during the period for 2006 to 2011. Further, they had sustained their larger Average Yearly Gains for 5 years, rather than 1 to 5 years. They indicate that democratically-controlled non-Turnaround Schools that began in the same reading achievement level as the Turnaround Schools can out-perform them.

The report also looked at levels of teacher turnover. 

Chicago's Turnaround Schools had the opportunity to choose an entirely new staff, and to staff Turnaround Schools with teachers AUSL said, were "turnaround- ready." Chicago's Turnaround Schools also had dramatic improvements in their facilities and extensive AUSL staff who worked to support teachers. AUSL carried out a one-year clinical preparation program for about half of their teachers.

So how did this investment in new staff pay off? 

...only an average of 42% of the original Turnaround Teachers who taught in the six Turnaround Schools in 2008-2009 were still teaching there four years later in 2011-2012. This creates a constant need in Turnaround Schools to identify new teachers and makes the goal of fundamentally changing a school's culture more difficult.

In contrast, an average of 71% of the teachers in the four profiled Comparison Schools who taught in 2008-2009, were still teaching in the same school in 2011- 2012. As noted in Section 1, two of these four schools face severe facilities problems, which is an established incentive for teachers to leave.

The Turnaround schools received significant additional resources - roughly $7 million per school -- to support the change process. The Academy for Urban School Leadership collected revenues to manage the schools, and extensive renovations were carried out at the school sites. These funds were not available to the 198 schools that were democratically run.

The Designs for Change report offers a several conclusions, including these:

This study indicated that the high-poverty schools achieving the highest reading scores were governed by active Local School Councils who chose their principals, and had experienced unionized teachers.

Related research indicates that high-poverty schools with sustained test score improvements tend to carry out a specific set of practices and methods of organization. These effective elementary schools have dedicated strong Local School Councils, strong but inclusive principal leadership, effective teachers who are engaged in school-wide improvement, active parents, active community members, and students deeply engaged in learning and school improvement.

A separate research study came out last week indicating the heavy toll teacher turnover takes on student performance, especially at high-poverty schools. A constantly changing staff creates turmoil, and damages collegial relationships that are central to school success. When teachers leave, students and teachers who stay behind are affected by the instability. Reform that is built on "forceful intervention" and "dramatic change" faces a real challenge if stability and continuity cannot be established.

No Child Left Behind was based in part on the "Texas Miracle" that George W. Bush touted when he came to office. Schools in Texas had supposedly figured out how to beat poverty by setting the bar high - so that was written into the law of the land. Unfortunately the miracle turned out to be a mirage, produced by holding students back and other manipulations. Now we have another national strategy that may be similarly drawn from a rather questionable model.

The Department of Education's Jason Snyder said it was important to have humility. But humility is hard to detect in a program that demands that teachers be fired or schools closed as a condition for receiving scarce funding. This approach of forceful intervention is steeped in the arrogance of technical expertise, because the authors insist they know what must be done, and those who have failed need to be removed in order for success to occur. We have now reached the point where promises of future growth must be replaced by reports of accomplishment - and that is why the numbers have become so important to understand.

The education reporters who shared the room with me Saturday are getting schooled in the toughest kind of reporting - making sense of competing claims. Numbers may not lie, but they are sometimes spun in ways that make the truth hard to see.

What do you think? Have you experienced a school turnaround or reconstitution? Are we seeing numbers spun here?