The Evidence is Stronger, But The Conclusions Remain the Same

Several months ago I published an article highlighting the lack of research concerning i-Ready: a genuinely strange state of affairs for a program currently being used by over 13 million students.

At the time, only four independent studies of this tool existed – and none had been published in a reputable, top-tier journal.

That has finally changed.

Last month, Online Learning – a Q1 educational journal – published the strongest evaluation of i-Ready to date. Although not a randomized controlled trial (it was a retrospective observational study), this is a careful, well-executed piece of work using sophisticated statistical methods and tens of thousands of students.

As such, this deserves to be taken seriously.

So, what did they find?

A QUICK REFRESHER

Until now, the strongest evidence on i-Ready came from several evaluations conducted by Johns Hopkins Researchers. Across those studies, the conclusion was bleak:

• Across grades 3 through 6, i-Ready showed a 0.8% improvement in mathematics performance with no measurable change in reading.

However, seeing as these studies were industry funded and non-peer reviewed, it could reasonably be argued that the conclusions are not definitive.

But now we have something stronger and much larger – so does it change the picture?

THE NEW STUDY

Researchers from Georgia State University examined two years of i-Ready implementation (2021-2023) across grades 1 through 9 in a large Georgia district with 131 schools – 81 of which used i-Ready.

Overall, 67,602 student-year cases were eligible to use i-Ready, yet only 27,717 - about 41% - actually used the program. The authors acknowledge they were unable to explain why most eligible students never used the software.

Importantly, the authors are refreshingly honest about the limitations of their data. Before presenting any results they write:

• “Our analyses are descriptive versus causal, as we cannot fully…understand or account for selection into treatment…[and] we were unable to account for other recovery interventions occurring during the same period.”

That sentence gives me confidence in the paper: rather than overselling the findings, the authors openly acknowledge exactly what their design can (and cannot) tell us.

THE RESULTS

After two years, students who used i-Ready scored approximately:

• +0.04 standard deviations higher in Reading
• +0.04 standard deviations higher in Math.

That’s it.

That’s the headline.

To put this into context, an effect size of 0.04 standard deviations is equivalent to moving the average student from the 50th percentile to about the 51.5th percentile - a gain of just 1.5 percentile points.

Although the paper reports some intriguing subgroup findings (particularly among students with intellectual disabilities) the absence of a reported correction for multiple comparisons makes those findings difficult to interpret.

Finally, because the study does not clearly establish that treatment and comparison students received equivalent amounts of total instruction, it cannot rule out the possibility that some - or even all - of the observed +0.04 standard deviation benefit reflects additional time spent learning rather than any unique effect of the software itself.

To clarify, if students in the control condition received their normal classroom instruction while students using i-Ready received that same instruction PLUS additional time on the platform, then the comparison is no longer between instructional methods - it’s between different amounts of instruction. Under those circumstances, any observed advantage could simply reflect the well-established benefits of increased practice and learning time.

Had the control group, instead, spent an equivalent amount of time engaged in evidence-based, non-digital practice (such as worked examples, guided practice, or teacher-led small-group instruction), it is entirely plausible that they would have achieved similar, or even greater, gains.

Because this was a retrospective study, however, the authors cannot determine whether i-Ready replaced existing instruction or merely supplemented it. That leaves this important alternative explanation unresolved.

The overall story, however, is remarkably simple: the strongest study ever conducted on i-Ready found almost exactly the same small effects as previous evaluations.

Ironically, the research so many people have been waiting for ultimately changes nothing: the evidence is stronger, but the conclusion remains the same.

A FEW ODDITIES

Although this paper is strong, that doesn’t mean every aspect of it is beyond criticism.

In fact, the researchers made three decisions that deserve closer scrutiny.

1. “Weeks of Learning”

The researchers translated a gain of +0.04 standard deviations into roughly two additional weeks of learning.

Although that might sound impressive, readers are never shown how this conversion was calculated: there are no equations, no adaptation tables, not even a citation.

Presumably, the authors estimated typical MAP Growth gains across each year level, assumed those gains accumulated at a constant rate throughout the school year, and divided accordingly. Converting a single end-of-year difference into "weeks of learning" therefore implicitly assumes that learning accumulates at a constant rate throughout the school year. That assumption is neither defended nor justified in the paper (nor is it reflective of the typically saltatory nature of learning gains).

Regardless, the “weeks of learning” reported in this paper is an inference, not an observed result. Accordingly, although it may make for an appealing headline, it is much less scientifically convincing.

2. The Cost Estimate

Following their learning analyses, these researchers make an odd reference to cost: they estimate that the district spent $831,510 on i-Ready subscriptions over the two years. This was calculated by assuming $30 per subscription across the 27,717 active users.

There are at least two problems with this estimate.

First, the authors acknowledge there were a total of 67,602 students with access. If the district purchased subscriptions for every student who had access (as is common with district-wide software contracts) the subscription cost would have been approximately $2,028,060 rather than $831,510.

Second, they offer no reference for their proposed $30 subscription fee - and it is unclear whether this figure reflects access to a single subject or to both i-Ready Reading and i-Ready Mathematics. Because the paper reports outcomes for both subjects but never specifies how licenses were allocated, the true program cost could be substantially higher, potentially approaching $4,056,120 if separate subscriptions were required for both Reading and Mathematics.

Extrapolated, that suggests between $500,000 and $1 million dollars were spent for each 0.01 standard deviation growth amongst users: a substantially altered cost-benefit calculation.

The authors themselves note a number of other interventions (such as small-group instruction and after-school programs) that almost certainly generate better results for far less cost.

3. What Was Actually Measured?

Finally, the study measured outcomes using MAP Growth.

As discussed in my previous article, MAP Growth is an adaptive computer assessment that closely resembles the instructional structure of many digital tutoring platforms – including i-Ready.

By no means does this invalidate the results, but it does raise an important question about transfer. Would we observe the same gains on assessments less closely aligned with the tutoring environment? Or would we have seen growth even lower than 0.04 standard deviations?

Unfortunately, this study cannot answer that question.

SO NOW THEN…

This paper represents the benchmark evaluation of i-Ready: it is statistically sophisticated, transparent, and avoids making causal claims its design cannot support.

And it reaches almost the exact same conclusion as every previous evaluation.

i-Ready appears capable of producing very small improvements on assessments closely aligned with the instructional model. Whether those improvements justify the financial investment, loss of instructional time, and opportunity costs is a separate question – one schools should be asking before investing millions of dollars into digital tutoring.

The methods may have improved, but the estimated impact has not.