The introduction of generative AI into school settings remains highly controversial almost four years after the launch of ChatGPT. One source of growing public scepticism about the educational value of AI is a persistent lack of evidence of its benefits, plus concerns that much of the experimental research that has been produced is deeply flawed and misleading in its conclusions. As a result, new proposals are appearing about what would constitute good “policy-relevant evidence” on AI in education.

These proposals are important to identify and unpack since they are being produced by actors that include government departments, consultancies, think tanks and even the tech industry itself. They appear to challenge academic research on AI in education as the location for evidence production, revise existing evidence standards, and alter the methodological apparatuses and practices for evaluating the impacts on AI in educational settings.

We can understand this as a struggle over “evidence governance,” where multiple different actors are competing to govern the methods, techniques and standards through which “policy-relevant” research about AI in education is generated.

Public-private evidence partnerships

One significant example of emerging approaches to evidence of AI in education is a collaboration between the English government departments responsible for science and education. They have partnered with the Education Endowment Foundation as their “evidence partner” to conduct “rapid evaluations in schools and colleges” as part of a national AI testbed initiative.

This initiative positions schools and classrooms as tech testing sites. It has the specific aim of identifying AI applications with the “highest promise” for being scaled up rapidly across the schools sector. In this case, rapid in-school AI testing is promoted as an ideal knowledge practice for generating policy-relevant evidence about the classroom impacts of AI.

Another example is the “Learning Outcomes Measurement Suite” launched by OpenAI in collaboration with research partners through its “Learning Lab”, the company’s own “learning research ecosystem.” As one of its research collaborators put it,

This research allows us to learn quickly while also laying the groundwork for a deeper understanding of how AI can be thoughtfully integrated into schools in ways that truly matter.

The measurement framework therefore positions OpenAI itself as an evidence producer with the international collaborations and research infrastructure to evaluate the effects and impacts of its own applications in a range of school settings.

The English AI testbed and the OpenAI learning measurement suite both involve public and private sector organizations operating symbiotically in new kinds of evidence partnerships. They both emphasize rapid evaluation or quick learning from technology trials. And they also both emphasize research evaluations and tests taking place in real-world school settings.

For example, in an early trial using the OpenAI measurement suite, the “setup was designed to reflect real world study conditions rather than a tightly controlled lab environment.” It has subsequently been scaled up for a major study with 20,000 students in Estonia as part of a national collaboration with the national ministry of education.

These examples indicate how both governments and AI industry are seeking to govern the standards and methods through which evidence about AI in education is produced. They also proceed from the assumption that the purpose of research and evidence is to support stronger integration of AI into educational practice.

Implementation R&D

A new report by the think tank Brookings gives a name to these kinds of evidence practices: “Implementation R&D.”

Titled “AI is rapidly changing education and research needs to keep up,” the brief report from Brookings starts from the claim that traditional technology impact evaluation methods such as randomized controlled trials “move slowly and are not equipped to evaluate AI education tools that change quickly.”

Instead, the Brookings authors argue, Implementation R&D “emphasizes rapid testing, feedback, and refinement.” They go on,

implementation research and development, or implementation R&D, is better suited to the evidence needs of AI technology. Traditionally used to study early-stage products and interventions, implementation R&D helps refine how a tool is designed, delivered, and used in real settings before a large-scale evaluation. It provides initial evidence on a product’s effects on teaching and learning. If the product doesn’t seem to work, implementation R&D helps us figure out why.

Notably, they don’t claim that implementation R&D replaces RCTs, but that it produces the foundational evidence for designing RCTs.

Implementation R&D is not a substitute for RCTs. It is the foundation on which to build research that proves cause and effect. It focuses on learning what works before testing whether it works at scale. When assessing the effectiveness of an AI-enabled professional learning tool, for example, rapid cycles of testing, feedback, and refinement take place, with a continued focus on how the tool affects core teaching and learning outcomes.

Along similar lines, the Education Endowment Foundation in England anticipates proceeding from rapid evaluation through the testbed to long-term RCTs afterwards. The testbed will function to identify applications that appear to “work” in limited real-world classroom settings before scaling up the experiment through larger RCTs involving multiple school testing sites.

As such, the current experiments resonate discursively with the “evidence-based education” movement, with its emphasis on RCTs (however problematic these may be as methods to measure educational interventions), but depart by focusing in the first instance on early-stage rapid evaluation through real-world implementation of AI in existing classroom arrangements. The logic here is implement first, then test, and only later move to a rigorous trial following all of the normal methodological and quality standards that have been practised as part of evidence-based education for decades.

Moreover, the Brookings report highlights cycles of testing, feedback and refinement. This means that AI application producers can benefit from the participation of students and teachers in the tests.

In the case of testbeds in England, edtech companies awarded prototype development funds by the government will, therefore, benefit additionally from pilot testing their early-stage products in schools, with teachers and students as implementers of the prototypes and providers of product feedback for subsequent refinement. With OpenAI’s measurement suite, the company itself stands to be the beneficiary of feedback on its Study Mode application of ChatGPT from use by thousands of students, with Estonia as the first national testing site.

Edtech-biz experimentalism

The Brookings report refers to these kinds of Implementation R&D projects as “design-focused experiments” that “operate on timelines that match how AI tools are developed” and “generate credible, early evidence” that can “guide decisions about adoption or scale”. It concludes,

If we want to ensure AI tools contribute to the improvement of teaching and learning, we need to study how those AI interventions are deployed in near-to-real time, using iterative experimentation to move from early design to long-term impact.

AI tools should be held to high standards. But to make those standards achievable, the education sector must move toward a new paradigm for research focused on producing iterative evidence and proof of impact on the outcomes that matter most.

These efforts to make schools into real-world testing sites for near-to-real-time iterative experimentation should be understood as exemplars of a growing trend in “experiments beyond the scientific laboratory.” Tech testbeds have been conceived as sites of “testing-in-the-wild” with certain sites understood as ideal or exceptional as settings in which t0 conduct technology experiments.

In a study of state-industry testbed trials of new government technologies, for example, Brice Laurent and colleagues refer to “tech-business experimentalism”: “a regime of innovation that explicitly refers to the laboratory and experimental practices in order to accompany entrepreneurial ventures.”

Along very similar lines, AI education testbeds are a form of edtech-business experimentalism that take place through turning schools into exceptional sites for entrepreneurial purposes, in addition to if not instead of their discursively stated educational purposes. This is particularly evident in the case of OpenAI, which has transformed the Estonian education system into a mass site for rapid testing, evaluation, feedback and product refinement as a route to greater market potential. The OpenAI Learning Lab is not only a scientific venture but an entrepreneurial enterprise dedicated to generating evidence to support further rollout through securing government agreements to boost AI adoption in schools.

Elsewhere, Google DeepMind has begun testing its Gemini-based chatbot tutor in schools in the UK too, first at limited scale and subsequently with a far larger school sample. This is enabled by a memorandum of understanding with the Department for Education that makes English schools into exceptional sites for AI tech testing. Clearly the intention is to generate evidence that can be considered “policy relevant” and therefore convincing to policymakers who might be able to authorize or enable wider-scale adoption.

Even where live AI testing is government-led, then, as in England, the emphasis is on producing evidence about selected commercial prototypes and early stage applications. Generating the relevant evidence to support the further scaling of these prototypes, as the testbed program is explicitly intended to do, therefore mixes edtech-biz experimentalism with a kind of gov-tech experimentalism where the social setting of the school becomes a test site in government-industry technology testing.

As the sociologist Noortje Marres has recently argued, testbeds represent a kind of “experimental innovation” in which “business, science and government are expected to work together to facilitate the delivery of intrinsically dynamic socio-technical services, and which seek to merge engineering methodology with forms of social organisation.”

So too do AI education testbeds and trials combine engineering methodology with new forms of educational organization. Through new kinds of business, science and government relations around AI in education, the prototype engineer and designer is elevated as an educational actor, authorized to experiment in the organization and practice of pedagogy and the experience of being a learner. At the same time, students and teachers become experimental test subjects in such state-industry-science experiments in how to reorganize schooling through AI.

Evidence futures

New evidence practices related to AI in education raise several issues and implications. First is that new actors including AI companies and think tanks are claiming authority over the kinds of evidence and standards that are taken to be relevant to informing policy and practice. OpenAI’s establishment of a “learning research ecosystem” is one prominent example although other big tech and AI companies have also launched initiatives claiming the authority from “learning science.”

The very nature of what is seen as valid evidence for policy and practice is up for grabs, seemingly throwing into question the evidence-based education evaluation methods and standards that have sedimented in the sector over the last couple of decades.

Second, this represents a struggle over evidence governance in which business interests are a key factor for many of the participating actors. Evidence is now taken as a source of feedback and actionable intelligence for companies. Where government departments are involved, such as in England, they also tend to privilege these business-oriented approaches, most notably by prioritizing the production of evidence for scaling.

For the think tankers at Brookings, conventional educational research methods and standards simply cannot keep pace with AI. As such, AI companies and their advocates are becoming knowledge governors that can control debates over what counts as appropriate evidence of Ai impacts, what methods of knowledge production are suited to measuring AI’s impacts, and who has the adequate resources and expertise to generate the scientific “facts” of AI in education.

A third issue, then, is that these practices involve the implementation of previously untested early-stage technologies in schools, as clearly signified by the Brookings emphasis on Implementation R&D as the ideal methodology for testing AI in real-world school settings. Students become subjects of implementation research and sources of feedback. For example, in Estonia students are treated by OpenAI as experimental subjects in an experiment that might influence how schools adopt AI worldwide, even though it is reported that “the student response has been mixed” owing to concerns among some about the detrimental impacts of AI.   

Fourth, these forms of implementation-based experimentation privilege engineering solutions to the social problems of schooling. AI and edtech companies are treated—by governments, consultancies and think tanks—as exceptional educational actors with the engineering expertise to implement, test and refine technologies for teaching and learning—tasks which also authorize them to redesign the very conditions of educating and being educated. Schools, meanwhile, are reduced to testing sites for the purposes of innovation R&D.

Finally, the production of evidence is now reimagined through an engineering lens as requiring revision to keep pace with AI itself. Because AI development moves at speed, it is argued by Brookings, new methods and evidence standards are required. This also makes AI appear as an exceptional kind of intervention in education that is exempt form the kinds of laborious evidence-based practices that have been deployed to generate evidence about other interventions.

These developments in evidence experimentation in AI evaluation in education are important to track and unpack as they are likely to become the epistemic basis on which future decisions about AI in schools will be made. Much of this experimentation is underpinned by clear commercial interests, and cannot be considered neutral or impartial.

It also demonstrates how specific kinds of design experiments such as AI prototype tests, implementation R&D, and evaluation measurements conducted by technology vendors themselves have been centred as authoritative evidence practices, challenging or replacing independent evaluation methodologies based on wider and multi-dimensional measures of educational impact.

All this experimentation in evidence generation coincides with growing public and student backlash over AI in education too. Huge asymmetry in who can shape the direction of AI in education is developing, with tech companies increasingly enabled by governments to produce evidence while educators, parents and students can only protest the rapid transformation of their schools and classrooms into sites of live AI experimentation.