AI Research Trials Are Booming, But the Old Playbook Won’t Work

EVERY randomized controlled trial, or RCT, in economics starts with the same small bureaucratic act: a researcher registers the study, months before anyone is enrolled, describing an intervention nobody outside the team has run yet. Registry records now show artificial intelligence built into close to one in five of those studies, up from one in thirty just four years earlier. The tools being tested keep changing faster than the process built to test them, and the standard randomized trial, designed for something that holds still, was never built for that. That gap, more than the raw growth number, is what the rest of this is about.

Money is pouring in to keep the growth going. Anthropic’s Economic Futures Research Fund alone has committed $200 million to large scale trials, and J-PAL, the Abdul Latif Jameel Poverty Action Lab, and other funders have launched calls of their own. None of that money guarantees the studies it buys will still mean anything by the time they’re finished.

The two registries behind these numbers, the American Economic Association’s RCT Registry and the Registry for International Development Impact Evaluations, are where economists log a trial’s design before results exist, and together they hold 10,214 research efforts, the raw dataset behind everything that follows. Total registrations climbed from a little under 1,000 in 2019 to around 1,650 by 2024, held roughly steady in 2025, and are on pace for a similar count in 2026, going by the roughly 900 filed by midyear. Of the full total, 739 are tagged as AI related, using descriptions mentioning machine learning, large language models, chatbots, computer vision, or named tools like GPT or Claude. That’s the one in five now, against the one in thirty from four years back.

The growth isn’t spread evenly. High income countries, home to just 17% of the world’s population, still register a disproportionate share of AI trials, though the low and middle income share is climbing at roughly the same steepness. Only 66.4% of AI related records even disclose a country, so this picture has real gaps in it.

Sorted by topic, AI trials cluster hard around behavior, education and labor, together over half of them, more concentrated than development economics research generally. Firms and labor studies are especially overrepresented next to Jessica Leight’s separately compiled sample of 1,497 development articles from 2021 to 2025. Where the money and attention concentrate says as much about what economists think AI is for as any of the results will.

The Trials Run Slower Than the Technology

The profession has been burned by an uncoordinated pile-up before. So many disconnected mobile health pilots hit Uganda at once that in 2012 the government simply banned new ones, a pattern researchers now call pilotosis. AI funding is arriving faster and in bigger checks than mobile health funding ever did, which is exactly the condition that produced pilotosis the first time.

The clearest evidence of the mismatch is timing. Across 741 AI related efforts with usable planned dates, the average trial runs 11.3 months from first enrollment to final measurement, and the wider category of randomized trials can take two years, or even longer, from planning through baseline to midline and endline; either span is long enough for the underlying model to be retired, renamed or quietly upgraded before a single result reaches a paper. Cost complicates it further: AI makes a digital intervention cheap to build, but if evaluation costs don’t fall at the same rate, the evidence budget eats a bigger share of every grant. There’s a subtler problem too. Which team built the tool may matter more than which tool got tested, since a skilled team’s mediocre model can outperform a clumsy team’s good one. And what does a control group even mean once the general purpose model everyone else is using keeps quietly improving too, a version of the same instability education researchers have separately flagged in AI classroom trials?

Government involvement, the thing most likely to get a pilot scaled rather than shelved, is lower for AI trials than for the registry as a whole: 10.7% against 14.1%, a gap of 3.4 percentage points. A trial nobody in government helped design is a trial nobody in government is likely to act on.

Not every AI trial is doomed to irrelevance. One eight week classroom study, run by Fab AI with Google DeepMind and Sierra Leone’s Ministry of Education, tested a guided learning tool on over 1,700 junior secondary math students and found a gain of 0.26 standard deviations, fast enough to beat its own subject matter’s shelf life. It’s the exception the authors point to, not yet the rule.

Fixing the Plumbing, Not Just the Method

The fixes on offer are mostly about plumbing rather than method: a shared tracker that captures midline results before the paper exists, a registry for pilots that quietly died before results ever came, and review boards mixing economists with technologists and policymakers rather than academics alone. Funders, in this telling, should also reward projects with local researchers and government co-design from the start, on the theory that scale follows relationships more reliably than it follows novelty. One proposal borrows from marketing science: test not just the best model available, but a cheaper Option C version closer to what a government could actually afford to run at scale. None of it will make a fast moving technology hold still for an evaluation, but it might stop so many separately funded trials from asking the same three questions in three different countries.

The registries will keep filling up regardless. The funding is committed, the interest is real, and the next wave of forms is already being filed somewhere this week. Whether any of it produces evidence a government still trusts by the time a slower reader gets around to the paper is the open question no registry can answer on its own.

FAQ