Lose the AI Helper and Many People Stop Trying

A SIDEBAR assistant, pre-loaded with every problem and its solution, would hand over the answer to a fraction question if a participant simply typed a word asking for it. Plenty did, right up until the sidebar vanished without warning.

In three randomized trials with 1,222 online participants, people who had leaned on a chatbot and then lost it solved fewer problems correctly than people who never had one, and in two of the three trials they also skipped more problems outright. Those gaps are real, but they come wrapped in fine print.

Brian Christian, a research fellow at the Center for Human-Compatible AI at the University of California, Berkeley, is one of five authors; the others hold posts at Carnegie Mellon University, the University of Oxford, the Massachusetts Institute of Technology and the University of California, Los Angeles. Their paper passed peer review at the Conference on Language Modeling, which published it this year.

In the first trial, 354 US-based participants recruited through the online platform Prolific (which fared well against rival platforms in an earlier comparison) worked through 15 fraction problems that grew steadily harder, from one-step sums to three-step chains. Of those, 191 were randomly given the assistant for the first 12 problems and 163 never were; the sidebar then disappeared for the last 3, and everyone was asked to finish on their own. Wrong answers cost nothing, and a skip button let anyone pass on a problem, a choice the researchers read as a measure of motivation rather than ability. After exclusions left 307 participants, those who had been helped solved 57% of the 3 test problems against 73% for controls, and skipped 20% against 11%.

A Wrinkle in the First Trial

There was a catch, and the authors flagged it themselves. Among the 47 exclusions were people who could not solve at least 3 of the 12 practice problems; but with a chatbot supplying answers, weak solvers could clear that bar anyway, so 41 of 163 controls were cut against only 6 of 191 helped participants. That lopsided attrition may have left a weaker AI group behind and padded the gap.

So the team ran it again with 667 participants, screening people with an easy pretest before the assistant ever appeared, and handing controls a sidebar of already-seen solutions so that both groups watched something disappear. Exclusion rates now matched (11.7% for the AI group, 12.9% for controls), which suggests the mismatch had eased. With 585 participants left, the gap shrank: 71% of test problems solved against 77%, 6 points, still statistically significant. Skipping, at 10% against 7%, was not.

The final trial swapped fractions for reading comprehension, using free practice questions for a US college admissions test. Of 201 participants recruited, 168 remained; they got 5 passages with the assistant and then 3 without. The helped group solved 76% of the test questions against 89% for controls, and skipped 8% against 1% (an answer entered in under 5 seconds counted as a skip, since nobody can read a passage that fast).

Christian, who keeps a notebook on his desk and writes in it every day, says the result matches a hunch he is far from alone in having. “It’s a striking finding, but in a way it supplies ammunition for a story that a lot of us kind of feel in our gut,” he says. “I experience it too.”

The People Who Just Asked for Answers

Why did skipping show up in some trials and not others? Afterward, all 308 AI participants in the second trial were asked how they had used the assistant. Of them, 189 (61%) said mostly to get answers directly, 82 (27%) said hints or clarifications, and 37 (12%) said they had not used it.

The direct-answer group solved 65% of the test problems and skipped 13%. Hint users solved 76%, level with the 77% of controls, and skipped just 5%; the 37 who barely touched the assistant solved 89%. Direct-answer users also finished 10 points below their own pretest solve rate, while controls ticked up 1 point. None of this was randomized, though: people chose how to use the assistant, so the comparison shows who struggled afterward, not what a style of use does to anyone.

How Far These Results Travel

The ceiling is low, and the authors say so. Everyone was tested immediately after the assistant was removed, so whether the drop fades in an hour or lingers for days is unknown; the exposure ran about 13 to 15 minutes (the abstract rounds it to about 10), not months of daily use; and the assistant was maximally helpful, with no tutor-style version in the mix. Add US-based online participants doing low-stakes puzzles, and the safest reading is a short-term effect on a narrow slice of tasks.

Why would a session that short do anything at all? The authors propose two ideas, neither tested here. An assistant that answers in seconds may reset expectations of how long a task should take, so unaided effort starts to feel slow. It may also strip out what the team calls productive struggle, the slog that teaches people what they can actually do, an old theme in research on desirable difficulties in learning.

Christian wants assistants that default to being more instructive, like a tutor, rather than to quick answers; in a separate field experiment with high school math students, a tutor-style chatbot largely avoided the later drop in scores seen with a standard chatbot.

“We need to work toward a future in which human abilities are, as much as possible, augmented rather than supplanted,” he says. Whether a hint-giving assistant really protects persistence remains open; the hint users here matched controls, but they picked their own style of use, and the authors call that pattern only a tentative pointer.

FAQ