Why We Reread Is the Puzzle AI Still Can’t Quite Solve

The old man the boat. Read at speed, the sentence looks broken. It is not: it means that old people are the ones manning the boat. But almost every reader first takes “man” as a noun, and the arrival of “the boat” forces a U-turn, the eyes skipping back to rebuild the sentence from the start.

Psycholinguists call such traps garden-path sentences, and the U-turn they trigger, a regression followed by rereading, has been on record since a 1982 eye-tracking study first showed readers stalling and doubling back at the exact word where an interpretation collapses. This month, two teams working independently published studies that bracket the phenomenon from opposite sides: one maps precisely where artificial intelligence fails to explain it, the other offers a different kind of AI that largely succeeds.

The idea under test is prediction. The language models behind chatbots learn by guessing the next word, and a leading theory of comprehension, surprisal theory, holds that a word’s difficulty is proportional to how unexpected it is in context. Mainstream models of eye movements in reading, such as the E-Z Reader model, already build word predictability into their timing, and decades of work treat the eye record as a moment-to-moment readout of comprehension itself. What no one had pinned down was how much of the full reading record, forward and backward, predictability can explain.

The PNAS team, from New York University and the University of Massachusetts Amherst, designed a direct test. They tracked the eyes of 368 adults at four universities reading six types of syntactically challenging sentences, garden-path constructions among them, mixed with naturalistic news and fiction. Then they asked 409 language models of six different architectures to estimate, word by word, how surprised a reader should be, and compared those estimates against the human record.

For the forward half of reading, the match was striking. When readers’ eyes moved straight ahead past the word that resolves a garden-path sentence, they lingered about 17 milliseconds longer than on the same word in an unambiguous control sentence; the best-fitting model, GPT-2, predicted 12 milliseconds. The same pattern held for other difficult constructions and for the word that followed. Prediction, it seems, explains the first pass, the routine work of recognizing a word and fitting it into a sentence that is going smoothly.

The backward half broke every model. Ambiguous sentences drove readers’ rate of regressions up by 92 to 102 percent over controls; the largest increase any model predicted was 21 percent, and most predicted far less. The rereading itself was worse: readers spent 313 to 584 milliseconds recovering after a regression, eleven to twenty times the best model prediction of 29 milliseconds. “20 percent of our eye movements when reading are backward, and AI models cannot explain when we decide to do that,” says Tal Linzen, an NYU associate professor of linguistics and data science and a co-author. “The predictability of a word really doesn’t even come close to explaining just how much time we spend on difficult words and garden-path sentences,” adds William Timkey, the linguistics doctoral student who led the study. Making the models bigger, more accurate, more human-like in memory, or more explicit about grammar changed nothing.

The PNAS authors conclude that predicting words is only one stage of reading, and that the cost of noticing a failed interpretation and rebuilding it needs a different kind of model. The Nature Human Behaviour study, led by Aalto University with partners in Hong Kong and Singapore, reads like an answer to that call. Its simulated reader treats reading as resource-rational control: getting the most understanding possible from a limited budget of time, memory, and eye-movement effort, attention spent the way a careful shopper spends money. The model works on three nested levels. At the word level it chooses which letters to fixate to pin down a word’s identity. At the sentence level it decides whether to move forward, skip, or regress, guided by two internal beliefs: how well the words read so far hang together, and how predictable the upcoming words are. At the text level it decides which sentence to read or reread while compressing what it has understood into gist. Each level was trained with reinforcement learning, the trial-and-error method used to teach robots, so skipping and rereading emerge from the budget rather than being programmed in. Prediction still does work here, but only as one input to a larger decision.

Checked against eight existing eye-tracking datasets and a new experiment of its own, in which 39 people read short texts under time limits of 30, 60, or 90 seconds, the model tracked human behavior across 15 measures with near-perfect correlation. Like people, it skipped short, frequent, predictable words, regressed more on ambiguous sentences (13 percent of trials versus 9 percent, close to the human 19 versus 7), and shifted strategy when rushed: sprinting forward and skipping more when time was short, looping back to repair understanding when time allowed. Its regressions were not noise but investments, targeted at the material most likely to restore comprehension.

The caveats cut both ways. Give the simulated reader unlimited memory and it turns superhuman, scoring higher on comprehension tests than people while barely rereading at all, which is exactly what a real reader does not do; the human-like behavior depends on the human-like limits. And the two teams never tested each other’s material: the reinforcement-learned reader has not yet faced the garden-path sentences that broke the language models, so whether it would produce the full human rereading cost is unknown. The Aalto team also describes its reader as a model of an average reader, not of any one person’s quirks.

Both groups are already looking at the same horizon. The NYU and UMass researchers frame their eye-tracking dataset, now public, as a benchmark for building models of rereading, with language learning and reading-related disorders as the prize. The Aalto-led team plans to test its model with readers who have dyslexia or low language proficiency, and imagines smart glasses that pace text to the situation or instant rewrites of dense documents for different readers. “We read all the time, yet throughout written history we have read texts that have been produced for mass use and not for an individual person and a specific situation,” says Aalto professor Antti Oulasvirta. “Now we are in a position to change that.”

The fairest test of that optimism is now specific and public. Put the budget-minded reader in front of “The old man the boat,” and see whether it trips where people trip, for as long as people do.

FAQ