In three short experiments, people who used an AI assistant during practice answered fewer later questions correctly without it than people who practiced unaided. The paper was presented at COLM 2026; because each solo test followed only 10 to 15 minutes of AI-assisted practice, the results do not show whether any effect lasts.
The researchers recruited 1,222 people in the United States through Prolific for trials in fraction solving and reading comprehension. After preset exclusions, 1,060 remained across the three trials. In the first two, participants could use an AI assistant beside fraction problems; researchers then removed it and asked them to solve three more on their own.
In the first trial, the AI group solved 57% of the final questions correctly, compared with 73% in the no-AI group. They skipped 20%, versus 11% for controls. The researchers' statistical tests found both gaps unlikely to be sampling variation alone (p<.001 for correct answers; p=.031 for skips). In the second trial, a pretest and a matching sidebar for the control group reduced differences in initial skill and interface. The AI group's unaided score was 71% versus 77%; the smaller gap still passed the study's test (p=.020). Its skip rate was 10% versus 7%, a difference the test did not establish (p=.239). A statistical test gauges whether a measured gap could plausibly arise from sampling variation; it does not say whether that gap matters in daily life.
A third trial used SAT-style reading questions. After the AI assistant was removed, participants answered 76% correctly versus 89% in the control group, and skipped 8% versus 1%. Both gaps passed the researchers' statistical tests (p=.007 for correct answers; p=.008 for skips). These were immediate results on a small set of test questions, not a measure of learning over time.
Skipping was less consistent than scores: in the second fractions trial, the skip-rate difference was not statistically significant. Across all three studies, tests came immediately after assistance ended. The design therefore cannot tell whether everyday AI use changes learning over weeks or months, whether the effect fades, or whether it applies to other users and tasks. Participants also used an assistant configured to give complete answers on demand, so the study does not establish what would happen with other forms of tutoring.
The authors argue that AI systems should support long-term competence as well as complete the immediate task, and identify a more Socratic assistant as a question for future research. Their experiments did not compare that design with answer-giving.
Image: Notebook and computer by Mossytravels05, licensed under CC BY 4.0.
