Syntes: Kestin et al. (2025) — a structured GPT-4 tutor beat an in-class active-learning lesson on an immediate post-test

Källa: Kestin, G.; Miller, K.; Klales, A.; Milbourne, T.; Ponti, G. (2025 (Scientific Reports 15, 17458)). AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting. https://doi.org/10.1038/s41598-025-97652-6

Verified numbers (full text read 2026-10-11, nature.com)

• Setting: Harvard, Physical Sciences 2 (introductory physics for the life sciences), Fall 2023; 233 enrolled, 194 eligible (consent, both conditions, all tests).
• Design: crossover; groups randomised (peer-instruction groups of 2–3 kept together). Week 1 group 1 had the AI lesson at home and group 2 the active-learning class; reversed in week 2. Same worksheet and activity introductions in both conditions (live versus pre-recorded video).
• Tutor: GPT-4 with expert-written, question-specific system prompts and pre-written step-by-step solutions in the prompt; the platform guided students sequentially through each part of each problem (the authors found a system prompt alone could not keep multi-part problems in order).
• Results: median post-test 4.5 (AI, N = 142) vs 3.5 (in-class, N = 174), baseline median 2.75; Mann–Whitney z = −5.6, p < 10⁻⁸. Regression effect size 0.63 (authors: underestimated due to ceiling); quantile regression 0.73–1.3 SD. Median AI time on task 49 min vs 60 min in class; no correlation between time on task and post-test. Engagement 4.1 vs 3.6 and motivation 3.4 vs 3.1 (5-point scale), enjoyment and growth mindset n.s.
• Limits (authors and design): post-test immediately after each lesson (no delayed test); one lesson per condition; comparison is an active-learning class, not one-to-one human tutoring and not no-instruction; authors overlap with the developers of the in-class pedagogy; the authors "do not presume that structured AI tutoring will always outperform in-class active learning in all contexts".

Why it is in the library

The most-cited randomised evidence that a carefully structured LLM tutor (step-by-step scaffolding through each part of a problem, expert-written prompts, solutions supplied to the model) can produce large short-term learning gains. It bears on C5's bound that large-language-model tutors "have not been shown to reach this level" and on the C2 boundary between answer-giving and scaffolded AI use. Its weak points for any verdict: immediate test only, one course, comparison is not human tutoring.