Experimental Evidence on the Learning Impact of Generative AI
Adults / lifelong learning AI and learning Foundations of optimal learning
Links to claims
Limits (1)
| Claim | Basis |
|---|---|
| C2 | Off-the-shelf GenAI access: delayed unaided knowledge tests +0.21 SD at ~1 week in v2 (p = .068; v1 said +0.27 SD), no loss; scopes C2 to outsourcing / unprotected answer-giving (weakened 2026-10-03). |
Challenges (1)
| Claim | Basis |
|---|---|
| C2 | Overall ITT positive for typical undergraduate GenAI access — kill of typical-GenAI-lowers-retention prong. |
Review state: first-pass draft, pending human review.
Source note
Syntes: Contractor & Reyes (2026) — off-the-shelf GenAI can raise delayed unaided learning
Källa: Contractor, Z., & Reyes, G. (2026). Experimental Evidence on the Learning Impact of Generative AI. arXiv:2607.08849. https://doi.org/10.48550/arXiv.2607.08849
Verified numbers (full arXiv PDF)
- Design: Undergraduates learn unfamiliar topic + write analytical essay with or without off-the-shelf GenAI; unaided knowledge tests and essays immediately and ~1 week later (mean gap 6.99 days). N=211 Session One; 204 both sessions.
- First stage: AI-allowed raised ChatGPT use by 67.3 pp.
- Session One test ITT: +6.7 pp / +0.27 SD (p=.034). TOT +0.40 SD.
- Session Two (no AI) test ITT: +5.1 pp / +0.27 SD (p=.027); ~76% of immediate effect persists.
- Use heterogeneity: Augmentation users (explain concepts) retain gains; automation users (draft finished text) show Session One essay-quality gains that fade when AI is removed; automation Session Two test +0.19 SD (ns).
- Mechanisms: less drafting time, more reading/search; higher enjoyment.
Role for C2
Kill-condition prong on "typical GenAI use does not lower retention" is met for overall off-the-shelf access (learning rises). Does not erase Bastani unprotected answer-giving harm or Strömberg outsourcing pattern. Forces scope: living claim is about outsourcing finished answers / unprotected answer-giving, not mere access to an answer-capable chatbot used mainly for explanation. - v2 correction (read 2026-10-16, arXiv v2 posted 2026-10-04): Session One ITT +7.2 pp / 0.28 SD (p = .021); Session Two (no AI, ~1 week) ITT +3.9 pp / 0.21 SD (p = .068); augmentation users 0.24 SD (p = .095), mixed 0.28 SD (p = .220). C2 claim text corrected 2026-10-16.
Metadata as recorded
- ID
- LEARN-017
- Tier
- RCT
- Closed-book follow-up
- ja (Session Two ~1 week later: unaided knowledge test and essay without AI)
- Titel
- Experimental Evidence on the Learning Impact of Generative AI
- Författare
- Contractor, Z.; Reyes, G.
- År
- 2026 (arXiv:2607.08849; also IZA Discussion Paper No. 18792; dated August 24, 2026)
- Källa/DOI
- https://doi.org/10.48550/arXiv.2607.08849
- URL
- https://arxiv.org/pdf/2607.08849.pdf
- Typ
- two-session HE RCT (n=211 Session One; n=204 both); AI-allowed vs AI-forbidden during learning+essay; unaided tests immediately and ~1 week later
- Åldersgrupp
- vuxen-livslangt (US undergraduates, Middlebury)
- Teman
- ai-och-larande;optimalt-larande-grunder
- Hämtad
- 2026-10-03
- Paywall
- OA (arXiv)
The original files (PDF or abstract) are kept in the project archive and are not published here.