AI Meets the Classroom: When Do Large Language Models Harm Learning?
Adults / lifelong learning AI and learning
Links to claims
Supports (1)
| Claim | Basis |
|---|---|
| C2 | Full text (preprint): in two pre-registered lab experiments with university students learning Python, LLM access had no significant overall effect on an unaided post-test in the same session; when copy-paste made substitutive use easier, understanding per topic fell (p = .009) while topics covered rose (p = .033). Exploratory usage analyses; same-session test, not delayed. Consistent with C2's split between outsourcing and mere access. |
Review state: first-pass draft, pending human review.
Source note
Syntes: Lehmann, Cornelius & Sting (2024 preprint) — LLM access: no overall effect, but substituting practice trades understanding for coverage
Källa: Lehmann, M., Cornelius, P. B. & Sting, F. J. (2024). AI Meets the Classroom: When Do Large Language Models Harm Learning?. arXiv 2409.09047; v2 2025-03-08; preprint, not peer reviewed. https://doi.org/10.48550/arXiv.2409.09047
Verified content
- [full text] Two pre-registered, incentivised lab experiments: no significant effect of LLM access on overall learning outcomes.
- [full text] Study 2 means (correct answers): post-test 7.9 control vs 9.0 treatment; gain over pre-test 4.8 vs 5.4.
- [full text] Study 3 (copy-paste allowed) increased substitutive "solution" requests (+3.07 per subject); copy-paste did not change overall learning (p = .737) but lowered topic understanding (p = .009) and raised topic volume (p = .033): 1.7 more questions worked through, 1.4 fewer solved in the post-test holding progress constant. 42% of solution requests came without a single attempt.
- [full text] In the field study (graduate programming courses, only substitutive use observed) the authors report a significant long-term decline in overall learning outcomes; LLMs helped students with more prior knowledge and harmed those with less (authors' summary).
Limits
- [assessment] Preprint, not peer reviewed (applies if it is ever used in a ledger). The lab post-test came in the same 90-minute session, not after a delay. Usage-type analyses are exploratory (behaviour was not randomised, except copy-paste across studies run three weeks apart). University students learning Python.
Why it is in the library
C2 already separates outsourcing answers from mere access. This preprint is a clean mechanism test from outside education research: access alone had no overall effect, while substituting practice with generated solutions lowered understanding per topic. It also reports a prior-knowledge gap, an angle C2's Bounds do not yet cover.
Added 2026-11-15 by the project (weekly source sweep, kind: new (preprint) / adjacent (human-AI interaction); read: full text). [abstract] = from the published abstract; [full text] = read in the full text; [assessment] = the project's own assessment.
Metadata as recorded
- ID
- AI-048
- Tier
- experiment
- Titel
- AI Meets the Classroom: When Do Large Language Models Harm Learning?
- Författare
- Lehmann, M., Cornelius, P. B. & Sting, F. J.
- År
- 2024 (arXiv 2409.09047; v2 2025-03-08; preprint, not peer reviewed)
- Källa/DOI
- https://doi.org/10.48550/arXiv.2409.09047
- URL
- https://arxiv.org/abs/2409.09047
- Typ
- preprint: one field study (two graduate Python courses at a Dutch university, spring 2023, 56 + 57 students) and two pre-registered, incentivised lab experiments at a German university (Study 2, 108 subjects; Study 3, exact replication with copy-paste allowed, 69 subjects after exclusions) comparing learning to code with and without LLM access; post-test without the LLM in the same session
- Åldersgrupp
- vuxen-livslangt
- Teman
- ai-och-larande
- Hämtad
- 2026-11-15
- Paywall
- open (arXiv preprint)
- Tillagd
- source sweep 2026-11-15 (kind: new (preprint) / adjacent (human-AI interaction)); links pending human review
The original files (PDF or abstract) are kept in the project archive and are not published here.