Tutoring software that works through problems with the learner step by step, checking each step and giving hints or feedback along the way, comes close to one-to-one human tutoring in how much learners learn, compared with the same content without tutoring, in controlled experiments.
Status
alive
Last change
2026-11-09
Serves
Q2 (How does that work in practice: in school, with screens, and with AI as one of the tools?)
Ages
overgripande (reviews across school and higher education, mostly maths and science); grundskola-hog for the two recent AI-tutor studies (Oreopoulos et al. 2026; Oreopoulos & Low 2026). School-use ages moved to C10 with the split (2026-10-08).
Area
Teaching and feedback
Relates to
C5 relates-to C2 (C2 is about AI that hands over finished answers during practice; C5 is about software that guides each step without handing over the answer, and C2's carve-out for tutor modes that withhold answers points here. Thematic, not a dependency). C5 relates-to C4 (step-based tutors give task and process feedback at each step. Thematic, not a dependency). C5 relates-to C10 (C10 is the everyday-school-use claim split from C5 on 2026-10-08: C5 is about controlled experiments that hold content constant, with one-to-one human tutoring as the benchmark; C10 is about routine use as part of regular teaching for a term or longer, against regular teaching. Neither depends on the other: either can stand or fall alone).
Bounds
The comparison is step-based tutoring systems (feedback and hints at each step of a solution) against one-to-one human tutoring and against no tutoring on the same content. AI that gives finished answers is outside this claim (that is C2). Evidence in the project: VanLehn 2011 (review: step-based tutoring systems were nearly as effective as human tutors relative to no tutoring, d ≈ 0.76 versus d ≈ 0.79 per source note; answer-based computer tutoring did less well; gains level off beyond step-level interaction); Ma et al. 2014 (meta-analysis of 107 effects, N = 14,321; full text read: against individual human tutoring the tutoring systems were not significantly different, g = −0.11, SE 0.10, 95 % CI −0.31 to 0.10, from only 5 comparisons with 404 learners; against small-group human instruction g = 0.05, 4 comparisons, N = 184 (Table 1, p. 8); the lower end of that interval reaches the 0.3 SD kill threshold, so these five comparisons cannot rule out a shortfall of that size, and the paper does not say whether they were randomised or used step-based systems; corrected 2026-11-09 from "g ≈ −0.11, ns, per source note"). Most of this evidence predates generative AI and is weighted toward maths and science. Gains depend on the test: Kulik & Fletcher 2016 (meta-analysis of 50 evaluations against conventional teaching, not against human tutors) report a median effect of about 0.66 SD, but about 0.73 on local or researcher-made tests and about 0.13 on standardised tests (source note), so the size of the benefit on standardised tests is much smaller. Large-language-model tutors have not been shown to reach this level: a coaching AI (Khanmigo) gave small gains similar to the same practice without AI, and the median pupil used it on about a third of practice days (Oreopoulos & Low 2026); an AI tutor inside a mastery rule gave at most a borderline delayed gain over the same practice platform without AI: a week later, 40.2% against 37.0% correct on one practised question (p = .065, called marginally significant by the authors), not seen when practised and unpractised questions were compared, and no gain from the AI tutor without the mastery rule (Oreopoulos et al. 2026; working paper, one session of about 50 minutes, about 6,300 pupils tested; corrected 2026-10-21 from "a modest delayed gain"). Steenbergen-Hu & Cooper 2013 (K–12 maths meta-analysis, full text read) note that VanLehn 2011's review counted randomised experiments that held content constant (no Cognitive Tutor curriculum studies), mostly in STEM with many college students, and took the largest-effect outcome per study, so the near-human-tutor comparison rests on controlled experiments of that kind. Only one small study in their review compared a tutoring system with human tutors (small-group, not one-to-one; Beal et al. 2010, as reported by Steenbergen-Hu & Cooper). Kill-condition test survived (2026-10-10, nightly attack): Steenbergen-Hu & Cooper 2014 (Journal of Educational Psychology 106(2), 331–347; meta-analysis of 39 studies of tutoring systems in college courses; full text read) pooled the only three studies that compared a tutoring system directly with human tutoring: g = −0.25 (95 % CI −0.72 to 0.22, p = .302), not significant and below the 0.3 SD threshold, so the systems did not fall clearly short; the authors call it "largely consistent" with VanLehn 2011. The test is thin: three studies, a wide interval, all kinds of tutoring system pooled, college students only, and one of the three (VanLehn et al. 2007) is also in VanLehn 2011. Direct comparisons with human tutors therefore remain few. Split (2026-10-08, Jonatan approved): what happens when schools use such software as part of regular teaching for a term or longer is a separate claim, C10. The school-use bound this claim gained on 2026-10-07 (Steenbergen-Hu & Cooper 2013) and qualified on 2026-10-08 (Pane et al. 2014) moved to C10, together with those two ledger entries. The words "in controlled experiments", added on 2026-10-07, stay and mark the line between C5 and C10. Full text: VanLehn 2011, Kulik & Fletcher 2016 and Oreopoulos & Low 2026 are abstract-only in the project; an attack that relies on their details needs the full text first. Starved for attack (added 2026-10-27; method note, not a narrowing): thirteen consecutive fruitless searches. The kill-test needs a usable full text of a meta-analysis that compares step-based tutoring systems with one-to-one human tutoring (Xu et al. 2019 on papers-needed is the leading candidate). Until that opens, this claim stays alive but is not given a rule-3e widened first slot.
Does-not-transfer-to
AI tools that hand over finished answers or solutions (C2), or routine use as part of regular school teaching for a term or longer (C10, split from this claim on 2026-10-08 together with the two attacks on school use).
What would kill this
Large meta-analyses in which well-designed step-based tutoring systems fall clearly short of one-to-one human tutoring (by more than about 0.3 SD on comparable outcomes) in controlled comparisons. Progress is blocked mainly by missing full text of the best available candidates (especially Xu et al. 2019); until one opens, ordinary searches continue but rule-3e widening is paused. Bank wording (Swedish, verbatim), K28: “Storskaliga K–12-metas visar att designad step-based ITS systematiskt underpresterar human tutoring med >0,3 d och att answer-giving GenAI matchar ITS på closed-book utan guardrails.” The answer-giving GenAI half is dropped here because it belongs to C2 (Jonatan 2026-10-06). The second prong before the split (a replicated, preregistered meta-analysis with strict standardised outcomes in which the median effect of tutoring systems against ordinary teaching is about zero; K22) is about everyday school use and moved to C10 on 2026-10-08.
Tutor consequence
none yet
Attack record
1 attack: 1 survived
Split on 2026-10-08 (Jonatan approved at 06:50 Europe/Stockholm): this claim now covers controlled experiments only; everyday use in school is a new claim, C10. The two attacks made on this claim before the split, on 2026-10-07 (Steenbergen-Hu & Cooper 2013, DOI 10.1037/a0032447) and 2026-10-08 (Pane et al. 2014, DOI 10.3102/0162373713507480), both narrowed it, both were about everyday school use, and each verdict said it did not touch the controlled-experiment comparison. Both moved to C10 at the split and are kept and counted there, not here, so no attacks or survivals are counted on C5 yet. The words "in controlled experiments", added after the first of those attacks on 2026-10-07, stay; they mark the line between C5 and C10. For the nightly rotation, C5 counts as last targeted on 2026-10-08. Note from 27 October 2026 (not an attack, so it is not counted). After 13 nightly searches in a row found no study that could test it: C5 is kept as it is, waiting for a study that can test it, rather than softening what would kill it. The honest kill-test (systems clearly more than about 0.3 SD below one-to-one human tutoring in a meta of controlled comparisons) awaits usable full text — Xu et al. 2019 is the leading papers-needed candidate. Claim sentence unchanged; the limits gain a short note.
2026-10-10 survived
Attack
In a meta-analysis of intelligent tutoring systems in college courses, the studies that compared a tutoring system directly with human tutoring put the systems below human tutors (g = −0.25), and the authors conclude the systems "are not yet as effective as human tutors", so C5's "comes close to one-to-one human tutoring" fails its kill-condition.
Target
kill-condition
Source
Steenbergen-Hu & Cooper 2014 | DOI 10.1037/a0034752
Steelmans to
C5 dies on meta-analyses in which well-designed step-based tutoring systems fall clearly short of one-to-one human tutoring (by more than about 0.3 SD on comparable outcomes) in controlled comparisons; this is a meta-analysis that pools direct system-versus-human-tutor comparisons, its point estimate (−0.25) is close to that threshold, it ranks human tutoring above tutoring systems, and its authors read it as human tutors still being ahead.
Why
Nightly-attacks 2026-10-10 Europe/Stockholm. Full text read (Duke repository copy of the published article, Journal of Educational Psychology 106(2), 331–347; not stored). 35 reports, 39 studies, 22 types of tutoring system in higher education; overall g = 0.32–0.37 against all other comparison conditions. Table 2, human tutoring as comparison: k = 3, g = −0.25, SE 0.24, 95 % CI −0.72 to 0.22, p = .302 (same under fixed and random effects); the three studies are Reif & Scott 1999, Stankov, Glavinić & Grubišić 2004 and VanLehn et al. 2007 (Study 1). The authors stress that the result "was not statistically significant" and rests on only three studies, and they average VanLehn 2011's step-based (d = −0.21 after reversing direction) and substep-based (+0.12) comparisons to d = −0.05, concluding their result is "largely consistent" with VanLehn 2011. Kill fails: the point estimate is below the 0.3 SD threshold and not significant, so the systems do not fall clearly short. Underdetermined rather than confirming: three studies, a wide interval that also allows a gap of about 0.7 SD, all types of tutoring system pooled (not only step-based), college students only, and one of the three studies (VanLehn et al. 2007) is also counted in VanLehn 2011. Not a duplicate: first ledger entry on C5 since the split; Steenbergen-Hu & Cooper 2013 (K–12, now on C10) is a different meta-analysis and had no direct human-tutor comparison beyond one small-group study. Claim field unchanged; Bounds note the survive. Status remains alive.
Rewrite
none
Corrections to our notes
These fix a number or a fact in our notes. They are not attacks and do not count toward the floor of 3 survivals.
— C5's limits call the delayed effect of an AI tutor inside a mastery rule "a modest delayed gain", but the trial's own tables show only a borderline gain on one practised question that disappears in the practised-minus-unpractised comparison, and no AI effect without the mastery rule.
— C5's limits cited Ma et al. 2014's human-tutor comparison "per source note"; the full text gives its size, interval and the few studies it rests on.
Grades 7–9 preregistered RCT (Germany): GPT-4o with Socratic strategy prompts or utility-value prompts did not raise tested domain knowledge over a plain answer-withholding chatbot across four lessons (χ²(1) = 1.28, p = .257); consistent with C5's bound that LLM tutors have not been shown to reach step-based tutoring levels. Short, immediate test, no non-AI control.
Meta-analysis (107 effects, 14,321 learners): against individual human tutoring, 5 comparisons with 404 learners gave g = −0.11 (95% CI −0.31 to 0.10), not significant; whether those comparisons used step-based systems or randomised designs is not reported. Full text checked 2026-11-09.
College meta-analysis: in the three direct comparisons with human tutoring, tutoring systems g = −0.25 (95 % CI −0.72 to 0.22, ns), below the 0.3 SD kill threshold; thin evidence. C5 kill-condition test survived 2026-10-10.
Meta-analysis against conventional teaching (not human tutors): gains much smaller on standardised than on local or researcher-made tests (abstract only).
Coaching AI (Khanmigo): small gains similar to the same practice without AI, with limited use; not a step-based system, no human-tutor comparison (abstract-only in the project).
The authors note that VanLehn 2011's near-human-tutor result came from randomised experiments that held content constant, mostly STEM with many college students, using the largest-effect outcome per study; only one small study compared a tutoring system with (small-group) human tutors. Kept on C5 at the split on 2026-10-08 for this point only; the school-use result moved to C10.
Structured GPT-4 tutor (step-by-step scaffolding, pre-written solutions) vs in-class active learning, crossover RCT, Harvard physics: post-test effect 0.63 SD (regression; 0.73–1.3 quantile), immediate test only, not vs human tutoring. Points against C5's bound that LLM tutors have not been shown to reach this level; queued for the nightly run, not a verdict.
Review state: first-pass draft, pending human review.