C5 Step-based tutoring
alive 1 attack: 1 survived · serves Q2 · last change 9 November 2026
Tutoring software that works through problems with the learner step by step, checking each step and giving hints or feedback along the way, comes close to one-to-one human tutoring in how much learners learn, compared with the same content without tutoring, in controlled experiments.
In practice
In controlled experiments, tutoring software that works through problems step by step with the learner, checking each step and giving hints, has come close to a one-to-one human tutor. That is a result from trials with the same content; what to expect from everyday use in school is a separate belief (C10).
Go deeper
Where it holds
This claim is alive but starved for a fair kill-test: the best next study is blocked mainly by missing full text. See Bounds for the note.
Main limits
- The near-human-tutor result is from controlled experiments with the same content, compared with no tutoring; the main review counted randomised experiments, mostly in maths and science with many college students, and took the largest-effect outcome per study (as noted by Steenbergen-Hu & Cooper 2013).
- Gains are much smaller on standardised tests than on local or researcher-made tests (Kulik & Fletcher 2016).
- The evidence is mostly from before generative AI and mostly maths and science; AI chat tutors have not been shown to reach this level (Oreopoulos & Low 2026; Oreopoulos et al. 2026).
- Direct comparisons with human tutors are few: in a college meta-analysis, three studies put tutoring systems about 0.25 standard deviations below human tutors, not a significant gap (Steenbergen-Hu & Cooper 2014).
- Everyday use as part of regular school teaching is a separate belief (C10).
All limits (19)
Inside the circle: the core of the claim. Outside: every bound and carve-out in the claim file, numbered, with the attack that forced or added it where the record shows one.
forced by a narrowing attackin the original wordingwritten at creation, or added without a narrowing
Warm cards in the list were added by later attacks; plain cards were in the original draft or written at creation.
The comparison is step-based tutoring systems (feedback and hints at each step of a solution) against one-to-one human tutoring and against no tutoring on the same content.
Written when the claim was created (6 October 2026); not forced by an attack
AI that gives finished answers is outside this claim (that is C2).
Written when the claim was created (6 October 2026); not forced by an attack
Evidence in the project: VanLehn 2011 (review: step-based tutoring systems were nearly as effective as human tutors relative to no tutoring, d ≈ 0.76 versus d ≈ 0.79 per source note; answer-based computer tutoring did less well; gains level off beyond step-level interaction); Ma et al. 2014 (meta-analysis of 107 effects, N = 14,321; full text read: against individual human tutoring the tutoring systems were not significantly different, g = −0.11, SE 0.10, 95 % CI −0.31 to 0.10, from only 5 comparisons with 404 learners; against small-group human instruction g = 0.05, 4 comparisons, N = 184 (Table 1, p. 8); the lower end of that interval reaches the 0.3 SD kill threshold, so these five comparisons cannot rule out a shortfall of that size, and the paper does not say whether they were randomised or used step-based systems; corrected 2026-11-09 from "g ≈ −0.11, ns, per source note").
Written when the claim was created (6 October 2026); not forced by an attack
Wording changed later: Added with attack #3 (corrected; it did not narrow the claim) · Ma, Adesope, Nesbit & Liu 2014 · 9 November 2026.
Most of this evidence predates generative AI and is weighted toward maths and science.
Written when the claim was created (6 October 2026); not forced by an attack
Gains depend on the test: Kulik & Fletcher 2016 (meta-analysis of 50 evaluations against conventional teaching, not against human tutors) report a median effect of about 0.66 SD, but about 0.73 on local or researcher-made tests and about 0.13 on standardised tests (source note), so the size of the benefit on standardised tests is much smaller.
Written when the claim was created (6 October 2026); not forced by an attack
Large-language-model tutors have not been shown to reach this level: a coaching AI (Khanmigo) gave small gains similar to the same practice without AI, and the median pupil used it on about a third of practice days (Oreopoulos & Low 2026); an AI tutor inside a mastery rule gave at most a borderline delayed gain over the same practice platform without AI: a week later, 40.2% against 37.0% correct on one practised question (p = .065, called marginally significant by the authors), not seen when practised and unpractised questions were compared, and no gain from the AI tutor without the mastery rule (Oreopoulos et al. 2026; working paper, one session of about 50 minutes, about 6,300 pupils tested; corrected 2026-10-21 from "a modest delayed gain").
Written when the claim was created (6 October 2026); not forced by an attack
Wording changed later: Added with attack #2 (corrected; it did not narrow the claim) · Oreopoulos, Liut, Sungu & Low 2026; EdWorkingPaper 26-1552 (August 2026 version; working paper) · 21 October 2026.
Steenbergen-Hu & Cooper 2013 (K–12 maths meta-analysis, full text read) note that VanLehn 2011's review counted randomised experiments that held content constant (no Cognitive Tutor curriculum studies), mostly in STEM with many college students, and took the largest-effect outcome per study, so the near-human-tutor comparison rests on controlled experiments of that kind.
Added on 8 October 2026 without a matching attack in the record
Only one small study in their review compared a tutoring system with human tutors (small-group, not one-to-one; Beal et al. 2010, as reported by Steenbergen-Hu & Cooper).
Added on 7 October 2026 without a matching attack in the record
Kill-condition test survived (2026-10-10, nightly attack): Steenbergen-Hu & Cooper 2014 (Journal of Educational Psychology 106(2), 331–347; meta-analysis of 39 studies of tutoring systems in college courses; full text read) pooled the only three studies that compared a tutoring system directly with human tutoring: g = −0.25 (95 % CI −0.72 to 0.22, p = .302), not significant and below the 0.3 SD threshold, so the systems did not fall clearly short; the authors call it "largely consistent" with VanLehn 2011.
Added with attack #1 (survived; it did not narrow the claim) · Steenbergen-Hu & Cooper 2014 · 10 October 2026
The test is thin: three studies, a wide interval, all kinds of tutoring system pooled, college students only, and one of the three (VanLehn et al. 2007) is also in VanLehn 2011.
Added with attack #1 (survived; it did not narrow the claim) · Steenbergen-Hu & Cooper 2014 · 10 October 2026
Direct comparisons with human tutors therefore remain few.
Added with attack #1 (survived; it did not narrow the claim) · Steenbergen-Hu & Cooper 2014 · 10 October 2026
Split (2026-10-08, Jonatan approved): what happens when schools use such software as part of regular teaching for a term or longer is a separate claim, C10.
Added on 8 October 2026 without a matching attack in the record
The school-use bound this claim gained on 2026-10-07 (Steenbergen-Hu & Cooper 2013) and qualified on 2026-10-08 (Pane et al. 2014) moved to C10, together with those two ledger entries.
Added on 8 October 2026 without a matching attack in the record
The words "in controlled experiments", added on 2026-10-07, stay and mark the line between C5 and C10.
Added on 8 October 2026 without a matching attack in the record
Full text: VanLehn 2011, Kulik & Fletcher 2016 and Oreopoulos & Low 2026 are abstract-only in the project; an attack that relies on their details needs the full text first.
Written when the claim was created (6 October 2026); not forced by an attack
Starved for attack (added 2026-10-27; method note, not a narrowing): thirteen consecutive fruitless searches.
Added on 27 October 2026 when the claim was reworded (not an attack)
The kill-test needs a usable full text of a meta-analysis that compares step-based tutoring systems with one-to-one human tutoring (Xu et al. 2019 on papers-needed is the leading candidate).
Added on 27 October 2026 when the claim was reworded (not an attack)
Until that opens, this claim stays alive but is not given a rule-3e widened first slot.
Added on 27 October 2026 when the claim was reworded (not an attack)
Does not transfer to: AI tools that hand over finished answers or solutions (C2), or routine use as part of regular school teaching for a term or longer (C10, split from this claim on 2026-10-08 together with the two attacks on school use).
Written when the claim was created (6 October 2026); not forced by an attack
Wording changed later: Added on 8 October 2026 without a matching attack in the record.
0 of 19 bounds and carve-outs were forced by a specific attack. 1 attack hit the boundary and did not move it: #1.
How it changed
The ledger records 1 attack (and 2 corrections to our notes), but no earlier wording of C5 is on record, so the text cannot be replayed step by step. Only the current wording is shown.
Tutoring software that works through problems with the learner step by step, checking each step and giving hints or feedback along the way, comes close to one-to-one human tutoring in how much learners learn, compared with the same content without tutoring, in controlled experiments.
What would change this
What would kill it: Large meta-analyses in which well-designed step-based tutoring systems fall clearly short of one-to-one human tutoring (by more than about 0.3 SD on comparable outcomes) in controlled comparisons. Progress is blocked mainly by missing full text of the best available candidates (especially Xu et al. 2019); until one opens, ordinary searches continue but rule-3e widening is paused. Bank wording (Swedish, verbatim), K28: “Storskaliga K–12-metas visar att designad step-based ITS systematiskt underpresterar human tutoring med >0,3 d och att answer-giving GenAI matchar ITS på closed-book utan guardrails.” The answer-giving GenAI half is dropped here because it belongs to C2 (Jonatan 2026-10-06). The second prong before the split (a replicated, preregistered meta-analysis with strict standardised outcomes in which the median effect of tutoring systems against ordinary teaching is about zero; K22) is about everyday school use and moved to C10 on 2026-10-08.
Attack record
Split on 2026-10-08 (Jonatan approved at 06:50 Europe/Stockholm): this claim now covers controlled experiments only; everyday use in school is a new claim, C10. The two attacks made on this claim before the split, on 2026-10-07 (Steenbergen-Hu & Cooper 2013, DOI 10.1037/a0032447) and 2026-10-08 (Pane et al. 2014, DOI 10.3102/0162373713507480), both narrowed it, both were about everyday school use, and each verdict said it did not touch the controlled-experiment comparison. Both moved to C10 at the split and are kept and counted there, not here, so no attacks or survivals are counted on C5 yet. The words "in controlled experiments", added after the first of those attacks on 2026-10-07, stay; they mark the line between C5 and C10. For the nightly rotation, C5 counts as last targeted on 2026-10-08. Note from 27 October 2026 (not an attack, so it is not counted). After 13 nightly searches in a row found no study that could test it: C5 is kept as it is, waiting for a study that can test it, rather than softening what would kill it. The honest kill-test (systems clearly more than about 0.3 SD below one-to-one human tutoring in a meta of controlled comparisons) awaits usable full text — Xu et al. 2019 is the leading papers-needed candidate. Claim sentence unchanged; the limits gain a short note.
1 attack: 1 survived. Counts of attacks, not a score.
Full attack list (1)
In a meta-analysis of intelligent tutoring systems in college courses, the studies that compared a tutoring system directly with human tutoring put the systems below human tutors (g = −0.25), and the authors conclude the systems "are not yet as effective as human tutors", so C5's "comes close to one-to-one human tutoring" fails its kill-condition.
Strongest reading and verdict reasoning
Steelmans to: C5 dies on meta-analyses in which well-designed step-based tutoring systems fall clearly short of one-to-one human tutoring (by more than about 0.3 SD on comparable outcomes) in controlled comparisons; this is a meta-analysis that pools direct system-versus-human-tutor comparisons, its point estimate (−0.25) is close to that threshold, it ranks human tutoring above tutoring systems, and its authors read it as human tutors still being ahead.
Why: Nightly-attacks 2026-10-10 Europe/Stockholm. Full text read (Duke repository copy of the published article, Journal of Educational Psychology 106(2), 331–347; not stored). 35 reports, 39 studies, 22 types of tutoring system in higher education; overall g = 0.32–0.37 against all other comparison conditions. Table 2, human tutoring as comparison: k = 3, g = −0.25, SE 0.24, 95 % CI −0.72 to 0.22, p = .302 (same under fixed and random effects); the three studies are Reif & Scott 1999, Stankov, Glavinić & Grubišić 2004 and VanLehn et al. 2007 (Study 1). The authors stress that the result "was not statistically significant" and rests on only three studies, and they average VanLehn 2011's step-based (d = −0.21 after reversing direction) and substep-based (+0.12) comparisons to d = −0.05, concluding their result is "largely consistent" with VanLehn 2011. Kill fails: the point estimate is below the 0.3 SD threshold and not significant, so the systems do not fall clearly short. Underdetermined rather than confirming: three studies, a wide interval that also allows a gap of about 0.7 SD, all types of tutoring system pooled (not only step-based), college students only, and one of the three studies (VanLehn et al. 2007) is also counted in VanLehn 2011. Not a duplicate: first ledger entry on C5 since the split; Steenbergen-Hu & Cooper 2013 (K–12, now on C10) is a different meta-analysis and had no direct human-tutor comparison beyond one small-group study. Claim field unchanged; Bounds note the survive. Status remains alive.
Corrections to our notes
These fix a number or a fact in our notes. They are not attacks and do not count toward the floor of 3 survivals.
- — C5's limits call the delayed effect of an AI tutor inside a mastery rule "a modest delayed gain", but the trial's own tables show only a borderline gain on one practised question that disappears in the practised-minus-unpractised comparison, and no AI effect without the mastery rule.
Details
What was checked: fact in the Bounds (description of Oreopoulos et al. 2026), not the claim text or the kill-condition.
Why: Nightly-attacks 2026-10-21 Europe/Stockholm, widened-search slot for C5 (a lead from the study-reading notes, re-checked tonight in the stored full text). Table 9 / Table A2 (mastery students, N = 3,209): practised Exercise 1, CAL-only 0.370 vs AI 0.402, difference 0.032 (SE 0.017), p = 0.065 (authors: "marginally significant", "should be interpreted as suggestive"); unpractised Exercise 1 difference 0.002; practised-minus-unpractised 0.026 (SE 0.034). Table 5 (full 2 × 2, N = 6,327): AI main effect on the combined practised-minus-unpractised outcome −0.014 (SE 0.034); Mastery × AI 0.040 (SE 0.048) combined, 0.085 (SE 0.039, p < .05) on the practised total. Comparison condition: the same platform with worked solutions after mistakes, not a human tutor and not no tutoring. Scope, direction and kill-condition of C5 unchanged (the sentence already said LLM tutors have not been shown to reach the human-tutor level), so this is a correction, not a narrowing; not a trial, does not count toward the floor and does not reset the fruitless streak. Status stays alive; claim text and brief unchanged (the brief does not describe this study). The working paper is used only in Bounds, not as a ledger trial, so no preprint recheck item.
- — C5's limits cited Ma et al. 2014's human-tutor comparison "per source note"; the full text gives its size, interval and the few studies it rests on.
Details
What was checked: precision of the Ma et al. 2014 Bounds line (human-tutor comparison); not a kill-condition test (the point estimate, −0.11, sits well inside the 0.3 SD threshold and is not significant, so the kill line, "fall clearly short by more than about 0.3 SD", is not met).
Why: Nightly-attacks 2026-11-09 Europe/Stockholm. Table 1, p. 8 (random-effects model): individual human instruction N = 404, k = 5, g = −0.11, SE 0.10, 95 % CI −0.31 to 0.10 (fixed effects identical); small-group human instruction N = 184, k = 4, g = 0.05, SE 0.28, 95 % CI −0.50 to 0.61; overall g = 0.41 (k = 107, N = 14,321). Text, p. 7: "Individual human instruction (i.e., human tutoring) appeared to offer a small, non-significant advantage over the use of ITS." The paper does not report whether those five comparisons were randomised or used step-based systems. Consistent with the 2026-10-10 survive (Steenbergen-Hu & Cooper 2014, g = −0.25 ns, k = 3). Claim text, Status and kill-condition unchanged. Not a survival; does not count toward the floor; does not reset C5's fruitless streak. C5 returns from rest under the targeting rules(a) on this verified full-text 3g flag (C5 stays carved out of the widened-search rule).
Evidence
9 linked sources: 4 supports, 4 limits, 1 challenges (first-pass draft, pending human review). supports: Ma et al. 2014, VanLehn 2011, Steenbergen-Hu & Cooper 2014, Fütterer et al. 2026 · limits: Kulik & Fletcher 2016, Oreopoulos et al. 2026, Oreopoulos & Low 2026, Steenbergen-Hu & Cooper 2013 · challenges: Kestin et al. 2025