C10 Tutoring software in school use

survived narrowedone dot per attack, oldest first

1. #1 narrowed 7 October 2026 · Steenbergen-Hu & Cooper 2013 · target: claim-text

   Moved: from C5 at the split on 2026-10-08. Made against C5's school-use clause, which became C10, so it names C5; the wording before and after this attack (Before/After) was C5's. Counted here, not on C5.

   In K–12 mathematics classrooms, where intelligent tutoring systems (mostly Cognitive Tutor) took over part of regular teaching for a term or a school year, their average effect on maths learning compared with regular classroom instruction was close to zero (g = 0.01–0.09; 0.02 on standardised tests; 0.01 at the end of the school year), so C5's near-human-tutor gain does not describe routine school use and the second prong of its kill-condition is approached.

   Strongest reading and verdict reasoning

   Steelmans to: Kill-condition prong 2 asks for a meta-analysis with standardised outcomes in which the median effect of tutoring systems against ordinary teaching is about zero; this K–12 meta-analysis (26 reports, 34 samples) finds g = 0.02 on standardised tests and 0.01 pretest-adjusted overall, far below VanLehn's d ≈ 0.76 and below what one-to-one human tutoring achieves, so the unconditioned claim overstates what pupils gain when schools use step-based tutors.

   Why: Nightly-attacks 2026-10-07 Europe/Stockholm. Full text read (Duke repository copy; not stored). Overall vs regular classroom: unadjusted g = 0.05 fixed (95 % CI .02–.09, p = .005), 0.09 random (CI −.03 to .20, p = .136); adjusted g = 0.01 under both models. Course-related measures g = 0.19 vs standardised tests g = 0.02 (CI −.02 to .06) (fixed; difference ns under random). End of school year g = .01 vs earlier g = .25 (fixed; ns under random). Robust moderators: effects larger for interventions shorter than a school year and for general students than low achievers. Kill withheld: (1) not preregistered and not restricted to strict standardised outcomes, so prong 2 is approached, not met; (2) the authors attribute the gap to VanLehn 2011 to inclusion rules (VanLehn: randomised, same content, no Cognitive Tutor curriculum studies, many college students, largest-effect outcome), so the controlled-experiment comparison that the claim rests on (VanLehn 2011; Ma et al. 2014) is not contradicted; (3) many included studies compared whole curricula in which the software was not the only difference; (4) only one small study compared a tutoring system with (small-group) human tutors. Not a restatement of the Kulik & Fletcher 2016 bound: that meta-analysis still reports a median ~0.66 SD against conventional teaching (0.13 on standardised tests); this one finds about zero for routine K–12 maths use. First ledger entry on C5. Claim and Bounds rewritten same turn; brief and the practical rule rewritten.

2. #2 narrowed 8 October 2026 · Pane et al. 2014 · target: bound

   Moved: from C5 at the split on 2026-10-08. Made against C5's school-use clause, which became C10, so it names C5; the wording before and after this attack (Before/After) was C5's. Counted here, not on C5.

   In the largest randomised trial of a step-based tutor in routine school use (Cognitive Tutor Algebra I, schools in seven US states, implemented as schools normally would), high-school students gained about 0.2 standard deviations on a standardised algebra test in the second year of use, so C5's school-use clause that the average gain in school maths was about zero on standardised tests and at the end of the school year overstates the null.

   Strongest reading and verdict reasoning

   Steelmans to: the 2026-10-07 school-use bound rests on a meta-analysis of studies from 1997–2010, most not randomised at school level; Pane et al. randomised matched pairs of schools, measured end-of-year standardised algebra proficiency, and found a significant second-year high-school gain (+0.22 in the preferred model; +0.21 in the authors' stricter re-analysis), so the routine-use picture is "no gain in the first year, a modest gain once schools have used it for a year", not "about zero".

   Why: Nightly-attacks 2026-10-08 Europe/Stockholm. Full text read = the authors' RAND working paper WR-1050 (Addendum, May 2014), which reproduces the published treatment estimates in its appendix (Tables A.1, A.2) and re-analyses them; the EEPA article (36(2), 127–144) is closed and only its abstract was read (ERIC EJ1024233). Published estimates, high schools: year 1 −0.19 (p = .10), −0.12 (p = .24), −0.10 (p = .34); year 2 +0.14 (p = .24), +0.19 (p = .05), +0.22 (p = .03). Middle schools: year 1 −0.20, 0.00, −0.03; year 2 −0.07, +0.17, +0.19 (all ns). Addendum re-analysis (actual pretests, late joiners excluded; 11,066 high-school and 5,519 middle-school pupils): high-school year 2 +0.21 (SE 0.10, p = .04); middle-school year 2 +0.10 (ns). Kill-condition prong 2 (median effect about zero against ordinary teaching) is therefore not met by routine use over two years. This does not touch the claim's main comparison (near human tutoring in controlled experiments): 0.2 SD after a year of adoption is far below the near-human-tutor size, so "has not been shown for routine use in school mathematics" stands. Not a restatement or duplicate: the 2026-10-07 entry added the near-zero school-use bound from Steenbergen-Hu & Cooper 2013; this entry qualifies that bound with a later randomised trial. Caveats: a blended curriculum, not software alone; first-year estimates negative though not significant; differential attrition about 5 %; middle-school pretest imbalance. Claim and Bounds rewritten same turn; brief and the practical rule rewritten.

3. #3 narrowed 10 October 2026 · Pane et al. 2010 · target: claim-text

   In a randomised trial in eight high schools of one US district, pupils taught with a Cognitive Tutor geometry course (40 % of class time on the step-based tutoring software) for a school year scored about 0.19 standard deviations lower on the district's geometry final exam than pupils taught the district's usual course, so C10's statement that routine use raises results compared with regular teaching does not hold in every setting: routine use can lower them.

   Strongest reading and verdict reasoning

   Steelmans to: C10 says routine use "raises results... but only by a small amount and not consistently"; its evidence so far is an average near zero and a trial with no first-year gain and a second-year gain, none of which shows a loss. Pane et al. randomised pupils within schools, class periods and teachers (each teacher taught both courses), used an independent, mandatory district exam that the developer had reviewed, and found a significant loss that held across model specifications and in the randomisation test and did not shrink as teachers gained experience; the kill-condition itself names "such use lowers results compared with regular teaching" as a way the claim dies.

   Why: Nightly-attacks 2026-10-10 Europe/Stockholm. Full text read (authors' RAND manuscript of the published article, Journal of Research on Educational Effectiveness 3(3), 254–281; publisher page closed; not stored). Baltimore County Public Schools, 2005–06 to 2007–08, 8 high schools, 19 teachers, 60 classrooms; 950 eligible randomised pupils, 699 in the final analysis (attrition 27.0 % treatment vs 25.8 % control; no significant baseline differences). Table 3: geometry posttest −0.19 SD (SE 0.08, p = .03, 95 % CI −0.34 to −0.04); multiple-choice −0.23 (p < .01), open-ended −0.16 (p < .10); items rated "thoroughly covered" by the developer −0.14 and "thoroughly or moderately covered" −0.08 (both ns, low power); attitudes −0.07 to −0.09 (ns). The authors: "we find a significant and substantial negative effect for the curriculum in our sample"; teachers implemented the materials and software time faithfully but the classroom pedagogy poorly. Kill withheld: (1) a single trial in one district with weak baseline skills, and the kill-condition asks for a preregistered meta-analysis or replicated school-randomised trials; (2) a whole course (software plus teacher-led lessons) against the usual course, so the software's own effect cannot be separated; (3) the raw gap was 2.2 points on a 50-point exam on which both groups scored low; (4) Pane et al. 2014 (seven states, cluster-randomised) found a second-year high-school gain of about 0.2 SD. Not a duplicate: this trial is one of the studies pooled in Steenbergen-Hu & Cooper 2013 (their single geometry effect size, g = −0.19, dropped from their adjusted overall estimate), but that entry's force was the near-zero average; this entry's force is that routine use can lower results, which no ledger line on C10 or on C5 before the split has carried. Not a restatement of a carve-out: C10 had no lowering bound. Claim, Ages and Bounds rewritten same turn; brief, the practical rule and the C10 diagram core rewritten.

4. #4 survived 13 October 2026 · Campuzano et al. 2009; NCEE 2009-4041 (ERIC ED504657) · target: kill-condition

   In the US Department of Education's randomised study of software products, algebra teachers randomly assigned to teach Cognitive Tutor Algebra I as their core course got no gain over two school years on a standardised end-of-course algebra test (about −0.08 SD, upper confidence limit about +0.06 SD), and none in teachers' second year with the course (about −0.12 SD); with the Cognitive Tutor geometry trial (−0.19 SD), that makes two randomised school trials ruling out gains of 0.1 SD or more, including a second year of use, so C10's kill-condition is met.

   Strongest reading and verdict reasoning

   Steelmans to: C10 dies if replicated school-randomised trials on independent tests taken without the software find the effect about zero, with a confidence interval ruling out a gain of 0.1 SD, including where schools are in their second year of use, or find that such use lowers results; Campuzano et al. is an independent federal evaluation (Mathematica Policy Research for the Institute of Education Sciences) with random assignment within schools in four districts, a standardised ETS test, measured product use, and a second year in which teachers already knew the course; its two-year interval for Cognitive Tutor ends below +0.1 SD, and Pane et al. 2010 found a significant loss, so the only positive evidence left would be one trial's second-year high-school estimate.

   Why: Nightly-attacks 2026-10-13 Europe/Stockholm. Full text read (NCEE 2009-4041, February 2009, public-domain PDF via ERIC ED504657; publisher page at ies.ed.gov checked; not stored; a copy fetched by the failed 2026-10-12 run was reused for reading, outside the repository). Cognitive Tutor Algebra I (Chapter III, section J; Table III.2; Figure 4): 4 districts, 11 schools, 29 teachers (15 treatment, 14 control), 755 students across two years; average 2,149 logged-in minutes a year over 24 weeks; two-year effect −1.28 percentage points on the ETS End-of-Course Algebra Assessment (p = .26; 95 % CI −3.62 to +1.06 in Figure 4); second-year-only effect −2.10 (p = .30; 9 treatment and 9 control teachers, 276 students, 1,840 minutes). Converted with the report's SD of 17 (Table II.15 note; our conversion): two-year about −0.08 SD (CI about −0.21 to +0.06), second year about −0.12 SD. Kill fails: (1) the second-year estimate, the part the kill-condition singles out, is imprecise (p = .30, nine teachers per arm), and its interval, on the same conversion, reaches about +0.1 SD, so it does not rule out a 0.1 SD gain; (2) in the report's own analysis of returning algebra teachers (Cognitive Tutor and Larson pooled, 24 teachers, 1,051 students), the effect rose from −0.34 points (effect size −0.02) in year one to +2.56 (0.15, p = .03) in year two, an increase of 2.90 (p = .05); (3) the school-randomised trial of the same course (Pane et al. 2014) found about +0.2 SD for high schools in the second year; (4) here the "second year" is the teacher's second year, not the school's, randomisation was by teacher within volunteering schools, and the authors warn that single-product estimates rest on small samples. Underdetermined rather than confirming: the result fits the claim's "only by a small amount and not consistently". Not a duplicate: this study is among those pooled in Steenbergen-Hu & Cooper 2013 (2026-10-07 entry, whose force was the near-zero average), but no ledger line on C10 has tested the replicated-trials and second-year prong of the kill-condition. Not a preprint (final federal evaluation report). Claim field unchanged; Bounds record the result; brief C10 note and the practical rule Limits added. Status unchanged (weakened from earlier narrowings).

5. #5 survived 15 October 2026 · Feng, Huang & Collins 2023 (preprint); WestEd research report; AIED 2023 late-breaking result; ERIC ED658161 · target: kill-condition

   In a preregistered school-randomised replication across 63 North Carolina middle schools, routine use of ASSISTments tutoring-style homework support for a school year, in teachers' second year with it, should show the effect on an independent state test to be about zero, ruling out 0.1 SD; with Pane et al. 2010 and Campuzano et al. 2009 that would make replicated randomised school trials without a gain, meeting C10's kill-condition.

   Strongest reading and verdict reasoning

   Steelmans to: C10 dies if replicated school-randomised trials on independent tests taken without the software find the effect about zero (CI ruling out 0.1 SD), including where schools are in their second year of use; this is the largest school-randomised replication of a feedback-and-hints maths system in the project (63 schools, 41 districts, 5,991 students), with an independent state test, preregistered, and second-year teacher use.

   Why: Extra nightly run 2026-10-15 Europe/Stockholm (approved by Jonatan). Full WestEd report read (PDF; not stored). Grade-8 End-of-Grade maths test one year after use ended: 535.13 (SD 8.49) vs 534.33 (SD 7.72), +0.8 points, Hedges' g = 0.10, p = .011, ITT two-level model with prior achievement; 5,991 students, 63 schools, school attrition 0 %; baseline g = 0.08 (ns). Kill fails: the estimate is positive and significant, so it does not show an effect of about zero; it fits "a small amount" exactly. Caveats: no grade-7 outcome (test cancelled by COVID-19), usage interrupted in spring 2020, 1,962 accelerated-course students without a grade-8 test excluded from this analysis, ASSISTments is homework support with hints and scaffolds (at the edge of 'step-based tutoring'), and the report is not a journal article (preprint flag, recheck item in the attack method, the preprint rule). Not a duplicate: no ledger line on C10 has used ASSISTments; earlier entries were Cognitive Tutor. Claim field unchanged; Bounds, brief C10 paragraph and the practical rule Limits record it. Status unchanged (weakened).

6. #6 survived 16 October 2026 · Wijekumar, Meyer & Lei 2012; Wijekumar, Meyer, Lei, Lin, Johnson, Spielvogel, Shurmatz, Ray & Cook 2014 · target: kill-condition

   In two large classroom-randomised trials, a step-checking reading tutor (ITSS) used weekly for a whole school year as a partial substitute for regular language-arts teaching should show the effect on an independent standardised reading test to be about zero, ruling out 0.1 SD; with Pane et al. 2010 and Campuzano et al. 2009 that would make replicated randomised school trials without a gain, meeting C10's kill-condition.

   Strongest reading and verdict reasoning

   Steelmans to: C10 dies if replicated school-randomised trials on independent tests taken without the software find the effect about zero (CI ruling out 0.1 SD); these are two of the largest randomised trials of a tutoring system used inside regular teaching for a full school year (258 classrooms in total), with a published standardised test, and the grade-4 trial was not significant on it.

   Why: Extra run 2026-10-16 (extra 2) Europe/Stockholm, ordered by Jonatan. Both full texts read (author copies at literacy.io; not stored). Design: within-school random assignment of classrooms (odd-numbered schools blocked with similar schools), 30–45 min a week over 6–7 months (Oct 2009–Apr 2010), partial substitute for the language-arts curriculum; ITSS's animated tutor models the strategy, then checks each part of the learner's answer (signalling words, text structure, main idea, recall) against a database and gives rule-based feedback, with pop-up help. Grade 4 (ETR&D 60, 987–1013): Gray Silent Reading Test posttest 29.19 vs 27.86 at class level, adjusted difference 1.07, t(82) = 1.77, p = .08, effect size .10; researcher-designed measures ES .11–.49. Grade 5 (JREE 7, 331–357): 65 ITSS classrooms (1,351 pupils) vs 63 control (1,294); GSRT adjusted difference 2.34 (SE .44), pooled SD 11.45, ES .20, p < .001; researcher-designed measures ES .15–.53. Kill fails: neither trial shows an effect of about zero that rules out 0.1 SD (grade 4's estimate is 0.10 with an interval reaching above it; grade 5 is 0.20 and significant); the pair fits the claim's "a small amount and not consistently". Caveats: reading rather than maths; developers evaluated their own system; not preregistered; classroom-level randomisation within schools (contamination checked by observation); the standardised test was mostly narrative texts while ITSS teaches expository structure; ITSS is a strategy tutor whose 'steps' are parts of a reading task rather than steps of a maths solution, so it sits at the edge of C10's scope (as ASSISTments did). Not a duplicate: no ledger line on C10 has used ITSS or a reading tutor. Not a preprint (two peer-reviewed journal articles). C10 reaches 3 survivals (floor) with this entry; the promotion rule triggered. Claim field unchanged; Bounds, Ages, brief C10 note and the practical rule Limits record it. Status unchanged (weakened).

7. #7 survived 15 November 2026 · Leite, Zhang, Rana, Hao, Busch, Kong & Kuang 2026 · target: kill-condition

   A new meta-analysis restricted to US K–12 experiments that meet What Works Clearinghouse design standards would meet C10's kill-condition if intelligent tutoring systems compared with regular teaching gave about zero on standardised tests and at the end of the school year (an interval ruling out 0.1 SD); it did not: standardised tests g = 0.200 (95% CI 0.166 to 0.234, 61 effects) and end-of-school-year measures g = 0.155 (0.125 to 0.185, 47 effects).

   Strongest reading and verdict reasoning

   Steelmans to: C10 dies if a meta-analysis of routine school use, on independent tests taken without the software, finds the effect about zero; earlier pooled estimates fell to about zero on standardised tests and at the end of the year (Steenbergen-Hu & Cooper 2013), and a synthesis limited to the most rigorous US K–12 experiments (2011–2021, double-screened, WWC-standard designs only, moderators for outcome type, timing and duration) is the fairest current test — if the classic near-zero result is real, it should reappear here.

   Why: Nightly-attacks 2026-11-15 Europe/Stockholm (ordinary rotation). Full text (submitted version): 18 experiments, 77 effects, 11 systems; overall g = 0.271 (SE 0.011); standardised tests g = 0.200 (0.166 to 0.234) vs researcher-made 0.483 (ns, 16 effects); measured immediately after 0.460 vs end of school year 0.155 (0.125 to 0.185); duration up to two weeks 0.334 (ns), two weeks to three months 0.449 (ns), three to six months 0.225 (0.171 to 0.279), more than six months 0.110 (−0.077 to 0.297, ns); maths 0.253, reading and writing 0.323; low achievers 0.270; rural settings 0.146; after-school use 0.014 (ns); journal articles 0.190 vs dissertations 0.625; fail-safe N (Orwin, target 0.10) 191. Kill fails: on standardised tests and at the end of the year the interval excludes zero and lies above 0.1. Caveats: not preregistered (an OSF resources page only); 39 of the 77 effects (51%) come from one reading-strategy tutor (ITSS; Wijekumar et al. 2012–2022) and 10 more from its Spanish-language version, so the pool overlaps C10's 2026-10-16 survival (Wijekumar 2012/2014); not all systems are step-based maths tutors; the longest-use subgroup (more than six months) is not significant; very narrow intervals from the robust-variance model; US only. Not a duplicate (different source; first synthesis limited to WWC-standard US experiments). Not a restatement of the Bounds. Not a preprint (published, peer reviewed; numbers from the submitted version — recheck item added). Claim field and kill line unchanged; Bounds get an evidence line; the practical rule Limits updated. Status unchanged (weakened). Fruitless streak 1 → 0.