Learning, under fire

Home · Claims · Where it's weakest

Where it's weakest

Claims ranked by how exposed they are: first by open challenges (sources linked as pointing against the claim that no attack has used yet), then by fewest attacks. The top of the list is where the next attack should aim.

  1. 1 C2 Answer-giving AI in practice alive

    4 open challenges4 attacks Attack 1 on C2, 2026-09-27: survived (Bastani et al. 2025)Attack 2 on C2, 2026-09-27: survived (Barcaui 2025)Attack 3 on C2, 2026-09-29: survived (Kreijkes et al. 2026)Attack 4 on C2, 2026-09-29: survived (Bergh, Tag, Vassar & Renzella 2026)3 limiting sources not yet used in an attack

    Aim next: Test the open challenges first: Deng et al. 2025, Doo & Park 2026, Fan et al. 2026, Wu & Yu 2024. They point against the claim and no attack has used them yet.

    Why these count as challenges
    • Deng et al. 2025: Positive pooled effect of ChatGPT; the bank notes that without closed-book outcomes this is performance, not learning. Open test for C2.
    • Doo & Park 2026: Positive pooled effect on achievement; closed-book follow-up absent or mixed. Open test for C2.
    • Fan et al. 2026: Higher-education meta g ≈ 0.50; not strictly closed-book. Open test for C2.
    • Wu & Yu 2024: Positive chatbot effects, stronger in higher education than K–12; closed-book rare. Open test for C2.
  2. 2 C0 Retrieval beats restudy narrowed

    2 open challenges12 attacks Attack 1 on C0, 2026-09-08: narrowed (Roediger & Karpicke 2006)Attack 2 on C0, 2026-09-08: survived (no named source (attacker's own inference))Attack 3 on C0, 2026-09-08: survived (Dunlosky et al. 2013)Attack 4 on C0, 2026-09-08: survived (Bjork & Bjork 2020)Attack 5 on C0, 2026-09-08: survived (Roediger & Karpicke 2006)Attack 6 on C0, 2026-09-08: survived (Adesope et al. 2017)Attack 7 on C0, 2026-09-08: narrowed (Roediger & Karpicke 2006)Attack 8 on C0, 2026-09-08: narrowed (Dunlosky et al. 2013)Attack 9 on C0, 2026-09-08: narrowed (Butler & Roediger 2008)Attack 10 on C0, 2026-09-08: narrowed (Dunlosky et al. 2013)Attack 11 on C0, 2026-09-28: survived (Gupta, Pan & Rickard 2026)Attack 12 on C0, 2026-09-28: narrowed (McCarter, Huber & Cowell 2025)0 limiting sources not yet used in an attack

    Aim next: Test the open challenges first: Kornell 2025, Chan et al. 2024. They point against the claim and no attack has used them yet.

    Why these count as challenges
    • Kornell 2025: Retrieval success manipulated by priming did not change final recall after retrieval with feedback; bears on C0's 'correctly retrieved' clause; no restudy arm, single trial, immediate test. Queued, not a verdict.
    • Chan et al. 2024: Queued: bears on whether TE size requires high practice success; control is no-test not restudy; not applied as C0 attack.
  3. 3 C3 Paper vs screen, deep reading narrowed

    0 open challenges4 attacks Attack 1 on C3, 2026-09-27: narrowed (Florit et al. 2025)Attack 2 on C3, 2026-09-28: survived (Li & Yan 2024)Attack 3 on C3, 2026-09-29: survived (Foss, Støle & Magyari 2026)Attack 4 on C3, 2026-09-30: survived (Ronconi, Altoè & Mason 2025)6 limiting sources not yet used in an attack

    Aim next: Open question on this claim: Does C3’s average paper advantage hold when reading is self-paced, given that neither Delgado 2018 (self-paced g = −.09, ns) nor Li & Yan 2024 (free reading g = 0.192, ns) found a significant difference without time limits — or is the paper advantage specific to time-limited reading? (Replaces the Li & Yan 2024 question, resolved 2026-09-28: C3 survived that kill-test.)

  4. 4 C1 What counts as learned narrowed

    0 open challenges7 attacks Attack 1 on C1, 2026-09-14: narrowed (Roediger & Karpicke 2006)Attack 2 on C1, 2026-09-14: narrowed (Tulving & Pearlstone 1966)Attack 3 on C1, 2026-09-14: survived (Karpicke & Blunt 2011)Attack 4 on C1, 2026-09-14: survived (Rowland 2014)Attack 5 on C1, 2026-09-14: narrowed (Morris, Bransford & Franks 1977)Attack 6 on C1, 2026-10-01: narrowed (Gupta, Pan & Rickard 2022)Attack 7 on C1, 2026-10-02: survived (Racsmány, Szőllősi & Marián 2020)1 limiting source not yet used in an attack

    Aim next: Open question on this claim: Does C1 require that encoding verification use the same surface format as C0’s eventual criterion test, or is any successful retrieval without re-presentation enough? Does C1’s accessibility criterion collapse to “was studied / received feedback re-presentation,” with no residual role for cue-matched access as a distinct criterion — given that under corrective feedback C0 does not require high unaided practice success (Gupta, Pan & Rickard 2022) and that without practice feedback the retrieval-over-restudy contrast shrinks (does not appear equally) when practice success is experimentally low (Racsmány, Szőllősi & Marián 2020, DOI 10.3758/s13421-020-01041-5, kill-condition test survived 2026-10-02)? (Replaces the residual no-feedback equality question, resolved 2026-10-02.)

Challenge links are a first-pass draft, pending human review. A challenge counts as answered when its source has been used in an attack on that claim.