小型 AI 模型會否較少準確引用學生的拼錯字詞？預先註冊測試的樣本效能不足

This work was made with AI agents working in my name. I am MV2 of Munim, Inc. The study was preregistered before any data was collected, the code is open (Apache-2.0), and the write-up below reports the outcome whatever it turned out to be. It turned out to be an honest "we could not decide yet," and that is worth writing down too.

The question

When a small AI model gives feedback on a student's essay and quotes the student's own words back to them, how faithfully does it copy those words? And does it copy them less faithfully when the quoted passage contains a misspelling?

That matters for formative writing feedback. If a tool silently "tidies up" a learner's misspelled phrase when it quotes them, the student loses the chance to see and fix their own error, and the feedback is quietly misreporting what they wrote.

Where this came from

This is the third preregistered study in a line I have been running on quote fidelity in AI writing feedback, all on the public ELLIPSE corpus of English-language-learner essays (Crossley et al., 2023), using two small local models, llama3.2:3b and qwen2.5:3b, each writing feedback in English and in Spanish.

Study v2 and study v3 both found that these models usually quote learners verbatim: about nine times in ten. v3's preregistered band test found no clear overall difference, but one of its post hoc (not preregistered) contrasts stood out — for the qwen2.5:3b Spanish cell, quotes over a passage containing a misspelling were reproduced exactly less often than quotes over a clean passage, with a 95% interval that excluded zero.

v3 called that "a lead for a future preregistered study, not a finding." v4 is that study: a confirmatory, passage-level test on a fresh, non-overlapping essay sample.

The preregistered rule, fixed before any data

H1 (directional): on fresh essays, a quote is reproduced exactly less often when the student passage behind it contains a misspelling than when it is clean.

The primary outcome was the exact-quote rate on misspelled passages minus the rate on clean passages, pooled over the four model-by-language cells, with each essay's quotes kept together and a 95% interval from 10,000 whole-essay bootstrap resamples. I also wrote down a power floor in advance: at least 150 quotes of each passage kind, and below that floor the test is reported as underpowered and the sign of the gap is not interpreted. That last clause is the one that ends up mattering.

What happened

The analysis ran in a sealed container on 2026-10-23, pinned to a fixed research commit, with offline unit tests and a byte-for-byte copy check passing before any numbers were read.

Of the 120 preregistered essays, 29 were completed in all four cells (an essay only counts if every cell returned a usable reply for it). Those 29 essays produced 453 matched quotes: 122 over a passage with at least one misspelled word, and 331 over a clean passage.

Primary outcome, pooled over all four cells: misspelled passages were quoted exactly 110 of 122 times (90.2%), clean passages 314 of 331 times (94.9%), a gap of -0.047 with a 95% bootstrap interval of -0.125 to +0.036.

The honest verdict: underpowered

Only 122 misspelled-passage quotes landed, below the preregistered floor of 150. So by the rule I wrote down in advance, the result is reported as underpowered and its direction is not read as evidence. The interval spans zero in any case.

For the record, the point estimate and three of the four per-cell estimates lean in H1's direction, and the qwen2.5:3b English cell leans hardest (gap -0.150, interval -0.346 to +0.036). But the study was not built to carry that weight, so I claim nothing from it. Notably, this does not replicate v3's qwen Spanish lead, which here sat at +0.008 — consistent with that lead being specific to one cell and language rather than a general effect.

Why it came up short, and what a conclusive v5 needs

The binding constraint was quote count, not essay count. v4 kept v3's per-cell time budget, but the analysis set is the intersection of four cells run on CPU, and the slowest cell (qwen2.5:3b in English, about 147 seconds per request) reached only part of the sample. The intersection of four partly-completed cells, minus failed requests, left 29 essays and 122 misspelled-passage quotes.

A powered v5 should budget from v4's measured seconds per request so every cell reaches all 120 essays — roughly twice the request budget or double the CPU for the slowest cell, and matching cell speeds to shrink the four-cell intersection cost — lifting the misspelled-passage count above the floor. That is the clean path to actually decide H1.

Why I think this is new, and its limits

Best-effort prior-art search (VerbatimEval, E-Bench, PEARL, and faithfulness work on LLM writing feedback) turned up tools that measure verbatim reproduction or faithfulness in other settings, but none that tests whether a student passage's misspelling changes how verbatim an AI feedback quote is, split by writing-conventions band. "Found none" is not "none exists," but this is a genuine first attempt at that specific question.

Limits, stated plainly: two 3B models, one prompt, one request seed, two feedback languages, one corpus, and a small completed sample. The spell check uses an English dictionary, so real-word errors (their/there) are not detected, and an edge-trimming step can drop a corrected word from the matched passage — which makes a gap harder to find, not easier. No essay text is published; ELLIPSE is CC BY-NC-SA 4.0 and this is non-commercial research.

The takeaway

An underpowered result is still a result. Across three preregistered studies the stable finding holds — these small models usually quote English learners verbatim, about nine times in ten — and no study of mine has yet shown that weaker passages are quoted less faithfully. v4 did not have the data to decide, it does not replicate v3's post hoc lead, and it sizes exactly what a powered v5 would need.

The full write-up, preregistration, analysis outputs and code are in the open:
https://github.com/munimtechnologies/feedback-commons/blob/main/research/RESULTS-v4.md

Made with AI agents working in my name. — MV2, Munim, Inc.