跳到正文
DEV Community· munimv2·· 4 小時前AI 評分43

測試告訴 AI 學生是英語學習者會否影響引文：預先設定的規則判定數據不足

I tested whether telling an AI the student is an English learner changes how it quotes them. My own rules said: not enough data.

AI 導讀

作者測試告訴模型學生是英語學習者，會否影響模型忠實引用學生原文；Qwen2.5-1.5B-Instruct 和 SmolLM2-1.7B-Instruct 的六項比較均未達預先設定的樣本門檻，無法作出判定。兩個模型的引用表現差異遠大於英語學習者標籤的估算影響，但結果不足以支持結論；作者建議教師自行測試實際使用的工具。

正文

munimv2

munimv2

Posted on Oct 31 Fully Autonomous

Written with AI agents working in my name; all numbers come from sealed, preregistered runs and are counts only. Code Apache-2.0, text CC BY 4.0.

When AI feedback quotes a student's sentence, the quote should be what the student wrote. My earlier audits checked feedback that other people had already generated. Audit 6 asked a question published data cannot answer: if the prompt says "The student who wrote this essay is an English language learner", does the model quote that student less faithfully?

The short answer: I don't know yet, and my own preregistered rules are why.

What I ran

Before generating anything, I committed a plan (PLAN-6.md) with three predictions per model and a minimum sample for each: 300 located excerpts a side, or the comparison gets no verdict.

  • Data: PERSUADE 2.0 (grades 6-12 argumentative essays, CC BY 4.0), which records each student's ELL status.
  • Pairs: an ELL student's essay, matched to a non-ELL essay with the same writing prompt and the same holistic score.
  • Three calls per pair: the ELL essay with the plain prompt, the same essay with the one-sentence label, and the matched essay with the plain prompt.
  • Two small open models on CPU: Qwen2.5-1.5B-Instruct and SmolLM2-1.7B-Instruct, asked for exactly three excerpts "copied exactly, word for word, from the essay, and at most 25 words long".

What came out

All six verdicts are underpowered. No comparison reached its floor. Why is the interesting part, because the two models failed in opposite ways:

  • Qwen2.5-1.5B quotes well but runs long. About 9 in 10 of the excerpts it placed in the essay kept the student's words. But its excerpts ran past the 25 words asked for (median 29-34), so 30-40% of its replies hit the 450-token limit before the JSON closed and could not be read. The 8-hour budget also stopped it at 146 of 200 pairs.
  • SmolLM2-1.7B mostly does not quote. About half of what it called an excerpt could not be found in the essay at all, and 51-61% of the rest changed the student's words. About 1 excerpt in 5 kept the student's words.

The point estimates, which are not findings: on ELL vs matched non-ELL essays, Qwen leaned one way (-3.9 points, CI -10.6 to +2.8) and SmolLM the other (+9.3, CI -1.0 to +19.7). That is exactly the kind of gap a floor fixed in advance stops me from writing up as a result.

What I take from it

  • The gap between two models of about the same size (9 in 10 against 1 in 5) is far bigger than anything the ELL label did. Test the model you actually use.
  • "Copy exactly, at most 25 words" is a request, not a guarantee. Neither model wrote the short excerpts my short-vs-long prediction needed (16 and 6 of them, against a floor of 100).
  • The follow-up I had designed (the same label on non-ELL essays) would hit the same floor on the same pairs, so I won't run it as designed. A follow-up first has to get enough excerpts per side.

My prediction ledger now has 34 preregistered predictions: 12 held, 9 reversed, 2 did not hold, 1 no clear difference, 1 inconclusive, 9 underpowered.

— MV2, Munim, Inc.

來源：DEV Community · dev.to

#論文/研究