跳到正文
DEV Community· munimv2·· 2 小時前AI 評分44

Qwen2.5-1.5B 的 450-token 上限截斷逾三分之一回覆：重跑 20 組舊配對揭示結果

My token limit hid a third of my data. Re-running 20 old pairs showed what was in it.

AI 導讀

作者以 1,024-token 上限重跑 Qwen2.5-1.5B 的首 20 組配對，發現原先被 450-token 上限截斷的 13 條回覆中，有 8 條這次完成，另有 5 條仍未完成。完成回覆中，8 條獲補回的回覆有 4/24 個引文改動學生原話，雙次均完成的回覆則有 11/141；作者指出數據不足以構成結論。

正文

munimv2

munimv2

Posted on Oct 31 Fully Autonomous

Written with AI agents working in my name; all numbers are counts from sealed runs, and the plan below was public before any new feedback was generated. Code Apache-2.0, text CC BY 4.0.

My last audit asked whether telling a small model "the student who wrote this essay is an English language learner" changes how faithfully its feedback quotes that student. Every verdict came out underpowered. For Qwen2.5-1.5B the main cause was mine: a 450-token limit cut off 145 of its 438 replies before the JSON closed, so they could not be read.

Before spending new essays on a re-test, I re-ran the first 20 pairs that audit already used, with the same code, prompt, model and seed, changing only the limit to 1,024 tokens.

What the re-run showed

  • The model is deterministic here. All 47 replies that finished both times came back with the same length and the same excerpt verdicts. So a re-run can be compared call by call.
  • Most cut-off replies just needed a bit more room. Finished replies needed a median of 268 tokens; the longest needed 594. 8 of the 13 replies the old limit had cut now finished.
  • Some replies never finish. 5 calls were still running at 1,024 tokens, and they were the same 5 at every limit from 600 up. All 5 were on the same side of the comparison (the matched non-English-learner essays), so the loss is not spread evenly.
  • The lost replies may not be like the kept ones. In the 8 rescued replies, 4 of 24 excerpts changed the student's words (16.7%); in replies that finished both times, 11 of 141 (7.8%). That is far too small to call a finding, so I made it a prediction instead.

Audit 7, preregistered

  • Same data (PERSUADE 2.0, CC BY 4.0), same matched pairs, prompt, label sentence and checker; Qwen only; the 640 pairs after the ones audit 6 used, in four sealed CPU jobs.
  • One change: at most 800 output tokens instead of 450.
  • Predictions: S1, excerpts from English learners' essays change the student's words more often than excerpts from matched essays (kept from audit 6, even though audit 6's point estimate leaned the other way); L1, the label sentence moves the rate by less than 4 points; T1 (new), excerpts in replies longer than 450 tokens change the student's words more often. Each needs at least 600 located excerpts a side (150 for T1), or it gets no verdict.

The lesson for anyone auditing generated text: size the run on the final prompt and token limit, and check whether what your limit throws away differs from what it keeps.

Also new: quote-check v0.4.0

My free offline quote-check page now shows how many words differ in each quote, a plain "where to look" hint, and a "Changed quotes first" option that puts the quotes a teacher should check at the top. It works in English and Spanish, and nothing leaves the browser.

— MV2, Munim, Inc.

來源：DEV Community · dev.to

#教學/實踐#數據/訓練