跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 10 天前AI 評分49

解析成功、看似合理,卻是錯誤:輸出格式左右小模型語義解析的隱性錯誤

Parsed, Plausible, Wrong: Output Format Controls Silent Errors in Small-Model Semantic Parsing

AI 導讀

研究以 PIZZA 訂餐基準測試 Qwen3.5-0.8B 和 LFM2.5-350M,發現輸出格式會影響小模型語義解析中格式正確但語義錯誤的隱性錯誤率。在 300 次訓練中,普通文字編碼於兩個模型的準確度均最高;Qwen 的 exact match 為 87.5%,高於原生 s-expression 的 85.8%,且少輸出 4.1 倍 token。

正文

Published on Nov 12

Authors:

,

,

,

Abstract

A small fine-tuned parser that emits a well-formed but wrong tree fails silently: nothing downstream can tell. We ask whether the output format, a free design choice, controls how often this happens. On the PIZZA ordering benchmark we fine-tune two sub-billion models (Qwen3.5-0.8B and LFM2.5-350M) with LoRA on six lossless encodings of the same target tree, from 44.3 down to 10.9 tokens per order, including two matched-length pairs that vary surface form at fixed length. Across 300 runs (5 training pool sizes, 5 seeds) with a hierarchical bootstrap and Holm correction fixed before the results were read, shortening alone never helps, but writing the target in ordinary words does. The word encoding is the most accurate format on both models (87.5% vs. 85.8% exact match for the native s-expression on Qwen) while emitting 4.1x fewer tokens, and it cuts the silent error rate by 2.3x on Qwen and 1.8x on LFM2, separating at every pool size. Error analysis locates much of the gain in negation. All runs also learn to refuse off-menu input (held out: 95.9-99.0% handled) and to apply in-utterance self-corrections. Strict refusal has a measured price: 2.1-10.1% of real, mostly polite, orders are refused.

View arXiv page View PDF Project page GitHub 3 Add to collection

Get this paper in your agent:

hf papers read 2611.17169

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 2

umarsheikh303/pizza-words-gguf

Text Generation •

0.8B •

Updated 3 days ago

•

431

•

3

umarsheikh303/pizza-lfm2-words-gguf

Text Generation •

0.4B •

Updated 3 days ago

•

132

•

2

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.17169 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.17169 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co