聯合評分還是逐選項評分?17 million 至 1 billion 參數的封閉集合決策緊湊編碼器
Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters
研究比較封閉集合決策中的聯合評分與逐選項評分,結果顯示在 32 million 參數設定下,逐選項方式表現較佳。在 37 個基準決策套件的留出樣本上,三個 seed 試驗中逐選項方式按相同 optimizer steps 和近似相同訓練計算量,分別高出 5.8 和 2.8 個經機會校正的技能分,但需要多 5 至 13 倍編碼器運算次數。
Published on Oct 22
Authors:
Abstract
Many language-model pipelines end in a closed-set decision, such as selecting a tool. Prompting a large language model for it is slow and order-sensitive. We ask how small an encoder for such decisions can be and whether it should read the options jointly or separately. In a three-seed study at 32 million parameters, per-option scoring beats joint scoring on a held-out sample of a 37-benchmark decision suite by 5.8 points of chance-corrected skill at matched optimizer steps and by 2.8 at approximately matched training compute; two external benchmarks agree, though not significantly on their unseen tasks. Crossing reader with initialisation shows that the advantage requires a reranker initialisation, whose query-candidate pair input matches per-option reading; from the raw encoder the two readers are indistinguishable. Per-option scoring needs 5 to 13 times more encoder passes: at most twice the median latency, but more than four times the mean. Of six released joint encoders (17 million to 1 billion parameters), the 68- and 150-million-parameter models exceed a 435-million-parameter DeBERTa-v3 zero-shot classifier by 4.9 and 5.9 accuracy points on 209 overlap-audited System1-Arena tasks and are at least on par with it on the 145 tasks unseen in training. The smallest answers a median request in 14 milliseconds on 16 processor threads, against 728 for the classifier. Reordering options changes 12 to 23% of the joint models' answers (per-option scoring: none). Up to 68 million parameters, they lead their size range on a public leaderboard's private, mainly in-distribution tests.
View arXiv page View PDF Project page GitHub 0 Add to collection
Get this paper in your agent:
hf papers read 2610.30970
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 8
Text Classification • 0.1B • Updated 4 days ago • 477 • 14tardellirs/crivo-150m
Text Classification • 68.4M • Updated 4 days ago • 91 • 3tardellirs/crivo-68m
527 • Updated 4 days ago • 2tardellirs/prumo
Text Classification • 32M • Updated 4 days ago • 175 • 1tardellirs/crivo-32m
Datasets citing this paper 1
tardellirs/decision-index-results
Spaces citing this paper 3
Collections including this paper 1
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co