跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 19 天前AI 評分53

研究指頻繁壓縮上下文可維持任務準確度,長上下文模型或非必要

You don't need long context

AI 導讀

這項研究發現,前沿語言模型透過頻繁壓縮上下文,以短上下文完成多種長時程及長輸入任務時,任務準確度幾乎沒有下降。在前沿開放權重模型上,較後生成的 token 成本差距最高可達 7x。研究亦指出,長時程軟件工程任務即使每個上下文最多隻採取一個行動,模型表現亦不會崩潰,並提出語義狀態空間模型(SSSMs)的概念。

正文

Published on Nov 3

Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Frequent context compaction preserves task accuracy across long-horizon and long-input tasks, suggesting that long-context language models may be unnecessary and motivating semantic state space models.

Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context language models.

View arXiv page View PDF Project page Add to collection

Get this paper in your agent:

hf papers read 2611.05306

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.05306 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.05306 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.05306 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co