以測試時回歸重新思考序列壓縮
Rethinking Sequence Compression with Test-time Regression
TTR-Compressor 以測試時回歸作為序列壓縮方法,透過輸入 token 建立回歸問題,並以擬合係數形成壓縮表示。在區塊式稀疏注意力中,它的 RULER 表現大幅勝過基於注意力的壓縮器;在視覺 token 合併中,它優於標準 MLP 合併器,並以少 55.6% 的視覺 token 達到相同性能。
Published on Oct 22
Authors:
,
,
,
,
,
Abstract
Test-time regression provides a learnable, closed-form alternative to pooling for compressing sequences in efficient language and multimodal models.
Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.31574
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.31574 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.31574 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.31574 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co