跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-05AI 評分40

Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

Closing the Context Gap: Activation Alignment for Tabular In-Context Learning

AI 導讀

激活對齊利用完整上下文訓練模型,使模型在只使用部分上下文時,仍能維持接近完整上下文的預測表現。研究在 TabArena 的 38 個分類數據集上評估 TabPFN-3 和 TabFM,兩者經對齊的學生模型在不同上下文預算下均顯著優於未對齊基線;低數據量時,對齊可追回教師模型近一半的預測優勢。對齊器在合成無標籤數據上訓練,無需 GPU,並可在數秒至數分鐘內收斂。

正文

Published on Oct 5

Authors:

Abstract

Tabular foundation models perform in-context learning (ICL) by conditioning predictions on labeled training examples provided as context. Unlike traditional models that separate training from inference, these models must process all training examples in every forward pass, making each prediction expensive. Restricting the number of training examples reduces this cost but substantially degrades performance. Instead of discarding context, we propose activation alignment, a method that leverages the full context to teach a model how to behave when seeing only a subset. This is achieved by training a lightweight linear transformation on synthetic unlabeled data to map the intermediate activations of a data-constrained "student" (using partial context) toward those of a full-context "teacher" (using all data). Training the aligner requires no GPU and converges in seconds to minutes on commodity hardware. We evaluate on 38 classification datasets from the TabArena benchmark using the leading two tabular foundation models, TabPFN-3 and TabFM. Across all context budgets, the aligned student yields broad, statistically significant improvements over the unaligned baseline for both models. In low-data regimes, alignment recovers nearly half of the teacher's predictive advantage. The method provides a practical, low-overhead approach to achieving the inference speed of compact contexts while closing a significant fraction of the performance gap to the full-context teacher.

View arXiv page View PDF Project page GitHub 1 Add to collection

Get this paper in your agent:

hf papers read 2610.06679

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.06679 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.06679 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.06679 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co