跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-16AI 評分42

Transformer 的條件式無毛定理:語言模型如何忘記提示詞

A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt

AI 導讀

研究證明,自回歸 Transformer 對提示詞的影響會隨生成 token 數增加而按冪律衰減,並將此結果稱為「條件式無毛定理」。訓練模型的衰減指數 α 約為 0.55–0.7,未訓練模型約為 0.2–0.5;隨 t 增大,提示詞對下一個 token 的影響趨於消失,能保留的特徵須由已生成文本承載。

正文

Published on Oct 16

Authors:

Abstract

Autoregressive transformers retain prompt influence via a power-law decay driven by attention dilution and relay mechanisms, with trained models forgetting faster than untrained ones.

We ask how much an autoregressive transformer retains of its prompt after generating t tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states decays approximately as a power law t^{-α}, where α=1 is the rate at which a fixed prompt's share of uniform attention shrinks and smaller values mean longer memory. Trained models forget fast (αapprox0.55--0.7), untrained ones slowly (αapprox0.2--0.5). We prove that this offset is Obig(P(log t)^{L-1}/tbig) for a prompt of length P and depth L. The bound holds for finite-depth pre-LN softmax transformers with bounded attention logits, under one additional condition: the pre-norm activations stay away from zero. The mechanism is geometric: the prompt's direct influence is diluted as the context grows, and relay --- continuation tokens passing on what earlier tokens read from the prompt --- can only add logarithmic factors. This is what we call the conditional no-hair theorem: given the text generated so far, the prompt's influence on the next token vanishes as t grows, so any property of the prompt that survives, such as its language, must be carried by the text itself. An exactly solvable linear relay model shows that the bound cannot be improved and reproduces the rise of the local exponent towards 1 seen in untrained networks. Cancelling the dilution with a logit bias stops the decay early in training and reverses it after, so dilution is necessary for the forgetting we measure; untrained networks and the solvable model show it is also sufficient.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.24133

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.24133 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.24133 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.24133 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co