Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale
Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale
研究在 1.7B 級模型中,以固定的 16 維二進制向量取代可訓練輸入嵌入表,並將向量按模型寬度重複,且不加入可訓練輸入投影。模型仍具備相當的語言建模能力,但可訓練輸入嵌入的對照模型在多項主要基準測試中表現較佳。詞彙表包含 49,152 個 token;每種介面只進行一次訓練,輸出頭仍可訓練。
When we teach students about LLMs, explanations inherited from word2vec can leave the impression that tokens are inherently semantic units and that their input embeddings are where meaning resides. This work offers a concrete experiment for questioning that picture.
We replace the trainable input embedding table with fixed 16-dimensional binary vectors, deterministically repeated to model width, with no additional trainable input projection. This is a token-identity encoding, not weight quantization. At 1.7B-class scale, these models acquire substantial language-modeling capability, although the learned-input control performs better on several key benchmarks.
There is nothing linguistically special about sixteen dimensions: it is the minimum fixed-length binary width needed to distinguish the 49,152 tokens in our vocabulary. Wider binary vectors—for example, 32-dimensional ones—or suitably constructed, distinct sine/cosine-based token codes are plausible alternatives, but we do not test them here. Preserving token identity is a coding requirement; whether a network can learn effectively from a particular encoding is an empirical question.
The conceptual distinction is between identifying a token and learning to interpret and use token sequences in context. Tokenization defines the input units; it does not make each unit a self-contained piece of meaning.
Nor does removing the trainable input table establish that meaning simply moves into the first trainable layer. The experiment does not determine where semantic information is represented or how it is distributed across layers. Instead, fixing the input coordinates provides a controlled setting for investigating representation formation—without making the model automatically interpretable.
For students, the takeaway is that token identity, input representation, and meaning should not be conflated. Word-vector analogies can remain useful illustrations without becoming the lens through which all LLM computation is understood.
We report one training run per interface, and the output head remains trainable. Models and research artifacts: https://huggingface.co/collections/Bochkov/do-language-models-need-a-trainable-input-embedding-table
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co