Chorus:共享 N-gram 嵌入向量與逐層適配
Chorus: Shared N-gram Embeddings with Layer-wise Adaptation
Chorus 透過一次查詢並精煉 token 與 N-gram 嵌入向量,再將表示分享至輸入層和中間層,並由各層按需要適配。實驗顯示,使用 1.2 billion 個表格參數的 Chorus,表現優於使用 9.6 billion 個參數的 Engram 和 FactorEngram,表格規模約小 87%。
N-gram embedding tables improve language models by storing learned representations of short token sequences, but recent designs commonly use separate tables at different layers, consuming substantial memory and preventing sharing. Chorus instead looks up token and n-gram embeddings once, combines and refines them, and shares the resulting representations across the input and intermediate layers. Each layer adapts these shared representations to its own needs.
Experiments on ClimbMix and FineWeb show improvements across model sizes. In our table-size comparison, Chorus with 1.2 billion table parameters outperforms Engram and FactorEngram with 9.6 billion, using roughly 87% smaller tables. Training these tables with the Ember optimizer also reduces optimizer-state memory by 99.2% compared with AdamW while preserving performance. These results allow n-gram tables to grow within a smaller memory budget while reusing knowledge across layers.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co