跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-14AI 評分43

COUNT:以 N-gram 計數擴展主機記憶體應用,實現免訓練適應

The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation

AI 導讀

COUNT 將預先計算的 N-gram 計數存於主機記憶體,無需新增學習參數即可建立嵌入向量。用於預訓練 1B 參數語言模型時,在固定計算預算下可改善驗證損失,改善幅度最高相當於使用多 57% 計算量的模型。推理時可換用特定領域語料的計數表,無需訓練即可適應新任務。

正文

Published on Oct 14

Authors:

,

Abstract

The scaling relationship of compute and performance for language models (LMs) has been extensively studied; more recently, models are being designed to scale memory use, as well. In particular, adding embeddings for n-grams scales memory separately from compute use, because each pass only processes individual embedding rows, which are much smaller than the storage used for the whole embedding table. However, existing methods train embeddings from scratch, which needs expensive GPU memory for fast reading and writing. Meanwhile, the ordinary memory of the host system is cheap and underutilized by these training methods. To scale on host memory, we introduce COUNT (Counting Offline Unlocks N-gram Transformers), which builds n-gram embeddings without adding learned parameters by storing precomputed n-gram counts on host. We use the corpus frequencies to weight together the LM's own unembeddings of the next tokens most often following the current n-gram history. At a fixed compute budget, COUNT can be added as an additional input embedding for pretraining 1B-parameter LMs to improve validation loss as much as an LM with 57% more compute. If GPU memory is limited, using COUNT to let backbone parameters fill GPU memory gives better performance than splitting memory with additional learned n-gram embeddings. Beyond efficiency, COUNT allows switching in a different table of n-gram counts from a domain-specific corpus at inference time to adapt to a new task without any training.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.20456

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 3

IanMagnusson/count-checkpoints-standard-transformer

Updated about 1 hour ago

IanMagnusson/count-checkpoints-count

Updated about 1 hour ago

IanMagnusson/count-checkpoints-learned-ngram

Updated about 1 hour ago

Datasets citing this paper 1

IanMagnusson/count-ngram-tables

Updated about 1 hour ago

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.20456 in a Space README.md to link it from this page.

Collections including this paper 1

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co