跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-22AI 評分51

OPTIC 以跨層潛在向量量化極限壓縮 Gemma 4 逐層嵌入

Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization

AI 導讀

研究提出 OPTIC,以跨層共用量化碼壓縮 Gemma 4 的逐層嵌入(PLE);約 100× 壓縮下,其在七項基準保留 BF16 E4B 性能的 97.8%,高於產品量化的 66.3%。直接以緊湊代碼提供 BF16 E4B 服務,可減少 29.7% GPU 記憶體用量;在 RTX 4090、相同 1 GiB KV cache 下,實際執行時間最多增加 0.40%。

正文

Recent work shows that increasing embedding capacity can improve language models, making it a promising direction for scaling. As more capacity goes into embedding tables, extreme compression becomes increasingly important: we want to keep the benefits of larger embeddings while fitting models into limited deployment memory.

We study this in Gemma 4, where per-layer embeddings (PLE) account for 35.2% of E4B's parameters. At 100× compression, only about five bytes remain for each token–layer vector. Encoding every vector independently gives each one a tiny code and misses the opportunity to share storage across layers.

Our method, OPTIC, gives each token one quantized code shared across all layers. Layer-specific affine decoders reconstruct the different PLE vectors from that shared code. We also fit the representation to its effect after the gate and output projection, so the compression objective reflects how the model actually uses the embeddings.

At about 100× PLE compression, OPTIC retains 97.8% of BF16 E4B performance across seven benchmarks, compared with 66.3% for product quantization. Serving BF16 E4B directly from the compact codes reduces GPU memory by 29.7%, with at most 0.40% more wall time on an RTX 4090 with the same 1 GiB KV cache. It also works alongside backbone weight quantization.

Paper · Code and method walkthrough · Released 100× PLE artifact

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co