跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 20 天前AI 評分35

FLAMIA:面向 Blackwell 的快速且因果安全訓練注意力

FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell

AI 導讀

FLAMIA 提出面向 Blackwell 的注意力核心,透過 MXFP8 及混合精度訓練路徑加速前向與反向計算,並防止因果資訊洩漏。涵蓋量化的 MXFP8 路徑,在 B200 的加速幅度為 1.10–1.40 倍,在 B300 為 1.06–1.42 倍。研究亦在 Llama 類預訓練模型上驗證訓練穩定性及品質維持。

正文

Published on Nov 2

Authors:

,

Abstract

FLAMIA accelerates Transformer attention on NVIDIA Blackwell GPUs via MXFP8 and mixed-precision kernels that reduce quantization overhead, prevent causal leakage, and support variable-length packed documents.

Attention remains a substantial computational cost in Transformer training, particularly at long context lengths. NVIDIA Blackwell GPUs provide high-throughput low-precision matrix multiplication, but exploiting it in attention requires addressing quantization overhead, causal information leakage through shared scales, and numerical inconsistencies between the forward and backward passes. We present FLAMIA, a family of attention kernels for Blackwell with MXFP8 and mixed NVFP4/FP8 training paths. Our design combines token-block scaling that reuses quantized operands across matrix orientations, targeted safeguards against scale-induced leakage, and consistent forward and backward computations. Fused quantization and architecture-specific kernel optimizations reduce overhead, while document-local scaling and workload-aware scheduling support variable-length documents directly in packed layouts without repacking. Across various causal packed-document workloads, the MXFP8 path accelerates the combined forward and backward attention call, including quantization, by factors of 1.10--1.40times on B200 and 1.06--1.42times on B300 relative to the fastest evaluated baseline for each workload. We validate training stability and quality preservation on pretraining Llama-style models. These results show how jointly designing quantization, training numerics, and document scheduling translates low-bit hardware throughput into practical attention acceleration.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2611.03344

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.03344 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.03344 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.03344 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co