跳到正文
NVIDIA Technical Blog:Agentic AI / Generative AI· Timmy Liu·· 4 天前AI 評分15

NVIDIA 如何協同設計稀疏注意力,提升長上下文推理效率

Co-Designing Sparse Attention for Efficient Long-Context Inference

AI 導讀

NVIDIA 探討如何協同設計稀疏注意力,以提升長上下文推理效率。隨着 Agent 工作負載、長文件和大型程式碼庫推動上下文長度由數萬個 token 增至數十萬個 token,注意力機製成為 Transformer 層中成本最高的部分;在 128K token 時,密集註意力可佔用大部分預填充時間。

正文

Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths,...

Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths, attention becomes the most expensive part of a transformer layer. Dense attention scores every query against every key, so its cost per query grows with the context length N, and at 128K tokens it can take most of the prefill time in a…

Source

來源:NVIDIA Technical Blog:Agentic AI / Generative AI · developer.nvidia.com