跳到正文
Hacker News 熱門(buzzing.cc 中文翻譯)·· 15 天前AI 評分45

Memorilla:面向大型語言模型(LLM)的潛在語義記憶

Memorilla:面向大型语言模型(LLM)的潜在语义记忆

AI 導讀

Memorilla 是一套供凍結式 LLM 使用的記憶系統,將整個文件集中的資訊整理成固定數量、依查詢而定的記憶 token。使用 16 個記憶 token 時,它在八項基準測試中的五項勝過 Mem0 和 MemoRAG;在 FactKG 上準確率達 84.33%,高於完整上下文推理的 80.43%,而表示來源文件所需的 token 少 108 倍。

正文

View PDF HTML (experimental)

Abstract:Memory enables large language models (LLMs) to use information beyond their parameters, such as past user conversations, long documents, or an agent's earlier actions. For a frozen LLM, access to this accumulated knowledge is limited to what a memory system places in its context window. However, gathering and combining relevant information needed for each query while keeping the model's context short remains challenging for existing systems. Here we introduce Memorilla, a memory system that gathers information from an entire document collection into a fixed number of query-dependent 'memory tokens' for a frozen LLM. Memorilla encodes documents (such as user chats, book chapters) once into reusable embeddings, then learns to retrieve and place query-specific information in the LLM's input embedding space. Using only 16 memory tokens, Memorilla achieves the best performance against baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document and knowledge-intensive question answering. Notably, on FactKG, it reaches 84.33% accuracy versus 80.43% with full-context inference, while using 108x fewer tokens to represent the source documents. Furthermore, Memorilla can facilitate continual learning in agentic systems where the document count (the agent's past turns) increases throughout the trajectory. In a TextWorld environment, we show that Memorilla reduces the agent's tendency to wander between rooms it has already explored, and improves its average success rate from 18.6% with full-history in-context learning to 24.2%. Overall, Memorilla aids frozen LLMs in accumulating and using knowledge effectively across long-context interactions and takes a step towards building continually learning AI systems.
Comments: Website: this https URL
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2610.21375 [cs.LG]
  (or arXiv:2610.21375v3 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2610.21375

arXiv-issued DOI via DataCite

Submission history

From: Vignesh Kothapalli [view email]
[v1] Thu, 15 Oct 2026 06:55:55 UTC (3,660 KB)
[v2] Tue, 27 Oct 2026 04:28:12 UTC (3,660 KB)
[v3] Tue, 3 Nov 2026 07:03:31 UTC (3,660 KB)

來源:Hacker News 熱門(buzzing.cc 中文翻譯) · arxiv.org