跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-02AI 評分34

Beyond the Parameter Monolith: Reconstructive Memories, Executable Skills, and Residual Assembly for Language Models

Beyond the Parameter Monolith: Reconstructive Memories, Executable Skills, and Residual Assembly for Language Models

AI 導讀

這項研究探索以無注意力機制的 Multi-Mesh 語言模型及模組化核心,分開研究語言模型的記憶、程序執行與上下文計算。模型以約 28.6 billion 個預測目標訓練;模組化核心約有半十億個參數,記憶在讀取審核中的 token 重構準確率約為 75%。研究團隊已釋出檢查點及部分研究程式碼,供檢視和進一步實驗。

正文

What changes when memory, exact execution, and contextual computation can be developed and updated separately?

This research grew out of experiments showing that useful language modeling can emerge while token input coordinates remain fixed. That separation between token identity and learned computation prompted a broader question: can internal representations and independently constructed components interact through an explicit shared interface? The finite-element analogy provides a design inspiration - local components, representations at different granularities, and explicit assembly - not a claim that language follows a physical equation.

The work explores these ideas through an attention-free Multi-Mesh language model trained on approximately 28.6 billion prediction targets, followed by a separate modular core that integrates reconstructive document memories and deterministic VM results through typed residual assembly. The goal is to investigate alternative organizations of language computation, not to claim superiority over Transformers.

The experiments deliberately proceed with imperfect components, including a roughly half-billion-parameter core and memory achieving approximately 75% token reconstruction accuracy in the reported readout audit. Controlled attachment tests examine how selected evidence affects predictions while the core weights remain unchanged. Negative results help distinguish separate challenges: storing information, retrieving evidence, executing a procedure, and rendering an answer.

These are research prototypes for investigating interfaces and operating modes, not a finished general-purpose system. Checkpoints and selected research code are released to support inspection and further experiments. Independent tests, simpler baselines, and discussion of what these interfaces enable - or fail to enable - are especially welcome.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co