跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 4 天前AI 評分60

語言模型的一般智能可彌補特定訓練資料缺失

General progress substitutes for specific training data in language models

AI 導讀

研究發現,語言模型的一般智能可彌補特定訓練資料缺失,資料過濾會延遲相關技能形成,而非永久排除能力。研究以棋類、分子字串、括號語言及網頁文本等任務隱去特定技能資料,並發現小模型表現可用來預測大模型技能延遲時間。另有 7-billion-parameter 模型在接觸被隱去的程式語言資料後恢復相關能力;模型智能越高,恢復技能所需資料越少。

正文

Developers can limit the capabilities of AI models by removing certain topics from training data, but they’ve never known for how long this slows development. Now, we’ve found that general intelligence compensates for these exclusions.

We hid specific skills from AI models trained on chess, Othello, molecular strings, bracket languages, and web text. We could predict how long a specific skill would be delayed in a large model by observing small ones, but what surprised us most was how close these models came to developing skills without any data. A chess model with no data to respond to check still ranked legal moves almost as well as a trained model!
When we gave the models the withheld data, they acquired the withheld skill with a tiny fraction of the data needed by an untrained model. The more intelligent the model, the less data was needed! If we gave the models an equally difficult, unrelated rule, they were much slower to learn it. We found the same pattern at scale: a 7-billion-parameter model had a programming language withheld from its training data, but recovered it when exposed.

We conclude that filtering training data doesn’t exclude capabilities forever but instead delays them. As model intelligence rises, the data needed to develop these skills diminishes.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co