按檔案大小和每項已解決任務能耗選擇量化 Qwen2.5-Coder 模型
Choosing Quantized Code LLMs by File Size and Energy per Solved Task
研究按 Ollama 列出的檔案大小及每項已解決任務的能耗,為量化 Qwen2.5-Coder 提供選型指南。測試涵蓋 24 個 GGUF 檔案,並以 542 項 EvalPlus 任務評估;在 1 GB 和 2 GB 檔案中,指南分別選出 1.5B 和 3B 的 q4_K_M。7B-q2_K 在解題數上未顯著勝過 3B-q4_K_M,但每項已解決任務的能耗高出 2.03 倍。
Published on Nov 13
Authors:
Abstract
A developer running a code assistant on a laptop downloads one model file, and local inference moves the energy of every request onto that machine. A common rule says a larger model at lower precision tends to beat a smaller one at higher precision. We ran 24 GGUF files of Qwen2.5-Coder at four sizes on a fanless MacBook Air M2 and priced each in energy per solved task on 542 EvalPlus tasks. The result is a selection guide indexed by Ollama's listed file size, with an evidence label on each pick. At 1 and 2 GB it picks the q4_K_M files of the 1.5B and 3B models, both on the correctness and energy frontier under every accounting. At 3 GB as listed, the larger-model rule picks 7B-q2_K, the only measured 7B file that fits, and there it does not pay. That file brings no significant gain in solves over the 2 GB pick, 3B-q4_K_M, and the 90% CI limits any 3B-q4_K_M deficit to 2.4 points. Yet 7B-q2_K needs 2.03 times the energy per solved task on CPU, GPU and memory rails, or 1.66 to 1.81 times on whole-system power. On HumanEval+, five alternating session pairs give a ratio of 1.925 against 1.956 in the main grid. At 8 GB, 7B-q6_K buys a significant gain over the 2 GB pick at a higher energy price. From 1.5B up, q4_K_M trails the highest precision measured at its size by less than 3 points (90% CI). We release per-task results, power logs and the analysis script.
View arXiv page View PDF Project page Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.17605 in a model README.md to link it from this page.
Datasets citing this paper 1
aditiPatodiya/code-llm-gguf-energy-per-solve
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.17605 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co