跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 13 天前AI 評分49

Infrathon:評測語言智能體在真實世界長時程 AI 基礎設施優化中的表現

Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization

AI 導讀

Infrathon 是一項評測語言智能體長時程優化真實世界 AI 基礎設施能力的基準測試;研究評估六個模型,進行 150 次、每次 24 小時的測試,合共使用超過 10,000 GPU-hours。

正文

Published on Nov 9

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

In this work, we study whether frontier agent can continually improve real-world AI infrastructure over long horizons. We introduce Infrathon, a benchmark of diverse, long-horizon AI infrastructure optimization tasks across different levels. We evaluate six models across 150 runs with 24-hour budgets, totaling over 10,000 GPU-hours. Our results show that model rankings vary with the evaluation horizon, while 135 of the 150 runs achieve further improvements during the second half of their budgets. These findings indicate that shorter evaluations can overlook both optimization gains and changes in relative model performance. We also find agents can discover solutions that outperform human-designed baselines: Claude Opus 5.5 increases the mean KV-cache compression ratio from 6.82% with the human-designed Quant-VideoGen codec to 19.86%. Analysis of agent trajectories identifies task violations, unreliable testing, and premature completion as barriers to sustained improvement. Together, Our Infrathon framework provide resources for constructing long-horizon tasks, evaluating frontier agents, and discovering good infrastructure solutions.

View arXiv page View PDF Project page Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.11995 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.11995 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.11995 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co