跳到正文
Hacker News 熱門(buzzing.cc 中文翻譯)·· 14 小時前AI 評分58

Gemini 4 Neon 看來可與 Opus 5.5 競爭

Gemini 4 Neon 看起来像是 Opus 5.5 的竞争对手

AI 導讀

作者假設 Gemini 4 Neon 的表現與 Gemini 4 Carbon 相同,按其 127 tokens per second 的速度推算,Neon 在 Agent Duel 排行榜的 Elo 為 1,066,略低於 Opus 5.5 的 1,078。

正文

TLDR: we project Gemini 4 Neon’s performance based on what a faster Gemini 4 Carbon might score and find it equivalent to Opus 5.5

Gemini 4 Carbon is the best model on the Agent Duel leaderboard if you exclude early voting. Carbon’s main problem is that it is slooooow.

Today, Gemini 4 Neon was released, and looks to be an incremental benchmark improvement with a very large speed improvement. Earlier today, it was being served extremely fast via OpenRouter: 127 tokens per second. This is way faster than Carbon at 46 tokens per second.

This got me wondering, could Neon be not just a good model but the best model?

Modeling our leaderboard

To investigate, I modeled the win rate in our leaderboard. The best model has three components, model identity, speed and whether the user voted early. My initial modeling did poorly, so I added an interaction between speed and early voting. This ended up working well.

Win rate vs. Projected win rate

And surprisingly, speed seems to be pretty much as important as model identity (another blog on that soon).

Feature importance

Projecting Gemini 4 Neon performance

If you assume the same performance as Gemini 4 Carbon (which is reasonable given even better benchmark scores), and 127 tps, we would project an Elo on the Agent Duel leaderboard of 1,066, just below Opus 5.5 at 1,078 in our later analysis snapshot.

Synthetic Neon Elo projection

The 90% interval for the projected Elo difference between Gemini 4 Neon and Opus 5.5 is -94 to +68, with Neon projected to win 48.3% of the time against Opus (counting ties as half a win).

來源:Hacker News 熱門(buzzing.cc 中文翻譯) · micjm.com