跳到正文
Hacker News 熱門(buzzing.cc 中文翻譯)·· 6 小時前AI 評分40

Astra 6.2 和 Sol 6.2 在 Simple Bench 測試中表現不佳

Astra 6.2 和 Sol 6.2 在 Simple Bench 测试中表现不佳

AI 導讀

SimpleBench 涵蓋 200 多條問題,測試空間與時間推理、社交智能及語言對抗穩健性;9 名參與者的小樣本非專業人類基準得分為 83.7%,高於所有受測 LLM。Claude Fable 得分為 81.9%,是受測模型中的最高分。

正文

We introduce SimpleBench, a multiple-choice text benchmark for LLMs where individuals with unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions). For the vast majority of text-based benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on our small sample of nine participants,

outperforming every tested LLM, including today's top model, Claude Fable, which scored 81.9%

. While we expect model performance to improve over time, the results of SimpleBench confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet.

For a deeper dive into our results and our methods, check out the full technical report here.

Use all of these models on the Simple Bench app - LMcouncil.ai

來源:Hacker News 熱門(buzzing.cc 中文翻譯) · simple-bench.com