Odysseys2：長時程網頁智能體基準測試

Lawrence Jang (CMU)
Alexander Yue (Browser Use)
Magnus Müller (Browser Use)
Gregor Zunic (Browser Use)
Andrew Keunwoo Jang (CMU)
Younghan Park (CMU)
Mareks Woodside (CMU)
Daniel Fried (CMU)
Jing Yu Koh (CMU)
Ruslan Salakhutdinov (CMU)

Oct 27th, 2026Research

Odysseys2: a collaboration between Carnegie Mellon University and Browser Use

Odysseys2 is a long-horizon web agent benchmark from Carnegie Mellon University and Browser Use: 100 long tasks on the live web, each graded on 8 to 12 pass/fail checks. It builds on Carnegie Mellon University's Odysseys. The full paper-style write-up, with interactive charts and all 100 tasks, is at odysseysbench.com/v2. This post is the short version.

The result

The best agent fully completes 47 of 100 tasks. GPT-6 Astra (default effort) has a mean rubric score of 84.6, yet falls short on the other 53.

Fig. 1Tasks fully completed, out of 1001 task fully completednot fully completed

GPT-6 Astra fully completes 47 of 100 tasks. No other agent reaches 20, and 5 complete none.

Fully completed = every rubric check met. 15 agents, 1 attempt per task, BrowserCode harness, Odysseys2 judge, default effort unless marked high. Mean = mean rubric score, 0 to 100.

GPT-6 Astra
mean 84.6
47
Opus 5.5high
mean 71.2
18
GPT-5.6 Sol
mean 62.0
15
Grok 4.7high
mean 68.2
13
GPT-5.6 Lunahigh
mean 59.7
8
Fable 5
mean 61.5
4
Opus 5
mean 58.3
4
Grok 4.5
mean 48.2
4
GPT-5.6 Terra
mean 44.6
2
Sonnet 5
mean 42.3
2
GPT-5.4 mini
mean 24.1
0
Gemini 3.1 Pro preview
mean 10.8
0
Gemini 3.8 Flashhigh
mean 10.7
0
Gemini 3.7 Flash
mean 5.8
0
Gemini Lite
mean 1.6
0

Sorted by tasks fully completed, then mean score. Opus 5.5 is second with 18. Fable 5: content filters blocked 2 tasks, so its mean uses 98. Scores come from an LLM judge that has not been validated against a human reference.

15 agents, one 60-minute attempt per task, all in BrowserCode. Opus 5.5 is second with 18 fully completed. Five agents complete none. Full leaderboard.

Fast is not good

Fig. 2Score banked over time

At 15 minutes GPT-5.6 Luna has 37 points and Opus 5.5 has 18. By 60 minutes, Opus 5.5 and Grok 4.7 finish well above Luna.

Mean rubric score (0 to 100) from attempts finished by each minute, over all 100 attempts. BrowserCode harness, high effort, 1 dot per minute.

Cumulative mean score against minutes, BrowserCode, high effort, 100 tasks. At 15 minutes: GPT-5.6 Luna 37, Opus 5.5 18, Grok 4.7 12. At 60 minutes: Opus 5.5 71.2, Grok 4.7 68.2, GPT-5.6 Luna 58.9.

At 15 min: Luna 37 · Opus 5.5 18 · Grok 4.7 12

Grok 4.7 catches Luna at about 30 minutes and Opus 5.5 at about 36. An attempt adds its score when it finishes, not while it runs. Luna's line ends at 58.9, below its 59.7 mean, because one attempt has no recorded time.

At 15 minutes GPT-5.6 Luna leads with 37 points to Opus 5.5's 18. Opus catches up at about 36 minutes and ends at 71.2. More.

The harness matters

Fig. 3Same model, three harnesses

In the Python library Luna stays near 30. In BrowserCode it climbs to 63.0, in Pi to 71.9.

Mean rubric score, 0 to 100, for GPT-5.6 Luna on all 100 tasks, by reasoning effort. One run per setup. Python library has no max setting.

GPT-5.6 Luna mean score by effort, 100 tasks per setup. Python library: 30.2, 27.9, 33.0, 33.5. BrowserCode: 29.8, 42.0, 59.7, 61.6, 63.0. Pi: 28.0, 37.3, 59.2, 68.6, 71.9.

Harnesses hand the judge different records of the run, so these gaps mix differences in execution with differences in the evidence available for grading. Pi was tested with Luna only.

Same model, GPT-5.6 Luna: about 33 in the Browser Use Python library, 59 to 72 in BrowserCode and Pi at high to max effort. More.

Grading changes scores

Fig. 4Same attempts, two judgesOdysseys judgeOdysseys2 judge

Under the original judge every agent scores 77 to 96. Under the Odysseys2 judge the same attempts spread from 2 to 85.

Mean rubric score, 0 to 100, for 14 agents, each graded twice on the same saved attempts. Gemini 3.7 Flash: 94.9 vs 5.8.

Mean score on the same attempts, Odysseys judge then Odysseys2 judge: GPT-6 Astra 95.6 to 84.6; Opus 5.5 91.6 to 71.2; Grok 4.7 91.2 to 68.2; GPT-5.6 Sol 93.9 to 62.0; Fable 5 94.6 to 61.5; Opus 5 90.0 to 58.3; Grok 4.5 93.6 to 48.2; GPT-5.6 Terra 89.3 to 44.6; Sonnet 5 85.9 to 42.3; GPT-5.4 mini 86.5 to 24.1; Gemini 3.1 Pro preview 77.3 to 10.8; Gemini 3.8 Flash 94.5 to 10.7; Gemini 3.7 Flash 94.9 to 5.8; Gemini Lite 81.5 to 1.6.

The original Odysseys judge grades one check at a time from the action history and screenshots and does not see delivered files. The Odysseys2 judge also reads the files and zeroes flagged attempts. These differences measure disagreement between the judges; lower scores alone do not establish greater accuracy.

On the same attempts, the original Odysseys judge and the stricter Odysseys2 judge, which also reads the delivered files, disagree a lot: Gemini 3.7 Flash scores 94.9 under one and 5.8 under the other. That is disagreement, not proof of accuracy. How tasks are built, how grading works, the two judges.

Limitations

The judge is not yet validated against a reliable human reference, reward-hacking flags are model judgments, and the live web changes. See validating our judge for the full list.

Get the data

Benchmark page and interactive results: odysseysbench.com/v2
100 tasks as JSON (instruction, start URL, rubric)
Judge prompt and EVALUATION.md
Release files (tasks, rubrics, scores, judge prompt): github.com/ljang0/odysseys2-benchmark (MIT license)
Original Odysseys paper: arXiv 2604.24964
Related: web agent benchmarks, the bitter lesson of browser agents

To cite Odysseys2:

@misc{jang2026odysseys2,
  title        = {Odysseys2: A Web Agent Benchmark Based on Real User Requests},
  author       = {Jang, Lawrence and Yue, Alexander and M{\"u}ller, Magnus and
                  Zunic, Gregor and Jang, Andrew Keunwoo and Park, Younghan and
                  Woodside, Mareks and Fried, Daniel and Koh, Jing Yu and Salakhutdinov, Ruslan},
  year         = {2026},
  howpublished = {\url{https://odysseysbench.com/v2/}}
}

Thanks to our co-authors at Carnegie Mellon University: Lawrence Jang, Andrew Keunwoo Jang, Younghan Park, Mareks Woodside, Daniel Fried, Jing Yu Koh and Ruslan Salakhutdinov.