Replit 發佈 ViBench 2.0 長期軟件開發基準測試

Today we’re releasing ViBench 2.0, a benchmark for long-horizon software development. It joins the Specialized Intelligence Index (SII) from Fireworks [1], our partner for open-weight model inference and training. The SII brings together practitioner-built benchmarks to compare open, closed, and specialized models on real work tasks across industries.

Most coding benchmarks, like DeepSWE [2], hand an agent a well-defined task inside an existing codebase. In ViBench 2.0, the agent starts with an empty project instead. It then builds a whole product one feature at a time, modeled on products like Uber, GitHub, or Google Docs.

ViBench 2.0 has seventeen products, 241 feature requests, and 544 test plans. The best models, GPT-6.1 Sol, GPT-6 Astra, and Opus 5.5, pass all test plans in only 49⁠-⁠65% of their builds.

Why we built ViBench 2.0

We modeled ViBench 1.0 [3] on what Replit users were building at the time, like collaborative whiteboards and resume builders. People now come to Replit to build entire businesses, which often include full-featured software products. ViBench 2.0 measures whether models can build products like these.

The agent builds a first version of the given product from a short spec, then adds up to eighteen features on top of its existing code. This design forces it to choose the schema and routes before knowing what’s next, so later features can be harder than earlier ones. ViBench 1.0 found that models score lower when they add a feature to their own code, a setting the paper dubbed Vibe-on-Vibe. ViBench 2.0 extends this setting even further by emulating the evolution of a product. Eight of the products are public on GitHub [4] in the Harbor format [5]; the remaining nine are held out.All results here can be rerun with one command from release v2.0.6, on our OpenHands-based harness [6].

Along the way, the agent has to handle:

Clean deployment: tests launch builds on an empty database, so a build that relies on leftovers fails them all.
Existing data: in the community chat product, messages that mention someone must show their new name after they change it.
Invariants: in the ride-hailing product, split-fare shares must always add up to exactly what the ride costs, to the cent.
Stale state: in the code-hosting product, an editor opened before someone else saved must never erase their newer change.
Concurrency: in the marketplace product, two return forms must never refund the same unit twice.

Once the last feature is built, a grading agent runs the test plans in a real browser, clicking, typing, and checking the result.

We use Opus 5.5 as the grader because it made the fewest mistakes of the models we tested. To check it, we planted 34 bugs one at a time in working builds, and it never passed a plan while the bug was visible. When GPT-6.1 Sol re-graded 477 of Opus’s passes, 99.2% were confirmed correct. We also saw no sign that Opus favors builds from Claude models.

What a test plan checks

Feature requests describe what a user expects, not how to build it. For example, part of the design tool’s undo feature request:

Add undo and redo for changes to objects: creating, editing, moving, deleting, grouping and ungrouping, and renaming or rearranging layers in the layers panel. … Each person has their own undo history, so undo only takes back your own changes, never a collaborator’s. Making a new change clears what you could redo.

A test plan then checks that rule the way a user would hit it. Alice and Bob take turns editing the same rectangle, and then Alice presses undo:

1. Alice: create rectangle "R" at (100,100), 100x100, fill #CCCCCC.
2. Alice: move R to (300,100).
3. Bob: rename R to "Rocket".
4. Alice: change R's fill to #00AA00.
5. Bob: set R's opacity to 50%.
6. Alice: Undo. Read R on both screens.
...
Verify:
- After 6, R's fill is #CCCCCC while its name is still "Rocket", its opacity still 50% and its position still (300,100).

Later steps repeat the check after Bob moves, ungroups, or deletes shapes. GPT-6.1 Sol, GPT-6 Astra, and Opus 5.5 fail this plan in eight of their twelve builds. The request and the plan are both public [4].

How the models compare

The official ViBench 2.0 results come from Fireworks’ runs: four builds of the seventeen products per model, on the same harness and settings.All builders run at medium effort and temperature 1.0, with the same limits. The README lists the full settings.

GPT-6.1 Sol leads with a pass@1 of 65%, followed by GPT-6 Astra at 60% and Opus 5.5 at 49%. All three pass more than 90% of their test plans, so most of their failed builds miss by only a few plans. Sol costs about $12 per build, and Astra about $90. DeepSeek V4.1 Flash passes 77% of its test plans for about $2.50 per build, though no open-weight model passes a full build. Opus 5.5 finishes a build in about 70 minutes on average, roughly 2.5 times faster than Sol. Opus produces only about 6% fewer tokens, so the gap mostly comes from API provider speed.

Scores across GPT generations

To see how scores change across model generations, we ran earlier GPT models ourselves on the eight public products, separately from the official results.

Pass@1 rises across generations, from 0% for GPT-5.4 to 31% for GPT-6 Sol and 53% for GPT-6.1 Sol, and plan pass rate climbs from 42% to 95%. Scores on the public products run lower than on all seventeen, so Sol’s 53% here sits below its 65% overall. GPT-6.1 Terra, OpenAI’s newest model at time of writing, passes 47% of builds at about half the cost of GPT-⁠6.1 Sol.

Where the top models fail

The plans the top models miss are often small details a real user would hit. Examples from four of the public products:

Code-hosting product: a draft pull request never shows its merge conflict.
Document editor: nobody can comment on a word inside a table cell.
Design tool: pressing undo can take back a teammate’s change, not just your own.
Community chat product: a server owner with no roles can’t moderate anyone.

Most test plans check how features work together. Two failure modes stand out. In about one in 40 of the top models’ builds, the product never starts on the empty database the agent was told to expect, because later schema code references a table before the statement that creates it. More often, builds break where features intersect, as in the examples above. Nearly three-quarters of those models’ failed builds fail only on these intersections.

How the builds look

ViBench doesn’t grade design, but design shapes how people judge a product. Since this is subjective, we’ve included the screenshots below alongside our read.The screenshots show builds from our own runs.

Models from the same family produce similar designs, whatever the product. The specs describe what a product does, not how it looks, so models fall back on their own taste. GPT models tend to add illustrations and big headlines. Claude models tend to build plainer screens that stay closer to the products they’re modeled on. With a plain harness and no design guidance in the prompts, even choices like a dark theme come down to the model. In the figure’s code-hosting tab, Sonnet 5.5 picks one in three of four builds, and in none of the other tabs. To our eyes, the best-looking builds were no more likely to pass than the plainer ones.

What’s next

Today’s best models build most of a product correctly on ViBench 2.0, but building it one feature at a time exposes their limits. New features have to fit code the agent wrote earlier, and that is where builds often break.

We expect scores like these to reach more cost-efficient models, including open-weight ones. GPT-6.1 Sol, which leads at about $12 per build, is an early sign of this trend.

A detailed write-up of the methodology is coming soon. For the latest results, see the SII leaderboard and vibench.ai.

Acknowledgements

Thanks to the Fireworks team: Vivek Chauhan and Charlie Gu for leading the partnership that brought ViBench to the SII, and Arleen Liu and Sandeep Singh for running the benchmark. Thanks also to Georgian, our collaborator on ViBench since the start, and in particular to Kartik Gupta for building the Harbor adapter, Aditya Shukla and Scott Cui for their support, and Pashootan Vaezipoor for shaping the project’s direction. Finally, thanks to the rest of the AI team at Replit for their contributions to ViBench.

References

Appendix

Product names

ViBench’s products are modeled on public ones for evaluation only. Product names are trademarks of their owners, who are not affiliated with ViBench and do not endorse it.

Metrics

ViBench reports four metrics:

Pass@1: the share of builds in which every test plan passes.
Plan pass rate: the share of test plans where all steps pass, averaged by product.
Average cost: the model’s cost for one build, in USD at list prices.
Average build time: wall-clock minutes for one build, from the first version to the last feature.

Ranges are 95% intervals, computed by vibench score [4], which uses exact binomial intervals near 0%. All models run in one agent harness built on OpenHands [6], so a model tuned for a different setup may score lower here.