五個模型以 $0.06 至 $0.45 嘗試建構同一款應用：多數失敗源於測試與生成器

We're building an app designer on Kumiko: you describe the app you need, a model designs it, and you get a running app back. The site is up at studio.kumiko.rocks, but the designer isn't open yet. Before it opens, I'm measuring how well it works on one case: a maintenance app for a metalworking shop, with machines, maintenance plans, inspection protocols with measured values, reminder mails, a PDF per protocol and an audit export. Each run has three steps. The model builds the app, then handles two change requests: "calibration every 6 months instead of 12" and "photos on the inspection protocol".

The model doesn't write the app code. It submits a spec as JSON, and a generator turns it into entities, screens, navigation, translations, seed data and an integration test. The model only writes the bodies of custom handlers in TypeScript. After each submission a check runs: typecheck, boot, a dev-server smoke test in a browser, and a comparison against the design we approved for this case. That comparison has 63 points (entities, screens, actions, read and write rights per role). Every open point goes back to the model as a finding.

Green, and the app didn't work

The first run in this mode, on Sonnet 5.5, passed all three steps for $1.16, with 63 of 63 points. Then the model listed what it had left out. The PDF actions only confirmed and never produced a file. A measurement outside its limits wasn't flagged. "Continued operation: no" didn't lock the machine. There were no overdue counters and no reminder mail. The 63 points only checked structure, and a maintenance manager would have found the gaps on the first day.

So every check now also runs seven behavior tests taken from the approved design: deadlines, the limit check, the lock, the condition that must be written down when a machine keeps running with one, the reminder, and two PDF tests.

Five models

Same case, same prompt, same tests. The OpenAI models ran through OpenRouter's EU endpoint.

The easy reading is that Sonnet wins and luna is cheap and useless. When I went through the findings one by one, most of the red came from our side.

What was ours

The limit check failed for every model. A measurement outside min and max should give notOk, and the test got undefined. The model saw only the assertion text, with nothing pointing to the spec construct for exactly this case (derived inRange). terra got everything else right and then stopped trying. Sonnet only fixed it during the first change request.

The behavior tests create records with fixed values, like asset.kind: "machine" and continuedOperation: "withCondition", and assume every other field is optional. None of that was in the prompt. sol made the inspector and the inspection date required and called the condition value conditional, so the test could never create a protocol, and every test that needed one failed with it. luna had no machine among the asset kinds and got 0 of 7 in both change requests.

Two boot rules cost every model one or two checks. One fires when a list has an edit screen behind it but no action to open it. Our generator only looked for detail screens, so it never added that action to the maintenance plan list. The model got a boot error about code it hadn't written.

Some of it was the models. Most of the 21 type errors across all runs were luna's, from new Date() called on a date-only value in its own handler code. Qwen never answered: 20 minutes of silence with one provider, and the first call ran into the 10-minute limit twice with another, while short test calls came back in 1.6 seconds. I don't know yet whether the provider, the thinking phase or the 57k-token request is the cause.

The fixes

• The generator adds the open action itself and counts card columns with the framework's own function, so generated code can't trip either boot rule anymore.
• A failing behavior test now names the spec construct behind it: limit check to derived inRange, lock to approvals with finding and cascade, condition to requiredWhen.
• The prompt lists the test records with their fields and values.

Sonnet again

$0.322 for all three steps, against $0.454 and a red build before. No boot errors, and the limit check was right in the first submission. The first two checks of the build still failed. The audit PDF action and the managers' write rights were missing, as they have been in almost every run since the first one. The condition test failed until the finding pointed at requiredWhen.

I haven't rerun the other four models yet. That's part 2.

What this run doesn't prove

The findings now name the construct, and for the limit check that's close to handing over the answer. These hints come from the design we approved for this one case. In the real designer nobody has approved a design behind the user's description, so nobody can write that hint. The second run shows that a spec, a check and precise findings get to green. It doesn't show how well a model gets the domain logic right on its own. Prompt and finding texts also changed between the two runs, so the comparison with the first table only goes so far.

An open question

How specific do you make the feedback for an agent in a test loop? "Expected notOk, got undefined" left every model stuck. "Use derived inRange" fixed it at once, and it's almost the solution. If you run agents against tests: where did you draw that line, and how do you check that your tests aren't just asking for your own solution back?