如何衡量免費編碼助手修復缺陷的重試成本

A team blocks ninety minutes to answer one question: can a free coding assistant fix a real defect in their repository? The first patch arrives in four minutes and reads confidently, so they apply it, watch the test suite go red, and ask again with a sharper prompt. By minute seventy they have spent six attempts on one accepted change, and nobody wrote down the number six. That missing number is the whole result of the exercise, because free access does not remove cost, it relocates cost from the invoice to the review queue.

The shift is easy to miss. Paid tiers meter tokens, so the visible constraint is spend, while free tiers meter nothing visible at all and the constraint quietly becomes human attention. Every rejected patch costs a developer a read, a diff review, and a decision about whether to retry or abandon, and those minutes do not appear in any usage dashboard. A ninety-minute spike that records only the final diff therefore answers the wrong question, since the interesting quantity is how much review labour the assistant consumed per unit of accepted work.

One metric that survives the time box

The retry multiplier is attempts divided by accepted patches, and it converts a vague impression of flakiness into arithmetic. An assistant that needs two tries per good patch has a multiplier of 2, which is usually tolerable for a backlog of small defects. An assistant that needs seven tries has a multiplier of 7, and the practical question becomes whether seven review cycles still beat the twenty minutes a human would have spent on the same fix.

Accepted patches need a definition that cannot drift during the spike, or the number is worthless. A patch counts as accepted only when the target test flips from failing to passing, the rest of the suite stays green, and a second person would merge it without further edits. Anything weaker, such as a patch that "looks reasonable" or compiles but skips the regression test, must count as a rejection even when it feels harsh.

A scaffold for recording attempts

The harness below is a sketch rather than an executed benchmark. Fill in the request shape from the provider's current documentation, because endpoint paths and model identifiers change faster than any article can track.

#!/usr/bin/env bash
# spike.sh -- one hypothesis, one repo, ninety minutes, a fixed attempt budget.
set -euo pipefail

REPO=${REPO:?path to the repository under test}
TASK=${TASK:?one-line task description}
BUDGET_MIN=${BUDGET_MIN:-90}
MAX_ATTEMPTS=${MAX_ATTEMPTS:-8}
DEADLINE=$(( $(date +%s) + BUDGET_MIN * 60 ))

RUN_ID=$(date -u +%Y%m%dT%H%M%SZ)
OUT="runs/${RUN_ID}"
mkdir -p "$OUT"
printf '%s\n' "$TASK" > "$OUT/hypothesis.txt"

attempt=0
while [ "$attempt" -lt "$MAX_ATTEMPTS" ] && [ "$(date +%s)" -lt "$DEADLINE" ]; do
  attempt=$(( attempt + 1 ))
  python3 attempt.py --repo "$REPO" --out "$OUT" --n "$attempt" || true
done

The per-attempt script isolates each try in a disposable worktree, so a bad patch can never contaminate the next one. The free model and the free server enter here as the thing under test: the operator of this account reports that the open-source MonkeyCode project offers free model access and a free server option, and that page is the correct place to confirm the allowance and terms in force on the day you run the spike. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

"""attempt.py -- one attempt, one worktree, one row of evidence."""
import argparse, json, os, pathlib, subprocess, time

BASE  = os.environ["MONKEYCODE_BASE_URL"]  # read this from current docs
KEY   = os.environ["MONKEYCODE_API_KEY"]
MODEL = os.environ["MONKEYCODE_MODEL"]     # whatever the free tier serves today

def sh(*cmd, cwd=None):
    return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)

def run(args):
    wt = pathlib.Path(f"/tmp/spike-wt-{args.n}")
    sh("git", "-C", args.repo, "worktree", "add", "--force", str(wt), "HEAD")
    started = time.time()
    patch = call_model(args.repo)                 # your transport goes here
    applied = sh("git", "apply", "--whitespace=fix", "-", cwd=wt, input=patch).returncode == 0
    suite = sh("python", "-m", "pytest", "-q", cwd=wt)
    accepted = bool(applied and suite.returncode == 0 and target_test_passes(wt))
    row = {"attempt": args.n, "applied": applied, "accepted": accepted,
           "wall_s": round(time.time() - started, 1)}
    with open(f"{args.out}/attempts.jsonl", "a") as f:
        f.write(json.dumps(row) + "\n")
    return row

Two lines of arithmetic then replace the post-hoc storytelling that usually surrounds assistant trials. The multiplier is the attempt count divided by the accepted count, and the wall-clock cost per accepted patch is total elapsed seconds divided by the same denominator. Both numbers should be written down before the room disperses, alongside the human estimate for the identical fix, because the comparison against that estimate is the only thing that makes the ratio meaningful.

Reading the numbers at minute ninety

A kill result is not a verdict on the assistant, and treating it as one is how teams waste the next ninety minutes. If the regression test never failed in the first place, the spike measured nothing at all, and the honest conclusion is that the repository was not ready to be measured. The kill memo should therefore be three lines long: the hypothesis, the observed multiplier, and the specific next question that the evidence could not settle.

Where the method breaks

A flaky suite inflates the multiplier without telling anyone why, so the baseline test run must be executed three times before the spike starts. Small denominators make the ratio unstable, and a single accepted patch out of three attempts should never be reported as a decisive improvement. Free tiers and model rosters rotate, which means a multiplier measured this month describes this month only, and any team that needs a pinned model version for a year should treat the result as advertising rather than engineering evidence. Private or regulated code should not leave the network for a hosted endpoint at all, and the free server option does not change that boundary, since it is a place to run the harness rather than a licence to export source.

Teams whose backlog lacks a failing test should not adopt this protocol, because there is nothing to accept or reject and the multiplier becomes a measure of reviewer mood. Teams that cannot spare a second reviewer should skip it too, since acceptance by the author alone is exactly the bias the whole exercise exists to remove. For everyone else, the free allowance is worth one properly instrumented ninety minutes, and the project page is the sensible first stop before the clock starts.