在信任免費模型編碼智能體前，如何用 90 分鐘和 1 個假設進行測試

Adoption of AI coding tools still runs on vibes. A demo lands, a team enables the tool, nobody measures the outcome. Meanwhile the models keep improving.

A time-boxed spike replaces vibes with evidence. Ninety minutes, one hypothesis, and a verdict you can defend.

The hypothesis goes in writing first

A spike without a written hypothesis becomes a demo. Write one falsifiable sentence before the clock starts.

Bad: "The free model access will make us faster." That cannot be false, so it is useless.

Good: "Ten frozen repo tasks pass with one prompt each, 8/10 correct, p95 under 20 seconds."

Commit that sentence with a timestamp. It is the artifact the spike gets judged against.

Budget the ninety minutes

Time-box means the clock is a constraint, not a suggestion. Split the window before you start.

If a stage overruns, stop the spike. Extending the clock is how a spike quietly becomes a project.

Set stop rules before minute zero

Stop rules protect the window. Pick your own thresholds, but pick them early.

1. Abort scoring if fewer than ten tasks are frozen by minute 10.
2. Treat any provider error as a task failure, never as a skip.
3. Kill at minute 70 if the pass rate is below 60 percent.
4. Kill if p95 latency is more than twice your interactive bar.
5. Never re-run a task to get a nicer answer.

Rule 2 matters most. Fail-closed scoring is what separates a spike from a highlight reel.

The harness

Keep the harness provider-agnostic. One function talks to the model; everything else is measurement.

# spike/harness.py
import json
import statistics
import time
from dataclasses import dataclass
from pathlib import Path

@dataclass(frozen=True)
class Task:
    name: str
    prompt: str
    must_contain: tuple
    must_not_contain: tuple = ()

TASKS = json.loads(Path('spike/tasks.json').read_text())

def complete(prompt: str, model: str):
    """Return (text, seconds). Plug in ONE provider here."""
    raise NotImplementedError

def score(text: str, task: Task) -> bool:
    if any(bad in text for bad in task.must_not_contain):
        return False
    return all(good in text for good in task.must_contain)

def main() -> None:
    out = Path('spike/raw.jsonl').open('w')
    passes, latencies = 0, []
    for raw in TASKS:
        task = Task(**raw)
        start = time.perf_counter()
        try:
            text, secs = complete(task.prompt, model='<your-model>')
            ok = score(text, task)
        except Exception as exc:            # fail closed
            text = f'ERROR: {exc!r}'
            secs = time.perf_counter() - start
            ok = False
        latencies.append(secs)
        passes += int(ok)
        out.write(json.dumps({'task': task.name, 'ok': ok,
                              'secs': round(secs, 2),
                              'text': text[:2000]}) + '\n')
    out.close()
    latencies.sort()
    p95 = latencies[min(len(latencies) - 1, int(len(latencies) * 0.95))]
    print(f'pass={passes}/{len(TASKS)} '
          f'p50={statistics.median(latencies):.1f}s p95={p95:.1f}s')

if __name__ == '__main__':
    main()

Frozen tasks are plain data. Keep acceptance checks mechanical, not aesthetic.

[
  {
    "name": "retry-fetch",
    "prompt": "Add bounded retry with jitter to fetch_user.",
    "must_contain": ["sleep"],
    "must_not_contain": ["while True"]
  }
]

The harness above is a plan, not a published result. I have not run it against your repository. The numbers have to come from your tasks.

Where free access changes the economics

A spike is disposable work. Buying a seat to answer one question is a poor trade. Free tiers fit the window.

MonkeyCode is the project that prompted this write-up. Its maintainers describe it as an open source project, and the operator supplies two availability claims: free model access and a free server option. The operator also reports a free token allowance on the order of 10 million tokens.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I have not benchmarked any of those claims. Quotas, model rosters, and server limits change, so read the current terms yourself. Design the spike so a quota change cannot break it.

Concrete ways to use it inside the window:

• Point complete() at the free model access and keep every task frozen.
• Run the harness on the free server option so laptop sleep cannot corrupt a long run.
• Log raw text, so re-scoring later needs no re-run.
• Spend minutes 70-85 on scoring, not on re-prompting.

Turn evidence into a verdict

Three verdicts are allowed: ship, extend once, kill. Anything else is scope creep.

The extend-once row is deliberate. One extension keeps the spike honest and still bounded.

Limitations and who should skip this

• Ten tasks is a smoke test, not a benchmark. Never quote the pass rate as a product score.
• The result covers one task class on one day. Re-run when the model or prompt changes.
• If code cannot leave your network, confirm where inference runs before task one.
• Shared free tiers have variable latency. Measure it instead of assuming it.
• If you cannot freeze ten tasks in ten minutes, your tasks are not ready.

Skip this approach if you need statistical power, if your repo lacks mechanical acceptance checks, or if one wrong answer costs money.

What a kill is worth

A kill result saves weeks of tool churn and a migration you would have regretted. It is a deliverable, not a failure.

Write the verdict in the repo next to the harness. Next quarter, someone will ask the same question.

If you want a free tier for your first spike, the MonkeyCode project page is a reasonable place to start. Bring your own ten tasks and your own thresholds.