先測任務門檻，再測並發延遲：比較免費、付費與自架 AI 編碼方案的兩項測試

Task Bar, Then Queue: Two Trials That Decide Free Tier vs Paid vs Self-Hosted

You cannot compare a free model tier with a self-hosted box until you have two numbers, and neither of them is a price. The first is the highest task difficulty the free tier actually clears on your own repository. The second is the latency that tier holds when four requests are in flight at once, because agents rarely run alone. Everything else — token allowances, hourly rates, GPU rentals — is arithmetic you do afterwards, and arithmetic is the easy part.

So why do so many comparisons start with the invoice? Because a price list is public and a task bar is not, and public numbers feel like evidence. They are not evidence about your code, and they never were.

Why the cost table answers the wrong question first

Most "free vs paid vs local" posts I read benchmark the same task: write a small function, explain it, maybe fix a typo. That task clears on nearly every tier, so the comparison collapses into price per million tokens. Meanwhile the work that decides your month is the cross-file rename, the flaky test, the migration that touches eleven call sites. A tier that fails there does not save you 100% of anything; it saves you the tokens you spend discovering that it failed. Is that a discount you would sign up for?

The order I use now is deliberate: capability first, concurrency second, cost third. Capability decides whether the tier is in the race at all. Concurrency decides whether it can run beside your other work. Cost, once both of those pass, is a spreadsheet you finish in ten minutes and never argue about again.

Trial 1 — Build the task bar from your own accepted diffs

Stop inventing test prompts. Your repository already contains a labelled dataset: every pull request you merged last quarter, and every one you rejected. Pull twelve of them, three per difficulty tier, and turn each into a task that checks itself.

1. Pick the tiers. T1 is a single-function edit with tests already present; T2 is a single-file change with a hidden edge case; T3 is a cross-file change; T4 is "write the failing test for this bug". Keep three tasks per tier so one lucky pass cannot move the average.
2. Freeze the input. Copy the repository at the parent commit into repo/, and store the prompt you actually wrote as prompt.txt. Do not edit that prompt later to help the model along; that is how benchmarks become flattering.
3. Write the check, not the answer. Each task folder gets a check.sh that runs one narrow test and exits non-zero on failure. The model never sees it, and you never grade by eyeballing a diff.
4. Record the ceiling. The tier where the pass rate drops below half is your task bar, and it is the only number here you should trust — yours, not mine.

The harness (a template, not a benchmark result)

This is dependency-light Python against any OpenAI-compatible endpoint; endpoint, key, and model come from environment variables, so nothing assumes a particular provider. I am deliberately not publishing pass rates, because a number from someone else's repository tells you nothing about yours.

# taskbar.py — run: python taskbar.py tasks/ | tee taskbar.json
import json, os, pathlib, subprocess, sys, tempfile
from openai import OpenAI

REPO = pathlib.Path("./repo").resolve()
client = OpenAI(base_url=os.environ["AGENT_BASE_URL"], api_key=os.environ["AGENT_API_KEY"])
MODEL = os.environ["AGENT_MODEL"]

def ask(prompt):
    r = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2,
    )
    return r.choices[0].message.content, r.usage.total_tokens

def run_task(task_dir, attempts=3):
    target = (task_dir / "target.txt").read_text().strip()  # file the model may rewrite
    tier = (task_dir / "tier.txt").read_text().strip()
    base = (task_dir / "prompt.txt").read_text()
    prompt, spent = base, 0
    for n in range(1, attempts + 1):
        code, tokens = ask(prompt)
        spent += tokens
        with tempfile.TemporaryDirectory() as tmp:
            work = pathlib.Path(tmp) / "repo"
            subprocess.run(["cp", "-r", str(REPO), str(work)], check=True)
            (work / target).write_text(code)
            res = subprocess.run(["bash", str(task_dir / "check.sh")],
                                 cwd=work, capture_output=True, text=True)
        if res.returncode == 0:
            return {"task": task_dir.name, "tier": tier, "pass_at": n, "tokens": spent}
        prompt = f"{base}\n\nYour previous patch failed. Test output:\n{res.stdout[-1500:]}{res.stderr[-1500:]}"
    return {"task": task_dir.name, "tier": tier, "pass_at": None, "tokens": spent}

if __name__ == "__main__":
    tasks = [p for p in sorted(pathlib.Path(sys.argv[1]).iterdir()) if p.is_dir()]
    rows = [run_task(p) for p in tasks]
    for r in rows:
        print(json.dumps(r))
    for tier in sorted({r["tier"] for r in rows}):
        sel = [r for r in rows if r["tier"] == tier]
        wins = sum(1 for r in sel if r["pass_at"])
        print(f"{tier}: {wins}/{len(sel)} cleared, {sum(r['tokens'] for r in sel)} tokens", file=sys.stderr)

Reading the output honestly

Two columns matter and you should print both. pass_at tells you whether the tier is usable at all for that class of task. The token sum tells you what the retries actually cost, which is the only fair way to price a free allowance. If most tasks need three attempts, your effective budget is a third of the headline figure, and your review load is three patches instead of one.

Trial 2 — The queue trial

Capability testing happens with a single request in flight, which is not how you work. Send the same short prompt at concurrency 1, 4, and 8, then watch the 95th percentile instead of the mean — the mean hides exactly the stalls that break an agent loop.

# queue_probe.py — compare p95 at your real concurrency
import os, statistics, time
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

client = OpenAI(base_url=os.environ["AGENT_BASE_URL"], api_key=os.environ["AGENT_API_KEY"])
MODEL = os.environ["AGENT_MODEL"]
PROMPT = "Return a 30-line Python function that deduplicates a list while preserving order, plus one paragraph of explanation."

def one(_):
    t0 = time.perf_counter()
    client.chat.completions.create(model=MODEL, messages=[{"role": "user", "content": PROMPT}])
    return time.perf_counter() - t0

def probe(k):
    with ThreadPoolExecutor(max_workers=k) as pool:
        lat = sorted(pool.map(one, range(k)))
    p95 = lat[max(0, int(round(0.95 * (len(lat) - 1))))]
    return {"concurrency": k, "p50_s": round(statistics.median(lat), 2), "p95_s": round(p95, 2)}

for k in (1, 4, 8):
    print(probe(k))

Now convert latency into a budget. If your loop makes three calls per task and your deadline slack is 90 seconds, each call gets 30 seconds. A p95 above that number means the tier cannot carry parallel work, no matter how free the tokens are.

The decision table

Combine both trials and the choice usually falls out on its own.

Notice what the table does not contain: a recommendation to move everything. The most common honest answer is a split, because your task mix is already a split. Self-hosting earns its keep on the tier your free access cannot reach, not on the whole backlog.

Where MonkeyCode fits in this workflow

MonkeyCode is an open-source coding agent that offers free model access and a free server option, which makes it a low-cost place to run both trials before you spend anything on the alternatives. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option matters more than it first sounds, because a queue probe run from your laptop measures your home network rather than the endpoint. The operator quotes a 10-million-token free allowance at the time of writing; treat that as a moving number, verify the current terms yourself, and never plan a quarter around a figure you have not re-read this month. Free access changes what a failed trial costs you — nothing — but it does not change what a failed task bar means for your deadline.

Who should not use this approach

Be honest about the boundary before you invest an afternoon in it.

• Teams that cannot send source outside a controlled network. Both trials assume you may post code to an external endpoint; if that is not allowed, measure a local model instead.
• Anyone without a merged-PR history or a person to write the checks. The harness is only as sharp as its check.sh files, and vague checks produce flattering results.
• Sub-second interactive workflows. Any queue at all violates the requirement, so concurrency testing is beside the point.
• Tasks that exceed the tier's context window. The harness will fail them for the wrong reason, and you will misread the task bar.
• People who read free as a service-level agreement. These trials measure today; they do not promise next quarter.

What I would do differently next time

I would write the checks before I write the prompts, because the first version of every check I have drafted was too generous. I would also keep the trial folder in the repository, so the next person who asks "can we just use the free tier?" gets a table instead of an opinion. And I would re-run both trials after any tier change, since the number you measured in October is a historical fact by November.

If you run this week, run it against two tiers rather than one — the comparison is where the lesson lives. I would genuinely like to know what your task bar turned out to be, because that single number is more useful to the rest of us than any published price list.