如何在 CLI 使用託管模型前建立免費方案預檢閘門

It was 23:40 on a Tuesday. My CLI had been talking to a free model endpoint for two weeks.

It returned HTTP 200 all night. Then it handed me half a JSON object. My build broke at 07:00 and nobody had measured anything.

That is the real problem with free tiers. They rarely fail loudly. They fail politely.

So I wrote a preflight script. It runs before my CLI is allowed to touch a hosted model. Every gate writes an evidence file. Missing evidence counts as failure, not as a pass.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project whose free tier lists 10,000,000 tokens and a free server option, as described by the team. I treat those two claims as operator-supplied, not as numbers I measured. The harness below works against any OpenAI-style chat endpoint.

Why a checklist beats a vibe check

A vibe check answers one question: did it work just now? A preflight gate answers a better one: what evidence proves it still works?

Free lineups rotate, rate limits shift, and servers get cold. So pin a model id in your environment and re-run the gates whenever that pin changes. Six minutes of checks beats a broken morning.

What usually goes unmeasured? Latency, truncation, and token accounting. Those three cause most of my silent failures.

The seven gates

Each gate must produce a file. If you cannot point at the file, the gate did not run.

Gate 5 is the one people skip. If the endpoint never tells you it truncated, you cannot detect truncation downstream. Ask for a deliberately tiny answer and check the reason field.

Gate 6 matters for budgets. You cannot audit a free allowance without per-call usage numbers.

The harness

Here is the whole thing. Standard library only, about 90 lines.

#!/usr/bin/env python3
"""preflight.py - fail-closed gates before a CLI trusts a hosted model."""
import json, os, sys, time, hashlib, urllib.request, urllib.error
from pathlib import Path

STAMP = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime())
EVID = Path("preflight") / STAMP
EVID.mkdir(parents=True, exist_ok=True)

BASE = os.environ["MC_BASE_URL"].rstrip("/")
PATH = os.environ.get("MC_CHAT_PATH", "/v1/chat/completions")
KEY = os.environ["MC_API_KEY"]
MODEL = os.environ["MC_MODEL"]
BUDGET_MS = int(os.environ.get("MC_LATENCY_BUDGET_MS", "4000"))

results = []

def gate(name, ok, detail):
    results.append({"gate": name, "ok": bool(ok), "detail": detail})
    print(f"[{'PASS' if ok else 'FAIL'}] {name}: {detail}")

def call(messages, max_tokens=64, timeout=25):
    body = json.dumps({"model": MODEL, "messages": messages,
                       "max_tokens": max_tokens}).encode()
    req = urllib.request.Request(
        BASE + PATH, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        raw = resp.read()
        status = resp.status
    return status, raw, (time.time() - t0) * 1000

# Gate 2 + 3: reachability and latency samples
samples = []
try:
    for _ in range(3):
        status, raw, ms = call([{"role": "user", "content": "ping"}], max_tokens=8)
        samples.append(round(ms, 1))
    (EVID / "latency.json").write_text(json.dumps({"model": MODEL, "samples_ms": samples}))
    gate("reachability", status == 200, f"status={status} model={MODEL}")
    gate("latency_budget", max(samples) <= BUDGET_MS, f"max={max(samples)}ms budget={BUDGET_MS}ms")
except Exception as exc:
    gate("reachability", False, repr(exc))
    gate("latency_budget", False, "not measured")

# Gate 4: schema conformance
SCHEMA_PROMPT = ('Reply with JSON only: {"ok": true, "n": 3}.')
try:
    _, raw, _ = call([{"role": "user", "content": SCHEMA_PROMPT}], max_tokens=48)
    (EVID / "schema-raw.txt").write_bytes(raw)
    parsed = json.loads(raw.decode())
    content = parsed["choices"][0]["message"]["content"]
    payload = json.loads(content)
    ok = set(payload) == {"ok", "n"}
    gate("schema_conformance", ok, f"keys={sorted(payload)}")
except Exception as exc:
    gate("schema_conformance", False, repr(exc))

# Gate 5: truncation visibility, gate 6: usage accounting
try:
    _, raw, _ = call([{"role": "user", "content": "Count from 1 to 200."}], max_tokens=5)
    parsed = json.loads(raw.decode())
    choice = parsed["choices"][0]
    (EVID / "truncation.json").write_text(json.dumps(choice))
    gate("truncation_visible", "finish_reason" in choice, f"finish_reason={choice.get('finish_reason')}")
    usage = parsed.get("usage") or {}
    gate("usage_accounting", "total_tokens" in usage, f"usage_keys={sorted(usage)}")
except Exception as exc:
    gate("truncation_visible", False, repr(exc))
    gate("usage_accounting", False, repr(exc))

# Gate 1: key presence, recorded last so it can never mask network results
gate("key_present", bool(KEY), "sha256 prefix=" + hashlib.sha256(KEY.encode()).hexdigest()[:8])

(EVID / "summary.json").write_text(json.dumps(results, indent=2))
failed = [r["gate"] for r in results if not r["ok"]]
print(f"evidence: {EVID}  failed: {failed or 'none'}")
sys.exit(1 if failed else 0)

The example output you would see is illustrative, not a measured run. Replace the endpoint path with whatever your provider documents.

Running it

export MC_BASE_URL="https://your-endpoint.example"
export MC_CHAT_PATH="/v1/chat/completions"
export MC_API_KEY="..."
export MC_MODEL="your-pinned-model-id"
export MC_LATENCY_BUDGET_MS="4000"
python3 preflight.py; echo "exit=$?"

Then wire it into the CLI you already have. One line at the top of your wrapper:

python3 preflight.py || { echo "model gate failed; using local fixtures"; exec ./fallback.sh; }

Now the CLI cannot run on an unverified endpoint. That is the whole point of fail-closed.

One failure fixture: the truncated object

Here is the shape that broke my build. The endpoint returned valid JSON with a cut-off inner string:

{"choices":[{"message":{"content":"{\"files\":[\"src/a.ts\",\"src/b"},
 "finish_reason":"length"}]}

Gate 4 fails this, because the inner content will not parse. Gate 5 passes, because finish_reason is present. That split matters. Truncation is visible; schema conformance is not. Keep them as separate gates so you know which one to fix.

If you strip finish_reason before parsing, gate 5 catches the real risk and tells you why.

A token ledger you can actually audit

Gate 6 gives you per-call usage. Multiply by call volume to see how long a free allowance lasts. The numbers below are my assumed workloads, not measured quotas.

Against a supplied 10,000,000-token free allowance, 39,500 tokens a day is roughly 250 days of that workload. That arithmetic is mine. The allowance figure is the project's claim, not my benchmark.

Log usage per call and sum it weekly. A budget you never count is a story, not a budget.

Rollback and abandonment criteria

Decide the exit before you need it.

1. If gate 4 fails twice on a pinned model in one day, switch that task to local fixtures.
2. If the latency budget fails for a full working day, move the task back to a local model.
3. If usage fields disappear, stop counting and stop trusting the free tier for anything billable-adjacent.
4. If the same prompt fails identically twice, do not retry it a third time. Store it as a regression fixture.

Write these down, in the repo, next to the script. Future you will not remember the thresholds.

Who should not use this

Skip this approach if you need documented latency SLAs, contractual data handling, or reproducible outputs across days. A hosted free tier cannot promise those, and gates only detect the gap, they do not close it.

Also skip it if your payloads contain customer data. Preflight gates check shape and timing, not data policy.

If you want a free endpoint to test the harness against, MonkeyCode publishes its repo, a free server option, and a free tier that the team lists at 10,000,000 tokens. Read the current docs before you pin a model id, because the lineup moves.

What is the one gate you would add as number eight, and what evidence file would it write? That answer decides whether my checklist is finished or just long.