信任免費模型前，先對同一提示詞測試兩次

The first number your eval should report is not accuracy. It's whether your eval says the same thing twice.

I keep seeing the same review pattern in October: someone swaps a paid endpoint for a free one, runs their test suite once, and posts a verdict. Green, or red, and then a blog post. But a single run of a nondeterministic system is not a measurement. It's a sample. And if your free tier samples differently than your paid tier, your comparison is two coin flips wearing lab coats.

So before you benchmark anything, run an A/A test. Same prompt. Same parameters. Nothing changes. Then ask how many times you got the same answer back.

What an A/A test actually buys you

An A/B test compares two things, and only means something if the harness is quiet. An A/A test is the control: it tells you how loud the harness is on its own. If five identical calls return five different texts, your diff-based regression suite is measuring sampling noise and calling it a regression.

Temperature 0 does not save you here. Shared serving stacks batch and reorder work, kernels reduce in different orders, and plenty of providers rotate the model behind a stable alias without telling you. Determinism at temperature=0 is a common assumption, not a guarantee.

That last point is a gift, by the way. A fingerprint probe also works as a tripwire for silent model swaps — if your agreement suddenly collapses on unchanged inputs, something upstream changed, and it wasn't your prompt.

The probe

Here's the whole experiment. One small script, one prompt, k repeats, one JSON line out. It assumes an OpenAI-compatible /chat/completions shape; adjust the request body if your endpoint differs.

#!/usr/bin/env python3
"""aa_probe.py - A/A test: one prompt, k samples, measure self-agreement."""
import argparse, collections, hashlib, json, os, re, urllib.request


def post(base, key, payload):
    req = urllib.request.Request(
        f"{base.rstrip('/')}/chat/completions",
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {key}"},
    )
    with urllib.request.urlopen(req, timeout=180) as r:
        return json.load(r)


def normalize(text):
    t = text.strip().lower()
    t = re.sub(r"```

.*?

```", " ", t, flags=re.S)  # drop code fences first
    t = re.sub(r"[^a-z0-9 ]+", " ", t)
    return re.sub(r"\s+", " ", t).strip()


def fingerprint(text):
    return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()[:16]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--k", type=int, default=5)
    ap.add_argument("--temp", type=float, default=0.0)
    ap.add_argument("--prompt", default=os.environ.get("AA_PROMPT", ""))
    a = ap.parse_args()
    if not a.prompt:
        raise SystemExit("pass --prompt or set AA_PROMPT")

    base, key = os.environ["LLM_BASE_URL"], os.environ["LLM_API_KEY"]
    model = os.environ["LLM_MODEL"]
    fps, totals = [], []

    for _ in range(a.k):
        data = post(base, key, {
            "model": model,
            "temperature": a.temp,
            "messages": [{"role": "user", "content": a.prompt}],
        })
        fps.append(fingerprint(data["choices"][0]["message"]["content"]))
        usage = data.get("usage") or {}
        if "total_tokens" in usage:
            totals.append(usage["total_tokens"])

    counts = collections.Counter(fps)
    print(json.dumps({
        "model": model, "temp": a.temp, "k": a.k,
        "modal_share": round(counts.most_common(1)[0][1] / a.k, 3),
        "distinct": len(counts),
        "usage_total_tokens": sum(totals) if len(totals) == a.k else None,
    }))


if __name__ == "__main__":
    main()

modal_share is the number that matters: the fraction of runs that landed on the single most common answer. Normalization is deliberately brutal — lowercase, punctuation stripped, whitespace collapsed — because you're not testing wording, you're testing behavior.

Make it a week-long record, not a one-off

One probe run tells you about one moment on shared hardware. Run it on a schedule and you get something better: a time series of how honest your harness is.

#!/usr/bin/env bash
set -euo pipefail
PROMPT_FILE="prompts/cache-key.txt"
export AA_PROMPT="$(cat "$PROMPT_FILE")"
PROMPT_SHA="$(sha256sum "$PROMPT_FILE" | cut -c1-12)"
mkdir -p runs

for run in 1 2 3; do
  python3 aa_probe.py --k 5 --temp 0 \
  | jq -c --arg sha "$PROMPT_SHA" --arg ts "$(date -u +%FT%TZ)" \
      '. + {ts: $ts, prompt_sha: $sha}' >> runs/aa.jsonl
done

# gate: how many samples fell below 0.9 agreement?
jq -s 'map(select(.modal_share < 0.9)) | length' runs/aa.jsonl

Hashing the prompt file matters more than it looks. When you edit a prompt, you want the fingerprint history to break cleanly at that exact revision, not silently drift underneath it. That's the same discipline as freezing a config hash — the run has to earn its comparison.

This is the kind of job that wants to live somewhere other than your laptop, and it's where a free tier actually fits: MonkeyCode advertises free model access plus a free server option, which is enough to host a probe loop and collect samples while you do other work. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caution about the server, though: it's shared, so do not point a latency benchmark at it and report the numbers as a property of the model. Measure latency there and you're measuring the neighborhood.

Reading your own numbers

Here's the triage table I use. The cutoffs are proposed heuristics, not measured constants — calibrate them against your own workload before you quote them at anyone.

A high share is not a quality signal. A model can be perfectly consistent and consistently wrong — that's what your B/B test is for, and you can't run a B/B until the A/A is quiet. Consistency is the floor you build on, not the score you report.

Where this breaks, and who should walk away

Token accounting is the weak seam. If your endpoint returns no usage block, you'll be measuring tokens with a local tokenizer that may not match the server's — and a budget comparison across two mismatched tokenizers is worse than no comparison. Either measure usage on both sides or drop the token column entirely and say so.

The approach also doesn't fit everyone. If you need pinned model versions, uptime commitments, or sub-second p99 latency, a shared free tier is the wrong instrument — not because it's bad, but because those are contract questions, not experiment questions. And if your product is regulated, an eval whose variance you haven't characterized is a liability with a dashboard.

What I'd actually ask you to take away is narrower than "try the free tier." It's that a free model with a known variance profile is more useful than a paid model you never probed, because only one of them lets you tell a regression from a reroll. Run the probe for a week before you commit either way — and if you do, I want to know what your agreement rate was, because that number is the real starting line.