模型切換應視為資料遷移，而非簡單改動設定

A Model Swap Is a Migration, Not a Toggle

Last Friday I changed a single environment variable, pointed the support-reply feature at a different provider, and went home feeling clever. On Monday the export job had three days of rows where refund_days was null, and nobody flagged it because the UI rendered the reply text and quietly ignored the missing field. My first instinct was to blame the new model. My second, more honest instinct was to ask why I had shipped a schema-affecting change with no fixtures, no diff, and no gate at all. We stopped migrating databases by hand-editing SQL in production years ago, so why do we still swap the thing that generates our persisted data by editing a config line?

I will state the position plainly, because hedging here helps nobody: a model swap is a migration, not a toggle. The output lands in your database columns, your retry classifier, your export files, and your customer's inbox, which means you carry schema obligations the moment you persist it. A toggle implies clean reversibility with no residue, and anyone who has rolled back a half-applied migration knows how fictional that is. So the useful argument is not which model wins this month. It is what evidence you demand before you change the one you already run.

Why your test suite will not save you here

Integration tests assert expectations you wrote by hand, which makes them excellent at catching the bugs you already imagined. They say almost nothing about how a new provider behaves on the four hundred real tickets your team handled last month, including the ugly ones with partial refunds and mixed locales. I have watched a green suite ship a provider change that broke three percent of exports, and I have also watched a red suite block a change that was strictly better. Neither outcome told me anything about traffic. Isn't that the gap worth closing first?

The cheapest honest evidence is a frozen fixture corpus captured from production before you touch the provider. Not synthetic prompts, not a curated golden set copied from documentation, but the actual inputs you already paid to process, with personal data stripped out. Once those fixtures exist, the swap becomes a comparison you can rerun on demand instead of an opinion you defend in a code review.

The harness: capture, shadow, diff, gate

Here is the version I use now. Capture writes one JSON file per real request, keeping the input and the baseline output so later runs have something to compare against.

# shadow/capture.py (sketch)
import json, pathlib, datetime

def write_fixture(root, request_id, payload, baseline_raw, provider):
    path = pathlib.Path(root) / f"{request_id}.json"
    path.write_text(json.dumps({
        "id": request_id,
        "captured_at": datetime.datetime.utcnow().isoformat() + "Z",
        "input": payload,
        "baseline": {"provider": provider, "raw": baseline_raw},
    }, indent=2))

Redaction happens before the write, never after, because a fixture folder that leaks ticket text is a second incident stacked on the first. I keep the redactor in the same module and drop anything matching an email, phone, or order-number pattern. The fixture then becomes a reviewable artifact inside the pull request, which is exactly what a migration needs.

The shadow run replays every fixture through the candidate and scores two things independently: whether the payload still parses into the shape the database expects, and how far the text drifted from the baseline.

# shadow/run.py
import json, time

REQUIRED = {"reply": str, "refund_days": int}

def shape_errors(raw):
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        return [f"unparseable: {exc.msg}"]
    return [f"{k} missing or wrong type" for k, t in REQUIRED.items()
            if not isinstance(data.get(k), t)]

def token_overlap(a, b):
    ta, tb = set(a.lower().split()), set(b.lower().split())
    return 0.0 if not ta or not tb else len(ta & tb) / len(ta | tb)

def shadow(adapter, fixtures):
    rows = []
    for fx in fixtures:
        started = time.monotonic()
        raw = adapter.complete(fx["input"])
        rows.append({
            "id": fx["id"],
            "shape_errors": shape_errors(raw),
            "overlap": round(token_overlap(fx["baseline"]["raw"], raw), 3),
            "latency_ms": int((time.monotonic() - started) * 1000),
        })
    return rows

Token overlap is a tripwire and not a judge, and I want to be explicit about that limitation. It punishes valid rewording and rewards lazy copying, so all it can do is tell you where to look next. I pair the score with a manual read of the ten lowest-overlap fixtures, which costs twenty minutes and has caught more real problems than any automatic metric I have tried.

The gate is where the opinion becomes enforceable. Shape failures are absolute, drift has a small budget, and the exit code is what CI actually reads.

# shadow/gate.py
import sys

def evaluate(rows, overlap_floor=0.90, max_drift=2):
    shape_failures = [r["id"] for r in rows if r["shape_errors"]]
    drifted = [r["id"] for r in rows if r["overlap"] < overlap_floor]
    if shape_failures:
        print("shape failures:", shape_failures[:10])
        return 2
    if len(drifted) > max_drift:
        print(f"drift {len(drifted)} > {max_drift}:", drifted[:10])
        return 3
    print(f"pass: {len(rows)} fixtures, {len(drifted)} drifted")
    return 0

sys.exit(evaluate(rows))

Notice what the table refuses to do: it never pretends to measure quality. Quality needs a human or a labelled set, and pretending otherwise is how teams end up trusting a number that only measured formatting.

Where the free tier fits, honestly

Here is the practical part that made me try this at all. MonkeyCode is an open-source project that offers free model access and a free server option, and that pairing maps onto this harness suspiciously well: the free model access gives you a candidate slot to diff against without a procurement conversation, and the free server option gives you somewhere to run the shadow job without borrowing production capacity. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That said, I treat both as convenience, never as architecture. I have not run long-term reliability or quota measurements, so I keep no state on the shadow box and I assume every job can be re-run elsewhere tomorrow. The harness is provider-agnostic by design, which is the entire point of the adapter interface: if the candidate slot disappears, I repoint one class and replay the same fixtures. Ask yourself whether your swap plan survives the candidate provider vanishing without notice, because that question is far cheaper now than during an incident.

One more boundary deserves stating out loud. The shadow worker gets read-only fixtures and no database credentials at all. If it can write to the table it is evaluating, it is not a shadow anymore, it is a second production writer with a different prompt.

When this is the wrong approach

Do not build this if your outputs are genuinely creative and structural comparison is meaningless, because you will spend a week building a metric that cannot see what your users care about. Do not build it if you have no real traffic yet, since twenty hand-written prompts teach you nothing and hand you false confidence in exchange. And do not build it if a human reviews every output before it reaches a customer, because in that world the reviewer is already the gate.

The approach also fails quietly for teams that capture fixtures once and never refresh them, which is just a migration test written against last year's schema. Refresh the corpus on a schedule, retire fixtures older than a quarter, and treat the fixture count as a health signal rather than a trophy.

A checklist you can reuse

1. Capture a fixture corpus from real traffic, applying redaction before the write.
2. Encode the required output shape next to the parser that depends on it.
3. Run the candidate in shadow with no write credentials and no shared state.
4. Gate on shape failures first, drift second, and latency plus cost as reviewer notes.
5. Read the worst ten overlaps by hand before trusting any aggregate number.

None of that is exotic, and the whole harness fits in a few hundred lines. That is precisely why I find "we will just try the new model and watch the dashboards" so hard to defend in 2026. Watching dashboards is how I lost three days of refund_days, and the dashboard stayed green the entire time.

So the argument I would put to you is simple. The next time someone proposes a provider swap, ask for the fixtures, the diff budget, and the exit code, in that order. If the answer is a spreadsheet of vibes, then the swap is a migration you are running blind. And if you do stand the harness up on the free server option, tell me which layer broke first: the capture step, the shadow worker, or the threshold you picked.