遷移日誌：切換 AI 供應商當天，黃金測試樣例可能仍通過卻已失去意義

The fixtures in your test suite are a recording of the provider you are leaving, and they will keep passing long after cutover makes them meaningless. That is the failure mode that makes this class of migration dangerous: the suite goes green, the team relaxes, and nobody notices that every assertion still describes a response shape only the old endpoint produces. The fix is small and boring — one provenance field on each fixture, one split in your assertions, and one command you can run on cutover night.

Earlier entries in this diary covered the streaming parser, the cost ledger, and the rollback rehearsal. This entry covers the thing that tells you whether those three are still working after the switch.

1. Why fixtures rot silently instead of failing loudly

A recorded fixture is a snapshot of a request and a response captured from a specific provider, model, and tool schema at a specific moment. When you change any of those three, the fixture does not become wrong in a way your test runner can detect — it becomes a test of the past that still exits zero. Your CI keeps rewarding you for not touching code that no longer reflects production.

The damage compounds because fixtures are usually the cheapest thing to keep and the most annoying thing to regenerate. Someone records them once during the original integration, they encode a few quirks of that provider, and every later refactor is validated against those quirks. By the time you plan a cutover, the suite is measuring fidelity to a memory rather than fidelity to a contract.

Provenance is the cheapest countermeasure available. If every fixture file states which provider and model produced it, a cutover checklist can ask a mechanical question — how many fixtures are still stamped with the old provider — instead of relying on somebody remembering.

2. Step 1: stamp every fixture with where it came from

The example below is illustrative rather than tested, so adapt the field names to whatever your harness already writes. The point is that the metadata travels inside the fixture file, not in a spreadsheet that drifts away from it.

# tools/fixture_provenance.py — illustrative skeleton
from dataclasses import dataclass, asdict
from pathlib import Path
import hashlib, json

@dataclass
class Provenance:
    fixture: str
    provider: str
    model: str
    recorded_at: str
    prompt_hash: str
    tool_schema_hash: str

def _hash(obj) -> str:
    blob = json.dumps(obj, sort_keys=True, ensure_ascii=False).encode()
    return hashlib.sha256(blob).hexdigest()[:12]

def stamp(path: Path, provider: str, model: str, recorded_at: str) -> Provenance:
    case = json.loads(path.read_text())
    request = case["request"]
    return Provenance(
        fixture=path.name,
        provider=provider,
        model=model,
        recorded_at=recorded_at,
        prompt_hash=_hash(request.get("messages", [])),
        tool_schema_hash=_hash(request.get("tools", [])),
    )

def write_manifest(suite_dir: Path, entries: list[Provenance]) -> None:
    manifest = {e.fixture: asdict(e) for e in sorted(entries, key=lambda e: e.fixture)}
    (suite_dir / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")

Two hashes matter more than the timestamp. The prompt hash catches fixtures that were edited by hand after recording, and the tool schema hash catches the quiet breakage where you renamed a tool parameter and every recorded tool-call fixture silently stopped matching.

3. Step 2: split each assertion into contract and content

Re-recording everything on cutover night is too risky, so you need to know which assertions must survive a provider change and which are allowed to differ. Sort them into two buckets before you touch the endpoint configuration.

Contract assertions should live in a live test that hits the real endpoint, because a recorded contract test can only prove you recorded it correctly. Content assertions belong with the fixtures and should be regenerated rather than argued over.

4. Step 3: make re-recording one command with a drift report

Re-recording by script is not optional at this size, and the script must print what changed in structure before it overwrites anything. A shape diff that only tracks key paths, not values, is enough to spot a real adapter break while ignoring harmless wording changes.

# tools/rerecord.py — illustrative skeleton, not a finished tool
def shape(obj, prefix=""):
    if isinstance(obj, dict):
        out = set()
        for key, value in obj.items():
            out |= shape(value, f"{prefix}.{key}")
        return out
    if isinstance(obj, list):
        return shape(obj[0], f"{prefix}[]") if obj else {f"{prefix}[]"}
    return {prefix}

def rerecord(suite_dir, client, write=False):
    drift = []
    for path in sorted(suite_dir.glob("*.json")):
        case = json.loads(path.read_text())
        fresh = client.call(case["request"])
        added = shape(fresh) - shape(case["response"])
        removed = shape(case["response"]) - shape(fresh)
        if added or removed:
            drift.append({"fixture": path.name, "added": sorted(added), "removed": sorted(removed)})
        if write:
            case["response"] = fresh
            case["meta"]["provider"] = client.provider
            path.write_text(json.dumps(case, indent=2) + "\n")
    return drift

Run it twice. The first run, without --write, tells you where the new endpoint's shape diverges from the old recordings; the second run commits the fixtures and refreshes the manifest. If the drift list contains a contract key, stop and fix the adapter instead of re-recording around it.

5. Step 4: keep exactly one live contract lane running nightly

Recorded fixtures cannot tell you that a free endpoint changed something on its side, so you need a small live suite that runs on a schedule and skips gracefully when credentials are absent. Keep it deliberately tiny: a chat round-trip, a tool-call request, and a streaming chunk sequence.

# tests/live/test_contract.py — illustrative
import os
import pytest

pytestmark = pytest.mark.skipif(
    not os.environ.get("LIVE_CONTRACT"),
    reason="live contract lane runs nightly only",
)

def test_chat_returns_message(live_client, chat_request):
    resp = live_client.call(chat_request)
    message = resp["choices"][0]["message"]
    assert message.get("role") == "assistant"
    assert isinstance(message.get("content"), (str, list))


def test_tool_arguments_validate(live_client, tool_request, validate_tool_args):
    resp = live_client.call(tool_request)
    call = resp["choices"][0]["message"]["tool_calls"][0]
    validate_tool_args(call["function"]["name"], call["function"]["arguments"])

Three tests, one nightly job, one alert. Anything more and the lane becomes flaky enough that people mute it, which is the same as not having it.

6. Step 5: sequence the cutover and write down the leftovers

Cutover order matters more than cutover speed, so run it in five moves. Freeze fixture writes, re-record against the new endpoint, review the drift report, flip the default provider behind a flag, then keep the old path warm for one rollback window.

The leftovers from this step are specific and easy to forget: scheduled jobs still pinned to the old model identifier, cached responses keyed by the old provider name, dashboards grouped by a field the new endpoint omits, retry budgets tuned for the previous latency profile, and a manifest full of stale provenance entries you never regenerated. Write each of these into the cutover ticket as a named line item with an owner, because "check the cron jobs" is not a task anybody completes.

7. Where a free endpoint and a free server actually earn their place

This is the part of the workflow where free model access and a free server option are genuinely useful rather than decorative. A second endpoint is what makes the nightly contract lane affordable, and a small host is enough to run the fixture re-record job without borrowing production capacity.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option, per the project's own materials — verify the current limits, regions, and terms on the project page before you design around them, because quotas and hardware change without notice. What I will not claim is anything about throughput, uptime, or model breadth, because none of that is verifiable from here and your own contract lane is the only measurement that matters.

If you want a cheap second target for the drift and contract steps above, MonkeyCode's free model access and free server are a reasonable place to start, provided you read the limits first.

8. Who should not take this route

Skip this approach if you need contractual capacity guarantees, if your data cannot leave your current jurisdiction or processor list, or if you have no way to run a scheduled live test — in that last case a free endpoint becomes an unmonitored dependency very quickly. Teams with hard latency SLOs should also keep production on a paid tier and use the free endpoint only for the fixture and contract lanes described here.

9. Pre-cutover checklist

1. Every fixture file carries provider, model, prompt hash, and tool schema hash.
2. Contract assertions are separated from content assertions in the test tree.
3. rerecord runs without --write and you have read the drift report line by line.
4. The nightly live lane exists, is small, and has a named owner for its alerts.
5. Leftover items are written into the ticket with owners, not left as verbal reminders.
6. The manifest is regenerated after the final re-record, not before it.

None of these steps are clever, and that is the point — the fixtures were never the interesting part of the migration, but they are the part that decides whether you find out about a break at 2am or at 2pm.