模型遷移手記：免費端點接管前如何重播提示詞

The cheapest cutover test is to replay your own recorded prompts through both endpoints and read the diff yourself. You can plan retry classes and response contracts for weeks and still lose on the long tail of inputs nobody remembers writing. Shadow replay converts "it looked fine in the demo" into a divergence count you can defend in a review. This entry covers that step end to end: freeze a corpus, normalize outputs, classify divergence, then decide what you still own after the flip.

There is a familiar shape to the current wave of posts about cheaper or free model access. One happy transcript, one benchmark number, and an implied promise that the swap is boring. That framing hides the expensive part, because the first twenty requests are easy and requests two thousand through twenty thousand are not. Your real job during cutover is finding quiet failures before users do.

Why a model swap fails quietly

Most substitutions do not crash; they degrade in ways that pass a smoke test and fail a review. The candidate model returns the same answer in a slightly different shape, and your downstream parser reads a string where it expected an object. Or it produces a shorter answer, which your UI happily renders as a blank-looking card. These failures are invisible until someone reads the output, which is exactly what replay is for.

The five drift classes worth naming are schema drift, refusal drift, truncation drift, formatting drift, and semantic drift. Schema and truncation are mechanical, so you can catch them with code. Refusal drift usually appears only on your ugliest inputs, such as billing disputes or medical wording. Semantic drift is the residue that needs a human, and you should treat it as a queue rather than an alarm.

Step 1 — Freeze a replay corpus before you touch anything

Record a sample of real production traffic and store it as JSONL, one request per line. Include the parameters you actually sent, because temperature and token caps change the comparison more than the model does. Redact identifiers at capture time rather than at review time, so the corpus stays usable for future migrations.

{"id": "sum-001", "source": "support-ticket", "messages": [{"role": "system", "content": "Summarize the ticket in two bullets."}, {"role": "user", "content": "..."}], "params": {"temperature": 0, "max_tokens": 300}, "baseline": {"model": "paid-flagship", "text": "..."}}

Store the baseline output alongside the request, not only in your logs. If you only keep a pointer, the first thing you will do during a review is re-run the old endpoint, and that reintroduces the cost you are trying to remove. A few hundred well-chosen fixtures beat ten thousand random ones, provided you sample the ugly cases deliberately.

Step 2 — Normalize before you diff

The naive comparison is string equality, and it will drown you in noise from bullets, whitespace, and trailing punctuation. Normalize first, then fingerprint, then compare fingerprints to separate cosmetic churn from real change. The helper below is a skeleton that was not executed against any live provider; treat it as a template you adapt.

# shadow_replay.py — skeleton, not executed against any live provider.
import hashlib, json, pathlib, re

BULLET = re.compile(r"^\s*[-*•]\s+", re.M)
WS = re.compile(r"[ \t]+")


def normalize(text: str) -> str:
    text = BULLET.sub("- ", text)
    text = WS.sub(" ", text)
    return "\n".join(line.strip() for line in text.strip().splitlines())


def fingerprint(text: str) -> str:
    return hashlib.sha256(normalize(text).encode()).hexdigest()[:12]


def json_object(text: str):
    try:
        value = json.loads(text)
    except json.JSONDecodeError:
        return None
    return value if isinstance(value, dict) else None

Normalization rules are an editorial decision, so write them down. If your product sells summaries, bullet markers probably do not matter, but numbers and negation absolutely do. A normalizer that lowercases everything is convenient and dangerous, because it erases "not approved" into "approved" for the diff.

Step 3 — Classify divergence with a decision table

Categories turn a pile of diffs into a work queue with owners and severities. Keep the taxonomy small enough to memorize, and give every class a routing rule rather than a vibe. The table below is the one worth copying into your migration doc.

REFUSALS = ("i can't help", "i cannot assist", "as an ai")


def classify(baseline: str, candidate: str) -> tuple[str, int]:
    if baseline == candidate:
        return "same", 0
    if fingerprint(baseline) == fingerprint(candidate):
        return "cosmetic", 0
    if any(p in candidate.lower() for p in REFUSALS):
        return "refusal", 3
    if len(candidate) < 0.6 * len(baseline):
        return "truncation", 2
    a, b = json_object(baseline), json_object(candidate)
    if a and b and set(a) != set(b):
        return "schema", 3
    return "semantic", 1

Severity is about blast radius, not about how strange the output looks. A renamed JSON key breaks your parser for every user, while one odd paraphrase breaks nothing. Route high-severity classes to code fixes and medium-severity classes to a review queue you actually staff.

Step 4 — Run the shadow window on a bounded budget

The replay run should be boring: fixed concurrency, a hard cap on requests, and a checkpoint per fixture. Checkpointing matters more than speed, because a half-finished run that you cannot resume tempts you into skipping fixtures. The parallel version below keeps the failure surface small and writes each verdict immediately.

import httpx
from concurrent.futures import ThreadPoolExecutor

OUT = pathlib.Path("replay-out")
OUT.mkdir(exist_ok=True)


def judge(base_url, api_key, model, item, timeout=60):
    payload = {"model": model, "messages": item["messages"], **item.get("params", {})}
    response = httpx.post(
        f"{base_url}/chat/completions",
        headers={"Authorization": f"Bearer {api_key}"},
        json=payload,
        timeout=timeout,
    )
    response.raise_for_status()
    text = response.json()["choices"][0]["message"]["content"]
    verdict, severity = classify(item["baseline"]["text"], text)
    return {"id": item["id"], "verdict": verdict, "severity": severity, "text": text}


def run(items, workers=4):
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = [pool.submit(judge, BASE_URL, API_KEY, MODEL, item) for item in items]
        for future in futures:
            row = future.result()
            with (OUT / "verdicts.jsonl").open("a") as fh:
                fh.write(json.dumps(row) + "\n")

Keep concurrency low, because free capacity is shared and a burst looks like an attack rather than a benchmark. Cap the run at a number you can afford to repeat, and record the endpoint and model identifier in every verdict row. When the results look good, re-run a small slice after a day to check whether the endpoint is behaving consistently.

Step 5 — Where MonkeyCode fits in this plan

Shadow replay needs a second endpoint, and that is the practical place a free option helps. According to its operator, MonkeyCode is an open-source project that provides free model access and a free server option, which lets you run the candidate side of the comparison without opening a second paid account.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it where the replay harness needs a target: point the candidate client at the free endpoint, keep your paid baseline frozen in the fixtures, and let the harness grade both. I have not independently verified the license terms, quota, or server specifications, so confirm the current numbers and limits in the project's own documentation before you quote them in a plan. Availability claims in this space change quickly, and a figure copied from an older post is worse than no figure at all. If you do try it, run the harness on your own fixtures first and let the divergence report decide whether the swap is safe.

The leftovers you still own after cutover

Endpoints change; the surrounding scaffolding does not disappear with them. Expect to keep vendor-specific parameters, prompt caching prefixes, streaming behavior differences, tool schemas, log fields, and the alert thresholds you tuned on the old latency profile. Each one is a small piece of the paid workflow that quietly outlives the account you closed.

The corpus is the most valuable leftover, and it is also the one teams forget to assign an owner to. Name a maintainer, version the fixtures, and add a case every time production surprises you. A replay corpus that nobody updates decays into a museum of prompts from the week of the migration.

Limitations and who should not use this

Replay measures output quality on recorded inputs, and it says nothing about latency under real concurrency. It also cannot grade inputs you never captured, which is usually the tail that hurts. If your prompts contain regulated data you cannot retain, this workflow is not for you without a redaction pipeline that you trust.

Skip shadow replay if a wrong answer is legally or financially catastrophic and every response needs a human sign-off anyway. Skip it too if you cannot freeze a baseline, because a comparison without a fixed reference is just two opinions. For everyone else mid-migration, it is the cheapest evidence you will get before the flip.