影子模型發出退款：.NET 大語言模型升級中容易忽略的 7 個錯誤

"Let's shadow the new model first, then canary it." Everyone agrees with that sentence in planning. It sounds like the careful option, and it is, as long as the shadow can't do anything and the canary is the same set of users on every server.

I built a model-upgrade pipeline for a .NET 10 support assistant on Microsoft.Extensions.AI 10.11.0. The assistant has three tools: get_order, issue_refund and escalate_to_human. Two of them move money or create work for people. That's on purpose, because a shadow setup that's harmless for a chatbot can be dangerous for an agent.

Here are seven mistakes I made, or nearly made, on the way. The code is trimmed to the shape, and the trimmed parts are marked.

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. The complete, working, tested version of this shadow and canary project is inside, with full source and an in-depth PDF guide. Join Tech Skill Builder

Mistake 1: Thinking you're upgrading a model id

The pull request says "switch to the new model". It usually also changes the system prompt "while we're at it", and maybe the temperature. Any one of those can change behavior. If your metrics only know the model id, you can't tell which change caused what, and two different prompts on the same model look like one release.

Fix: release the three together and give the combination a fingerprint:

public sealed record ModelRelease(string Name, string ModelId, PromptVersion Prompt,
    float? Temperature = null, int? MaxOutputTokens = null)
{
    public string Fingerprint => Hashing.Short(/* model id + prompt hash + options, culture-invariant (full version in the complete project) */);
}

In the project, a ReleaseChatClient (a plain DelegatingChatClient) pins a provider client to one release, so nothing in the app can quietly swap the prompt. The prompt registry is append-only: change a released prompt and you have a new release with a new fingerprint.

Mistake 2: Mirroring the whole request

This is the one in the title. The obvious shadow design sends each request through a second pipeline and compares the final answers. With tools, that second pipeline runs its own tool calls. In my demo, the "shadow" candidate rounded a refund up, and the order book ended with two refunds: 20.50 EUR from the live model and 21.00 EUR from the shadow. The shadow wasn't watching. It acted.

Fix: put the shadow under FunctionInvokingChatClient, so it compares one model turn at a time, and give the candidate the tools as declarations only:

public static ChatOptions DeclarationsOnly(ChatOptions? options)
{
    var o = options?.Clone() ?? new ChatOptions();
    // replace every AIFunction in o.Tools with f.AsDeclarationOnly(): same name and schema, not invocable
    // (full version in the complete project)
    return o;
}

The candidate decides with the same information, and its tool calls come back as FunctionCallContent that nothing executes. As a belt-and-braces check, the shadow runner asks the pipeline for GetService<FunctionInvokingChatClient>() and refuses to start if it finds one.

A nice side effect: comparing per turn gives you a precise, cheap signal. issue_refund {"amount":20.5} against issue_refund {"amount":21} doesn't need a judge to be flagged.

Mistake 3: Letting the shadow job hold references

The shadow runs later, on a background worker. If the job keeps references to the request's objects, it compares against something that has changed in the meantime.

Three ways this bit me:

• Middleware that isn't under function invocation receives the caller's own message list. The app appends to it, so by the time the worker ran, the "input" had 3 messages instead of 1.
• ChatMessage.Clone() is shallow. The clone and the original share one Contents list.
• The request's cancellation token is cancelled when the request ends, which is exactly when shadow work starts.

Fix: snapshot on the request thread, with new messages and copied content lists, and give the runner its own timeout (through TimeProvider, so tests run on fake time) that's linked only to host shutdown.

Mistake 4: Trusting the channel to count what it dropped

The shadow must never slow down a real request, so jobs go into a bounded channel and are dropped when it's full. Fine. Then I counted the drops with the return value of TryWrite.

With BoundedChannelFullMode.DropWrite, TryWrite returns true for items it drops. In the demo, five writes into a channel of capacity two all returned true, and three of the jobs were gone.

Fix: count in the itemDropped callback you pass to Channel.CreateBounded, which is the only honest signal:

_channel = Channel.CreateBounded<ShadowJob>(
    new BoundedChannelOptions(capacity) { FullMode = BoundedChannelFullMode.DropWrite },
    dropped => { /* count it, and remember which job it was (omitted) */ });

Why it matters beyond ops: a queue that drops under load drops at peak hours, so your shadow sample quietly turns into a sample of quiet hours. The project's shadow gate holds when drops exceed a tenth of the enqueued jobs.

Mistake 5: Gating on "agreement"

The first gate I wrote said: promote if the candidate agrees with the current model often enough. On one simulated day, the shadow compared 898 turns and agreement was 58.4%. Was that bad?

Mostly it was style. The new prompt asked for a warmer tone, and "Good news! Your order is shipped..." scores low against "Order shipped." A terse model that gets dates wrong could score higher. Agreement measures voice as much as quality.

Fix: gate on mechanical red flags, each with its denominator, instead of on overall similarity:

• the candidate refused where the current model answered (2.6% that day; my limit is 1%)
• a different tool or different arguments on tool turns (13.0%; limit 2%)
• text-versus-tool decision flips, candidate failures and timeouts, latency, and the drop rate

That first candidate held on refusals and tool divergence. And a shadow gate never rolls back on its own, because no user is exposed yet. It holds and waits for a person.

Mistake 6: Bucketing users with GetHashCode()

Once the shadow is clean, real users get the candidate. Each user must land on the same side on every request, every pod and every restart. Math.Abs(userId.GetHashCode()) % 100 looks fine and isn't: string hashing in .NET is randomized per process, so your 1% is a different 1% on every instance.

Fix: a stable hash with a per-rollout salt:

public static int Of(string salt, string subject)
{
    // SHA-256 over salt + separator + subject, first 8 bytes as an unsigned number,
    // modulo 10,000 buckets (full version in the complete project)
}

public static bool InCanary(string salt, string subject, int canaryBasisPoints) =>
    canaryBasisPoints > 0 && Of(salt, subject) < Math.Min(canaryBasisPoints, Buckets);

The salt keeps the same unlucky users from testing every model you ever ship. The "less than" makes the ramp monotonic, so going from 5% to 25% only adds users. Requests without a user stay on the current model during the canary.

Mistake 7: Fallbacks that hide failures, and gates that can't say "not sure"

When the candidate throws, retry the turn on the current model, so users on the canary get a slower answer instead of an error. Good. But if the retry also hides the failure from your metrics, you've built a slower way to promote a broken model. In the demo, a candidate that failed every third call caused 32 failed turns in 200 requests and zero user-visible failures, and the gates still saw all 32. A hard stop (at least 5 failures at a 20% rate or more) rolled it back immediately.

The second half of this mistake is a gate with only two answers. With small samples, "0 violations in 400 turns" doesn't prove much. The project's rate checks use a Wilson interval and have three outcomes: clearly worse (roll back), clearly fine (pass), or not conclusive yet (hold). At a half-point margin, 400 clean turns still leave the upper bound at 0.95%; you need about 800. At 1% of traffic that can take days, which tells you how short a 1% step is allowed to be.

Not every failed gate should roll back, either. Harm (failures, contract violations, p95 latency, negative feedback) rolls back automatically. A correct but slow candidate at 50% was rolled back on a p95 of 1,971 ms against 575 ms. Cost only holds for a person, because a more expensive model can be the right call.

A pre-flight list for your next model upgrade

• [ ] Model id, prompt and options ship together under one fingerprint
• [ ] The shadow compares single turns and can't execute tools
• [ ] Shadow inputs are deep-copied on the request thread
• [ ] Dropped shadow jobs are counted from the drop callback
• [ ] Gates use refusals, tool divergence and flips, not raw agreement
• [ ] Canary buckets come from a salted stable hash, not GetHashCode()
• [ ] Fallbacks protect users but never hide failures from the gates
• [ ] Every rate gate can answer "not conclusive yet"

What I tested, and what I didn't

The project has 206 passing tests (193 unit, 13 WebApplicationFactory integration tests), and none needs an API key. An offline support model has switchable bad habits (rounds refunds, refuses legal questions, fails every third call, runs slow), so each scenario above is a test, and a 10-scene CLI demo runs it all on fake time. The library behaviors I mention (the shallow Clone(), the shared message list, TryWrite under DropWrite, randomized string hashes) were checked with spikes against 10.11.0 and .NET 10, and are pinned as tests.

What I didn't do: run a rollout against real providers with real users. The traffic and every number in this post come from simulated days, so your thresholds will need tuning on your own traffic.

Get the complete project

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. This project is already inside: the full .NET 10 solution with releases and the prompt registry, the shadow pipeline that never acts, the response comparer, the sticky canary router with fallback, the Wilson-bounded gates, the rollout controller with its kill switch, the offline models, a minimal API and the 206 tests, plus a PDF guide that goes much deeper than this post, including how to decide when thousands of shadow disagreements are safe enough to promote. You also get the other complete, tested .NET + AI projects in the membership, with new ones added regularly.

Join Tech Skill Builder for $5/month

Which of these seven is hiding in your current upgrade plan?