KServe 上的機器學習模型金絲雀發布：回放流量、Sidecar 指標與自動回滾

Part of the homelab build series. A football match-outcome model ships every day through Kubeflow Pipelines and KServe on RKE2. Each new model is staged next to the live one, tested, given 10% and then 50% of replayed traffic, measured revision by revision, and promoted or rolled back by the pipeline itself. The service stays highly available throughout: the serving revision keeps its traffic until the new one has earned it.

Canary Releases for an ML Model on KServe: the new revision's share of replayed requests at each stage, from staged at 0% through 58 of 600 at the 10% step and 305 of 600 at the 50% step to promoted at 100%, with any failed check rolling back to 0%

Introduction

A canary with no traffic is a coin you never flip.

The textbook canary goes like this. Send 10% of live requests to the new model, watch error rates and latency against the old one, step up, promote. Every guide says it, and every guide assumes one thing: that there are live requests.

My model has almost none. It predicts Premier League results once a day. The pipeline writes the predictions to the bucket, and the website reads them from there. Nobody calls the model server in normal use. A 10% split of nothing is nothing, so a canary built from the guides would pass every time without measuring anything.

So this canary brings its own traffic, measures each revision from its own sidecar, and decides on its own: promote, or put everything back the way it was. This article is how that works, what KServe actually does under the words "canary rollout", and the traps that look like model failures but aren't.

Where it sits

The model is an XGBoost classifier, served by KServe's built-in XGBoost server from a model directory on Ceph. A Kubeflow pipeline runs every morning:

The last four are this article. They run only if finalize decided the new model is worth serving at all; a model that can't beat the old one on the backtest never gets staged.

The rollout: the new model is staged at 0%, smoke tested, given 10% then 50% of replayed traffic, and promoted to 100%; any failure rolls it back and the old model keeps all the traffic

What KServe means by "canary"

Most of the confusion I ran into, mine included, comes from two different KServe APIs with the same word in them.

The old v1alpha2 API had a separate canary block next to the default predictor. The current v1beta1 API has no such block. A canary in v1beta1 is one field on the predictor, canaryTrafficPercent, plus the fact that KServe keeps the last fully rolled-out revision around:

spec:
  predictor:
    canaryTrafficPercent: 10
    model:
      modelFormat:
        name: xgboost
      storageUri: s3://mlops-data-lake/football/models/<new-version>

Change the model and set the field, and KServe creates a new revision for the new model, gives it 10%, and leaves the previous rolled-out revision serving the other 90%. Then the field means three different things depending on its value:

The last row is the one to remember. Rolling back means setting the field to 0. Deleting it, which sounds like "cancel the canary", does the opposite and ships the new model to everyone. An AI assistant I tried got exactly this backwards, along with the v1alpha2 block, which would have shipped the bad model during an incident.

The payload: replayed matches

Without callers, the canary needs a payload, and the right payload is real inputs with known answers. Finalize already has both. After training it takes the 500 most recent matches from the training set and writes three things per match to the bucket: the features the model sees, the real result, and the new model's own answer.

Real inputs, artificial caller. The canary sends those rows to the model, 600 requests per step, 8 at a time.

Two things this payload can and can't tell you:

• It can catch a broken model: one that loads but answers garbage, answers slowly, or errors on real feature rows.
• It can't tell you the new model forecasts better. Both models were trained on those matches, so comparing them there measures fit, not forecasting. That question is answered earlier, by finalize's walk-forward backtest, before anything is staged.

Staging: the new model at 0%

The deploy step patches the InferenceService with the new model and canaryTrafficPercent: 0, then waits until the new model is actually running. "Actually" is the hard part.

Straight after the patch, the InferenceService still reports the previous revision as Ready, so a plain readiness check returns immediately and the pipeline goes green while the new model may never load. KServe's observedGeneration is no help either; on this version it reported a number higher than the object's own generation. The check that works is to find the revision by the model it carries: KServe annotates every revision with its storage location, so staging is done when the latest ready revision carries the new model's location.

# the annotation the deploy step matches on, per Knative revision
metadata:
  annotations:
    internal.serving.kserve.io/storage-initializer-sourceuri: s3://mlops-data-lake/football/models/<new-version>

The smoke test

Before any traffic moves, a pod in the same namespace calls the new revision directly, on its private service, with one known request. The answer must match finalize's answer for that request within 1e-4. Then it sends a burst of 200 requests from 8 threads: all must succeed, all with the same answer, p95 under a second.

If any of that fails, the pipeline stops there. The field stays at 0 and the old model keeps 100%. I tested this end to end with a one-off pipeline version whose tolerance could never be met: the smoke test failed, promotion never ran, and the serving revision kept all the traffic.

The canary: two steps, measured per revision

The canary step runs as one job and does four things before it moves any traffic:

• asks both revisions directly for every replay row, and checks the new one gives exactly the trained model's answers;
• compares the two models' fit on the replay set (log loss and accuracy against the real results);
• finds each revision's pods;
• checks the shared route answers at all.

Then, for 10% and then 50%:

step:
  set: canaryTrafficPercent = <step>
  wait: until Knative reports the split, plus a few seconds for the sidecars
  replay: 600 requests through the shared route, 8 at a time
  measure: each revision's requests, errors and latency, from its own sidecar
  judge: fail the step if any rule below is broken

How the canary measures each revision: the canary Job replays past matches through Knative's weighted route to both revisions, reads each revision's Istio sidecar counters before and after every step, then rolls back to 0% or lets the promote step remove the field

Measuring per revision without help from the response

The response doesn't say which revision answered it. On a day with no new results, both revisions are the same model and give identical answers, so the answers can't tell them apart either.

The sidecars can. Every model pod runs an Istio sidecar that counts the requests it receives, by caller. The canary reads each pod's counters just before and just after the replay and keeps only its own job's requests. The difference is exact: how many requests each revision got, how many failed, and their latency.

The rules

Before any traffic moves, the run stops if the new revision's answers differ from the trained model's, or if its fit on the replay set is more than 0.02 log loss worse than the serving model's. After that, a step fails if any of these hold:

The two-part latency rule matters at this scale. Both models answer in a few milliseconds, so "3× slower" alone would fail a release over a 4 ms difference.

Rollback and promotion

Once traffic has moved, every way out that isn't a pass rolls back: a broken rule, an exception, even the job being killed. Rollback sets the field to 0 and waits for Knative to confirm the serving revision has everything again.

I tested the rollback the same way as the smoke test, with a one-off pipeline version whose error rule could never be met. The 10% step ran first: the new revision took 70 of the 600 requests, all answered correctly. Then the step failed, the job set the field back to 0, Knative confirmed the serving revision held 100% again, and the run failed before the promote step could start. The failure was forced, not caused by a genuinely bad model, but every move after the decision was the real one.

On a pass, the promote step removes the field. It has to send an explicit null for that:

# JSON merge patch body, in YAML form
spec:
  predictor:
    canaryTrafficPercent: null

The KServe Python SDK can't send that. It leaves out every field that is unset when it builds the request, so "set to nothing" never reaches the cluster, and an SDK-based promote silently does nothing. The pipeline uses the plain Kubernetes client with a merge patch instead.

Promote doesn't stop at "Ready" either. It waits until the revision carrying its own model location holds 100% of the traffic, the same check staging uses. That one line turned out to matter more than I expected; see Two releases at once.

When the inputs change

The canary assumes both revisions read the same rows. Most releases keep that true. One didn't: adding bookmaker odds as features took the model from 23 inputs to 26, and the serving model rejected every 26-input replay row with an HTTP error. That isn't a failure of the new model. There is simply nothing to compare it with, and a 10% split would send a tenth of the traffic to a model that can only fail.

So the canary checks the serving model before any traffic moves. Every revision KServe creates is annotated with the model location it serves, so the canary loads that model file and asks it how many inputs it takes. If the number differs from the replay rows, the input schema changed and the canary skips the split:

canary: "SKIPPED: the serving revision takes 23 inputs and the replay rows have 26, so the input schema changed"
result: canary=skipped
promote: on the strength of the checks that did run

The first version of this check didn't read the model at all. It skipped whenever the serving revision answered the replay rows with an HTTP error, and that turned out to be far too wide; see Two releases at once.

The new revision has, by then, answered every replay row exactly as the trained model does, and passed the smoke test. That is what it is promoted on. It happens once, on the release that changes the inputs. From the next run, both revisions read the same rows again and the split comes back.

Before any traffic moves: the canary's seven checks in order, which of them skip the split and which fail the run, and the serving model keeping 100% until the first split

Two releases at once

The pipeline runs once a day, so I never wrote a lock. Then the lab came up late after a cold boot, I started a manual run to stand in for the missed one, and two minutes later the scheduler started the missed run by itself. Two releases, one InferenceService.

Two releases at once: the manual run and the scheduler's catch-up run on one InferenceService, which run's canary skipped and why, the promote that failed, and the fixes

That first skip was a bug of mine, not the race. The input-change check, as it was then, treated any HTTP error from the serving revision as "different inputs". The real input change answered 500; this was a 503, a pod that wasn't ready yet. An escape hatch that wide skips the split whenever the serving model is briefly unavailable, which is exactly when a careful rollout should wait, not wave the new model through.

The fix takes HTTP errors out of that decision altogether. Whether the inputs changed now comes from the serving model's own file, as described above. When the inputs match, the serving revision has to answer: a 502, 503 or 504, a refused connection or a timeout is retried with backoff for up to three minutes, and if it still can't answer, the canary fails before any traffic moves. Any other error fails at once. The serving model keeps 100% either way, and the run fails loudly instead of shipping without a split. The first release with the new check split normally: 76 of 600 requests to the new revision at 10%, 305 of 600 at 50%, no errors.

Nothing broke for callers: a model that had passed its own smoke test served the whole time. The part that worked is the promote check. A promote that only waited for "Ready" and 100% would have reported success for a model that was never serving. The part that didn't is the canary: the second run skipped its split, because the first run had already moved the field. Both models were the same recipe on the same data that day, so nothing was lost. On a day with a real change, the second model would have shipped without a split.

The fix is cheap and the order matters: check whether the scheduler is about to catch up before starting a stand-in run. A lock on the InferenceService would close it properly, and it's on the list.

The 403 that wasn't the model

The first canary runs failed with 403 on every request through the shared route, while the same requests to each revision's private service worked. Nothing was wrong with either model.

The namespace's mesh policy only admits callers from inside the namespace. The private service goes straight to the pod, carrying the canary job's own identity: allowed. The shared route, though, went through Knative's activator, the component that buffers requests while pods scale. Through the activator, a request arrives carrying the activator's identity, from another namespace: refused.

The 403 that wasn't the model: the canary Job reaches a revision directly as its own namespace and is allowed, but through Knative's activator the request arrives as knative-serving and is refused, until target-burst-capacity "0" takes the activator out of the path

The fix is one annotation on the predictor, which takes the activator out of the path whenever a pod is up:

spec:
  predictor:
    annotations:
      autoscaling.knative.dev/target-burst-capacity: "0"

The staging step sets it on every new revision. A serving revision staged before the annotation existed still had the activator in front of it, so the first run after the change skipped the split and said so, and every run since has split normally.

Who owns the InferenceService: YAML or the pipeline

The InferenceService is not in ArgoCD, and that's deliberate. A canary is a loop: set 10%, measure, decide, set 50%, measure, decide. Git describes an end state, not a loop. If ArgoCD owned this object with self-healing on, it would put the Git values back in the middle of a run.

So the pipeline owns it after the first deploy, and the YAML file in the repo is a first-deploy file only, with a loud header saying so. It matters: before I rewrote it, that file pointed at a three-week-old model and lacked the activator annotation. One kubectl apply would have rolled the model back three weeks and broken every canary run with 403s. The file now matches the live object except for the model location, which the pipeline moves on every release. So kubectl diff against it should show exactly one changed line, storageUri, and anything more is drift.

Watching it in Grafana

The decision is the canary job's, from exact before-and-after sidecar counts. Prometheus is for looking back afterwards, so the Model Serving dashboard has a "Canary runs" row: the canary's error share and p95 latency by revision, one point per run.

Two traps made those panels lie before they were right:

• Empty is not zero. An error-share query divides errors by requests. On a clean run there are no error samples, so the division returns nothing, and a clean run looks exactly like a broken query. The query now falls back to zero wherever the revision had canary requests, and the same formula pointed at known-bad traffic shows 100%, so it can show errors.
• Not every error is the model's. The activator's own requests, all failing with 503 URX,UF (nothing listening yet) or 0 DC (it hung up), come in bursts whenever a pod starts. They are cold boots, not model errors. Counting them would show 100% errors on every revision, so the canary panels count only the canary job's requests.

The dashboard's top row is now a set of traffic lights, green, orange and red, for 5xx share and p95 latency across callers. The first trap came straight back there: with zero errors, the 5xx tile said "no traffic" until its numerator fell back to zero too. A second one was new:

• Last is not now. A stat tile over a seven-day range, reduced to "last value", kept showing a p95 of 488 ms hours after traffic had stopped. Every tile is now an instant query, and "no requests in the last ten minutes" reads as "no traffic" rather than as an old number.

The Model Serving dashboard over the half hour of a scheduled run: the top row reads no real traffic, and the traffic split shows the canary's replay reaching both revisions, 00074 and 00075, with xgboost-canary as the caller

The Canary runs row for the same run: error share 0% on both revisions, and p95 latency between 23.6 and 24.3 ms per revision

A real run

A scheduled daily run, end to end, with nobody watching. It promoted revision 00075 over 00074.

The scheduled run in Kubeflow Pipelines, every step green: backfill matches, publish to Iceberg, Katib sweep, final train and register, stage on KServe at 0%, smoke test, canary at 10% then 50%, promote to 100%

The measured split tracked the requested one: 58 of 600 requests at 10%, 305 of 600 at 50%. Knative splits by weight per request, so a few requests either way is normal; the rule only fails a step below a quarter of the expected share.

The canary step's own log for that run: the replay set, equal fit for both models, the same answer on 500 of 500 rows, 58 then 305 of 600 requests to the new revision with no errors, and canary=passed

How reliable is it?

The model service stays available through every release: the serving revision keeps its traffic until the new one has passed every check, and a failure at any stage leaves it exactly as it was.

Honest limits

• The wait-then-fail path for an unavailable serving revision hasn't happened live yet. Tests cover it; the live case so far is the one from before the fix.
• Both failure paths were tested with forced failures. The smoke test and the canary each failed a live run on an impossible threshold. A genuinely bad model would take the same path, but none has come along yet.
• It catches broken models, not worse forecasts. Both models have seen the replayed matches. Forecasting quality is decided earlier, by the walk-forward gate.
• On a day with no new results, both revisions are the same model. The canary then tests the rollout machinery more than the model.
• A release needs room for two predictors. The new revision runs next to the serving one, so the namespace quota must fit both for the length of a run.
• One release at a time is assumed, not enforced. There's no lock on the InferenceService. Two concurrent runs end with one model serving and the other run failed, which is safe, but the second run's canary is skipped.
• Some releases skip the split by design. A first deploy and a change of inputs both promote on the smoke test and the replay check alone. The input change is read from the serving model's file, so a serving revision that is briefly unavailable no longer counts as one.
• One pod per revision. Each revision runs a single pod, two pods in total while a canary runs, and nothing has been load-tested beyond the canary's own 600 requests. That suits a model called by its own pipeline, not a public endpoint.

To follow the rest of this build, 🚀 Follow me on LinkedIn

Author: George Ezejiofor