為免費模型通道設置 Kill Switch，預先設計故障回退

Put the Free Lane Behind a Kill Switch

Borrowed capacity fails differently than capacity you own. Design the exit first, then wire the discount.

That is the whole review in one line. Everything below covers the seam, the data flow, the failure domains, and the drill that proves the seam holds.

The constraint you cannot negotiate

A free model endpoint has no SLA you can enforce. A free server has no restart notice you can plan around. You do not get to page anyone when they vanish.

So the boundary must absorb the loss. Your call sites should not move an inch.

You also cannot assume the terms stay the same. Free today is a fact about today. It is not a contract, a roadmap, or a promise about next quarter.

Write the code as if the lane is already scheduled for removal. Then the removal is a config change, not an incident.

Treat borrowed capacity as one module with one door. One interface, one deadline, one fallback path. Delete the module and the system still starts and still serves.

That is the test I apply to every free-tier dependency. If deleting it breaks startup, it is not a dependency. It is the foundation, and you built on sand.

Data flow, drawn as arrows

A job enters a dispatcher. The dispatcher asks a lane for a completion, with a deadline attached to the call. The lane either answers inside that deadline or the dispatcher moves on.

Every prompt is also an egress event. Your text leaves your network and lands on a machine you do not control.

Log a hash and a byte count, not the full body. Unless your own policy says the body is safe to keep, keep the hash.

Keep the response shape identical across lanes. Callers should never branch on which lane answered the request.

If any caller asks "was this free?", your abstraction already leaked. Fix the interface, not the caller.

Four failure domains worth naming

The first two are loud. The third is the one that costs you a weekend.

A free lane can swap model versions without telling you. Your parser still succeeds. Your schema still validates. Your answers are just quietly worse.

Pin one prompt set, store the outputs, and diff them on a schedule. Ten fixed prompts beat a long postmortem written three weeks later.

Deadline math you can argue about

The remaining budget is the job deadline minus elapsed time. Never hand a lane more time than the job has left.

This one rule kills most of the retry storms I see. A retry that starts after the deadline is wasted work with extra load.

Track the deadline in a monotonic clock. Wall clocks jump, and a jump turns your timeout into a hang.

The seam, in about forty lines

Here is a runnable sketch. It is illustrative Python, not a published library, and you should adapt the client calls to your SDK.

# lane.py
from __future__ import annotations

import time
from dataclasses import dataclass
from typing import Protocol


@dataclass(frozen=True)
class Job:
    prompt: str
    deadline_s: float = 8.0


class Lane(Protocol):
    name: str
    def complete(self, job: Job, budget_s: float) -> str: ...


class FreeLane:
    name = "free"

    def __init__(self, client):
        self._client = client

    def complete(self, job: Job, budget_s: float) -> str:
        return self._client.complete(job.prompt, timeout=budget_s)


class PaidLane:
    name = "paid"

    def __init__(self, client):
        self._client = client

    def complete(self, job: Job, budget_s: float) -> str:
        return self._client.complete(job.prompt, timeout=budget_s)


class Breaker:
    def __init__(self, fails: int = 3, cooldown_s: float = 60.0):
        self.fails = fails
        self.cooldown_s = cooldown_s
        self._count = 0
        self._open_until = 0.0

    def allow(self, now: float) -> bool:
        return now >= self._open_until

    def record(self, ok: bool, now: float) -> None:
        if ok:
            self._count = 0
            return
        self._count += 1
        if self._count >= self.fails:
            self._open_until = now + self.cooldown_s
            self._count = 0

Now the router. It walks lanes in order and stops at the first answer inside the deadline.

# router.py
def route(job, lanes, breaker, clock=time.monotonic, audit=print):
    start = clock()
    for lane in lanes:
        remaining = job.deadline_s - (clock() - start)
        if remaining <= 0:
            break
        if lane.name == "free" and not breaker.allow(clock()):
            audit("skip free: breaker open")
            continue
        try:
            out = lane.complete(job, remaining)
        except Exception as exc:
            audit(f"{lane.name} failed: {type(exc).__name__}")
            breaker.record(False, clock())
            continue
        breaker.record(True, clock())
        audit(f"{lane.name} served, {len(out)} chars")
        return out, lane.name
    raise TimeoutError("no lane answered inside deadline")

Notice what the router does not know. It does not know model names, prices, or hosts. It knows order, deadline, and whether the breaker is open.

That ignorance is the feature. It is why you can delete the free lane in one commit.

The drill that proves it

The test below fails the free lane on purpose. Then it asserts the paid lane served the request.

# tests/test_router.py
import pytest
from lane import Job, FreeLane, PaidLane, Breaker
from router import route


class SlowClient:
    def complete(self, prompt, timeout):
        raise TimeoutError("deadline")


class FastClient:
    def __init__(self):
        self.calls = 0

    def complete(self, prompt, timeout):
        self.calls += 1
        return "ok"


def test_free_failure_falls_back_to_paid():
    paid = FastClient()
    lanes = [FreeLane(SlowClient()), PaidLane(paid)]
    out, name = route(Job("hello"), lanes, Breaker(fails=1))
    assert (out, name, paid.calls) == ("ok", "paid", 1)


def test_breaker_opens_after_repeated_failures():
    b = Breaker(fails=2, cooldown_s=30)
    b.record(False, 100.0)
    b.record(False, 101.0)
    assert b.allow(102.0) is False
    assert b.allow(131.0) is True

Run it before you trust the free path, not after.

python -m pytest -q tests/test_router.py

Then do the part most people skip. Turn the lane off and run every test you have.

FREE_LANE=off python -m pytest -q

If anything beyond a latency assertion breaks, you leaked the free lane into your logic. Fix the leak before you add features on top of it.

Do the same against the free server. Kill it and watch your startup path.

curl -sS -m 3 http://127.0.0.1:8080/health || echo "free server down"

Your app should start, degrade, and write one clear log line. It should not crash. It should not hang waiting for a socket that will never answer.

Where MonkeyCode fits, honestly

If you need a free lane to rehearse this against, MonkeyCode is an open project that offers free model access and a free server option, per the team. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Verify the current terms on their site before you plan around them. I am not quoting quotas, model names, or hardware here. Those numbers move, and I have not measured them on your workload.

Use it as the removable part, not the foundation. The lesson lives in the drill, not in the discount.

What I would change next

Replace the hardcoded lane order with a list read from config. Add a per-lane counter so exhaustion becomes a state you observe, not an exception you catch by accident.

Then add a quality canary. Fixed prompts, stored answers, a nightly diff. When the free lane drifts, a diff tells you first.

Last, record which lane served each job in your audit log. Without that field you cannot answer the only question that matters later. What changed, and when did it start?

Who should skip this

Skip this if you have a hard latency SLO you must meet on every call. Skip it if the data cannot leave your network boundary. Skip it if you need a pinned model version to reproduce results months later.

Free capacity is a fine place to prototype and a bad place to depend. That sentence survives every pricing page you will ever read.

Run the kill-switch drill on whatever free lane you already have. If you want a practice target, MonkeyCode's free tier is one option to point it at.