重試策略本身就是一次負載測試

Your Retry Policy Is a Load Test

Your retries are traffic. Every one you never counted becomes someone else's p99.

I spent a night tuning an LLM call. The model was fine. My client was the incident. That is the short version.

Here is the longer one. My p99 looked terrible under load. So I blamed the model first. Then I blamed the network. Then I opened my own retry code and felt silly.

The number I was not logging

Amplification is simple. Count attempts, count logical requests, divide. That is your retry tax.

Mine was above three. Meaning every user request was quietly becoming three requests. Nobody had scheduled that load test. My client ran it for free, every night.

Why does nobody log this? Because retries live in a wrapper, and wrappers rarely get metrics. Does your retry branch emit anything at all?

Build the mock before you touch production

You cannot debug a shared endpoint. It moves under you. So I wrote a mock that degrades as concurrency rises.

That is the whole trick. Queueing is the villain, not the model. Here is the single file I used.

# amplify.py - one mock server, one client, one number: amplification.
import asyncio, random, time

INFLIGHT = 0
PEAK = 0

async def call():
    global INFLIGHT, PEAK
    INFLIGHT += 1
    PEAK = max(PEAK, INFLIGHT)
    try:
        # service time degrades with concurrency: that is queueing, not magic
        await asyncio.sleep(0.010 * INFLIGHT + random.uniform(0, 0.02))
        return "ok"
    finally:
        INFLIGHT -= 1

async def logical_request(mode, timeout, sem, stats):
    for attempt in range(4):
        stats["attempts"] += 1
        try:
            if sem is not None:
                async with sem:
                    return await asyncio.wait_for(call(), timeout)
            return await asyncio.wait_for(call(), timeout)
        except asyncio.TimeoutError:
            if mode == "none":
                continue
            if mode == "fixed":
                await asyncio.sleep(0.1)
            elif mode == "full":
                await asyncio.sleep(random.uniform(0, 0.1 * 2 ** attempt))
            elif mode == "decorrelated":
                await asyncio.sleep(min(2.0, random.uniform(0.05, 0.3 * 2 ** attempt)))
    stats["gave_up"] += 1
    return "gave up"

async def run(mode, workers, sem_size, total=2000, timeout=0.25):
    global PEAK
    PEAK = 0
    stats = {"attempts": 0, "gave_up": 0}
    sem = asyncio.Semaphore(sem_size) if sem_size else None
    queue = asyncio.Queue()
    for _ in range(total):
        queue.put_nowait(1)

    async def worker():
        while not queue.empty():
            queue.get_nowait()
            await logical_request(mode, timeout, sem, stats)

    t0 = time.perf_counter()
    await asyncio.gather(*(worker() for _ in range(workers)))
    return {"mode": mode,
            "A": round(stats["attempts"] / total, 2),
            "gave_up": stats["gave_up"],
            "wall_s": round(time.perf_counter() - t0, 1),
            "peak_inflight": PEAK}

if __name__ == "__main__":
    for mode, workers, sem in [("none", 40, 0), ("fixed", 40, 0),
                               ("full", 40, 0), ("decorrelated", 40, 8)]:
        print(asyncio.run(run(mode, workers, sem)))

The queue.empty() check is demo-grade. Do not ship it. It exists to keep the file under a screen.

Also note the timeout is the load injection. A fixed timeout plus rising latency equals manufactured failures. Sound familiar?

What my run showed

Four modes, 2000 logical requests, 40 workers, 250 ms timeout. One run each on my laptop.

These numbers are from the mock above, not from any hosted service. Rerun and they will move. Expect the shape, not the digits.

Read the last two rows honestly. Jitter spread the pain. The limiter removed it. And I changed two variables at once, so I cannot credit jitter alone.

The fix that actually moved the number

Concurrency first, jitter second. A permit you never issue is a retry you never send.

Then cap retries as a fraction of successes. A token bucket does this in ten lines.

class RetryBudget:
    def __init__(self, ratio=0.1, window=10.0, floor=5):
        self.ratio, self.window, self.floor = ratio, window, floor
        self.ok = self.retries = 0
        self.t0 = time.monotonic()

    def allow(self):
        now = time.monotonic()
        if now - self.t0 > self.window:
            self.ok = self.retries = 0
            self.t0 = now
        return self.retries < max(self.floor, self.ok * self.ratio)

    def record(self, success):
        if success:
            self.ok += 1
        else:
            self.retries += 1

Call record(False) before every retry attempt. Call allow() first. If it says no, fail fast and tell the caller why.

A failing fast client is a system that recovers. A stubborn client is a system that melts. Which one are you running today?

Where a free tier fits in the loop

My harness needs two things: a real model call and a machine that is not my laptop.

That is where MonkeyCode's free model access and free server option fit for me. The app under test makes live calls against free model access. The mock server and the generator live on the free server option. My laptop stays cool and my night stays short.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. It is an open-source project, so you can read the client code before trusting it.

One hard rule. Do not point this generator at a shared free endpoint. Shared capacity is not your test rig. Aim the load at the mock, always. Check the project page for the current terms and limits yourself, because offers change and I will not quote stale numbers.

When this approach is wrong

This is a lab, not a load test. The mock is a caricature of queueing. Do not report vendor numbers from it.

If you have a hard SLO and real traffic, you need real staged load tests. If you are debugging your own client, this is enough. Know which one you are doing.

Also: amplification above one is sometimes correct. Some idempotent reads deserve a retry. Just make it a decision, not an accident.

Your turn

Add one counter tomorrow. Log attempts and logical requests side by side. Watch what your client has been doing at 2 a.m.

Then run the four modes above against the mock. If you try it on the free tier, I want your amplification number. Mine was 3.4, and it was mine the whole time.