n8n 備援鏈如何用冷卻機制避免重試故障供應商

Stop Retrying a Dead Provider: The Cooldown Rule for n8n Fallback Chains

This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.

The chain that burned five attempts on a corpse

I had a fallback chain I was proud of: primary model, then two cheaper fallbacks, five attempts total, bounded retry with backoff. Clean on paper. Then one provider went down for an afternoon, and the chain did exactly what it was designed to do — five attempts, all against the dead provider first, then the fallbacks, then a deferred job that would retry the whole thing an hour later and burn five more.

The queue didn't fail. It just got slower and slower, because every job paid the full price of discovering, again, that a provider was down. The fallback chain was working. The retry budget was working. What was missing was memory: the system had no way to remember, even for ten minutes, that a provider was already known-down.

A fallback chain without a cooldown is a chain that re-learns the same outage on every job.

Three pieces, in order

1. Honour Retry-After, then add jitter.

When a provider rate-limits you, it often tells you how long to wait in a Retry-After header. Use it — it's the provider telling you the cheapest wait, and ignoring it is how a rate limit becomes a ban. Then add random jitter to the backoff. Without jitter, every queued job retries at the same second, and the provider that just recovered gets hit by the whole queue at once. Jitter turns a stampede into a trickle.

2. Cooldown: skip a provider that just failed repeatedly.

After N consecutive failures from one provider — I use three — mark it cooled-down for a window (ten minutes is my default) and skip it entirely. Don't send it a probe request "just to check." A probe is still a request, and on a down provider it's a wasted one. The job goes to the next provider in the chain, or to the deferred queue if none is left.

The cooldown lives in the same job row I already keep for dedup and retries: provider, consecutive_failures, cooldown_until. A Code node before the provider call reads it:

// Code node: skip a provider in cooldown, don't probe it
const row = $json.provider_state; // from the job row
const now = Date.now();
if (row.cooldown_until && now < new Date(row.cooldown_until).getTime()) {
  return [{ json: { ...$json, skip_provider: true, skip_reason: 'cooldown' } }];
}
return [{ json: { ...$json, skip_provider: false } }];

When the window expires, the next job tries that provider again normally — no manual reset, no separate "health check" workflow. If it succeeds, consecutive_failures resets to zero. If it fails again, the cooldown re-arms. Recovery is automatic because the cooldown is a timestamp, not a flag someone has to clear.

3. Validate the fallback output, not just its shape.

A fallback model that returns well-formed JSON can still return a useless draft. So I run the same fixed set of sample inputs through every model in the chain and validate each result against the same output schema — and then check whether the draft actually answers the brief, not just whether it parses. And I record which provider produced each draft in the job row, so fallback drafts can be reviewed separately from primary ones. A fallback that silently degrades quality is worse than a fallback that fails loudly, because nobody reviews what looks fine.

What cooldown is not

Cooldown is not giving up. A cooled-down provider doesn't delete the job — the job either moves down the chain or waits in the deferred queue, and the sweeper revisits it after the window. The difference between "dead" and "down" is time, and the cooldown is how you buy that time without paying for it on every job.

Two mistakes to avoid. First, don't cooldown on your own bugs: a 4xx validation error means your request was wrong, not that the provider is down — cooling down on 4xx just hides a bug for ten minutes at a time. Cooldown on 5xx, timeouts, and rate limits, not on bad requests. Second, don't set one cooldown for all providers. A provider that fails twice a week and one that fails twice an hour need different windows; I keep the threshold and window per provider in the same row, and I tune them from the failure counts I'm already recording.

The test that proves it

Mock a provider that always 500s, then run ten jobs through the chain and count requests at the mock. Without cooldown: fifty requests, ten per job, all wasted. With cooldown: three requests for the first job, then zero for the next nine until the window expires. Count at the provider, never at n8n — the n8n execution count will look healthy either way, which is exactly why this bug hides.

Fallback chains are about surviving provider failure. A chain that forgets each failure the moment it happens isn't surviving — it's just retrying with extra steps. Give it a memory, even a ten-minute one.

I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.