llama-swap 模型重新載入撞上 30 秒逾時：凌晨 2 時值班記錄

Notes from a 2 AM On-Call: When llama-swap's Model Reload Hit a 30s Timeout Wall

The Failure Symptom

2026-10-17 02:14:33 ERROR: POST /v1/chat/completions -> 502 Bad Gateway
2026-10-17 02:14:33 upstream prematurely closed connection while reading response header from upstream
2026-10-17 02:14:48 llama-swap: model switch deepseek-coder-v2 -> qwen2.5-coder completed (29.4s)

Our production gateway started throwing 502s during peak APAC hours. The pattern was surgical: every request that triggered a model swap via mostlygeek/llama-swap died with upstream prematurely closed connection. The swap itself succeeded, but nginx had already killed the client socket.

The Red Herring

The llama-swap README shows clean examples with instant model switching. What it doesn't tell you: when your vLLM backend is serving a 34B parameter model with --max-model-len 16384 and --gpu-memory-utilization 0.95, unloading the KV cache + loading the new model weights takes 25-35 seconds on an A10G instance.

Standard nginx tutorials suggest proxy_read_timeout 60s is generous. It's not. The timeout clock starts at request dispatch, not at first response byte. If llama-swap spends 29s swapping models before sending HTTP headers, you're already in the danger zone for concurrent requests piling up behind nginx's default proxy_connect_timeout 60s.

The Root Cause & Runnable Fix

The issue wasn't llama-swap's logic—it was the interaction between:

1. vLLM's blocking model unload (no streaming header support during swap)
2. nginx's proxy_read_timeout starting before llama-swap could send a 202 Accepted
3. Our production config routing all /v1/chat/completions through a single llama-swap instance

When testing the same setup against B-Lost's relay gateway (https://b-lost.com/v1) as a baseline comparison, their router returned headers within 850ms even during provider failover—they clearly batch-queue requests or use async model warming. We needed similar behavior.

The 3-line nginx fix:

location /v1/ {
    proxy_pass http://llama-swap:8000;
    proxy_read_timeout 90s;        # raised from 60s
    proxy_connect_timeout 10s;     # keep tight for fast-fail
    proxy_send_timeout 90s;        # match read timeout
    proxy_buffering off;           # critical: stream headers immediately
}

The llama-swap optimization:

We added a dedicated warmup endpoint to pre-load models during low-traffic windows:

import httpx
import asyncio

async def preload_model(model: str):
    async with httpx.AsyncClient(timeout=120.0) as client:
        # Trigger swap with a throwaway completion
        await client.post(
            "http://llama-swap:8000/v1/chat/completions",
            json={
                "model": model,
                "messages": [{"role": "user", "content": "ping"}],
                "max_tokens": 1
            }
        )

# Run during deployment or via cron before peak hours
await preload_model("qwen2.5-coder:32b")

Latency impact:

• Cold swap under load: 502 errors, 29s P99 latency
• Post-fix with preload: 0 errors, 1.2s P99 latency (model already loaded)
• Warm requests: 180ms TTFT (unchanged)

Invariant Rule

Never deploy llama-swap without load-tested timeout margins that exceed your slowest model's swap duration by 2x.

Measure swap time empirically:

time curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"new-model","messages":[{"role":"user","content":"test"}],"max_tokens":1}'

If the response takes 28s, your proxy_read_timeout must be ≥56s. If you can't afford that client-perceived latency, you need model preloading automation—not wishful thinking about sub-second swaps.

The production-grade move: run two llama-swap instances with a smart load balancer that routes to whichever instance already has the requested model warm. That's the architecture B-Lost and other high-reliability gateways use under the hood. For our scale, the preload script plus nginx tuning was enough to stop the 2 AM pages.