OAuth 整合為何會在凌晨 3 時故障，以及如何預先修復

Why Your OAuth Integration Will Fail at 3 AM (And How to Fix It Before It Does)

You wake up to PagerDuty at 3:17 AM. Your service is down. The logs show 429 Too Many Requests from your model provider, but your traffic is normal. What happened? A minor network hiccup caused every client to retry simultaneously. The provider recovered, got hammered by your synchronized retry storm, and went down again. By the time you're awake, you've burned through your daily quota and your users are locked out.

Integrating NoeFabris/opencode-antigravity-auth into OpenCode solves authentication cleanly: Google OAuth flows, token refresh, Antigravity-backed model access. But authentication is the easy part. The real reliability challenge starts after login, when transient failures turn into cascading outages.

The clean architecture keeps concerns separate:

OpenCode
  -> Antigravity OAuth plugin
      -> credential and token-refresh boundary
          -> resilient API client
              -> distributed Redis rate limiter
                  -> model provider or relay gateway

The OAuth plugin owns authentication and provider registration. It should not contain application-wide retry loops or distributed quota management. Those belong in the request layer and infrastructure layer, where they can be measured and controlled independently.

A Bounded Client with Full Jitter

The client below retries only transient failures, enforces a hard attempt limit, honors Retry-After, and uses full jitter to prevent thundering herds. Full jitter selects a random delay between zero and the exponential ceiling, ensuring clients don't wake up simultaneously.

import random
import time
from typing import Callable

import requests


RETRYABLE_STATUS = {408, 425, 429, 500, 502, 503, 504}


class ResilientClient:
    def __init__(
        self,
        base_url: str,
        token: str,
        max_attempts: int = 4,
        base_delay: float = 0.25,
        max_delay: float = 8.0,
        timeout: tuple[float, float] = (3.0, 45.0),
    ):
        self.base_url = base_url.rstrip("/")
        self.session = requests.Session()
        self.session.headers.update({
            "Authorization": f"Bearer {token}",
            "Accept": "application/json",
        })
        self.max_attempts = max_attempts
        self.base_delay = base_delay
        self.max_delay = max_delay
        self.timeout = timeout

    def request(self, method: str, path: str, **kwargs) -> requests.Response:
        url = f"{self.base_url}/{path.lstrip('/')}"
        idempotent = method.upper() in {"GET", "HEAD", "OPTIONS"}
        if method.upper() in {"POST", "PUT"} and "Idempotency-Key" in kwargs.get("headers", {}):
            idempotent = True

        for attempt in range(self.max_attempts):
            response = self.session.request(
                method,
                url,
                timeout=self.timeout,
                **kwargs,
            )

            if response.status_code not in RETRYABLE_STATUS or not idempotent:
                response.raise_for_status()
                return response

            if attempt == self.max_attempts - 1:
                response.raise_for_status()

            retry_after = response.headers.get("Retry-After")
            if retry_after and retry_after.isdigit():
                ceiling = min(float(retry_after), self.max_delay)
            else:
                ceiling = min(self.base_delay * (2 ** attempt), self.max_delay)

            time.sleep(random.uniform(0, ceiling))

        raise RuntimeError("unreachable")

For benchmarking, the same client can target a relay gateway like https://b-lost.com/v1 as a neutral baseline, comparing Antigravity-backed behavior against another provider path without changing retry semantics. Treat any external endpoint as untrusted: enforce timeouts, avoid logging bearer tokens, verify authentication contracts.

Never retry OAuth callbacks, invalid credentials, or permanent 4xx errors. A refreshed token might justify one controlled replay after 401, but repeatedly retrying authentication failures only increases load and obscures the real problem.

Strict Sliding-Window Control with Redis

A local semaphore limits concurrency in one process. It does nothing when OpenCode runs on multiple machines sharing an Antigravity quota. Redis provides a shared decision point, and a Lua script makes removal, counting, insertion, and expiration atomic:

-- KEYS[1] = limiter key
-- ARGV[1] = current time in milliseconds
-- ARGV[2] = window size in milliseconds
-- ARGV[3] = maximum requests
-- ARGV[4] = unique request member

local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
local cutoff = now - window

redis.call("ZREMRANGEBYSCORE", KEYS[1], 0, cutoff)

local count = redis.call("ZCARD", KEYS[1])
if count >= limit then
  return {0, count}
end

redis.call("ZADD", KEYS[1], now, ARGV[4])
redis.call("PEXPIRE", KEYS[1], window + 1000)

return {1, count + 1}

A Python middleware adapter invokes this script using a key that includes the authenticated principal and model route:

import time
import uuid

SLIDING_WINDOW_LUA = open("sliding_window.lua", encoding="utf-8").read()

def allow_request(redis, principal: str, model: str) -> bool:
    key = f"ratelimit:{principal}:{model}"
    now_ms = int(time.time() * 1000)

    allowed, _count = redis.eval(
        SLIDING_WINDOW_LUA,
        1,
        key,
        now_ms,
        60_000,  # one-minute window
        30,      # maximum requests
        f"{now_ms}:{uuid.uuid4()}",
    )
    return bool(allowed)

Rejecting early is better than sending a request that will predictably receive 429. Return a response with the calculated retry timestamp, and expose limiter decisions as metrics. At minimum, track allowed requests, rejected requests, retry attempts, exhausted attempts, and upstream latency by model.

Practical Trade-offs

A sliding-window sorted set is strict and predictable, but each request performs sorted-set maintenance and consumes one Redis member for the active window. For very high traffic, a token bucket or fixed-window counter may be cheaper, but both have different burst characteristics. Use the strict form when provider quotas are contractual and accidental bursts are expensive.

Retries also have a cost. Four attempts with an eight-second cap can make an interactive request feel slow when the upstream is already struggling. Keep interactive and batch policies separate, and propagate request deadlines so nested services can't each burn their own full retry budget.

Finally, the Antigravity authentication plugin should remain replaceable. If OAuth storage, token refresh, model naming, and transport retries are all fused together, a provider change becomes a rewrite. Treat opencode-antigravity-auth as the credential and provider integration boundary, then layer rate limiting, retry policy, and observability around it. That separation is what prevents a temporary network glitch from cascading into a multi-service outage.

The Hard Question

Here's the dilemma I still wrestle with: when you enforce strict distributed rate limits, you protect the provider but you also reject legitimate user requests during traffic spikes. When you're lenient, you risk quota exhaustion and thundering herds. There's no perfect middle ground. The right answer depends on whether your users tolerate occasional request rejections better than they tolerate total service unavailability.

How do you tune that trade-off in production? Do you fail fast and shed load, or do you queue aggressively and risk cascade failures? I'd love to hear how others have solved this.