你的 Agent 是訪客：Agent 羣組的出口流量政策

Most agent code I review is careful about what it does inside the sandbox and careless about what it does on the way out. Then it hits a site that returns 403 to anything with python-httpx in the User-Agent, retries forty times, and burns an afternoon.

The web is getting less tolerant of agents. Publishers are blocking crawlers outright, CDNs fingerprint TLS and headers, and more sites are moving to "pay for the API or go away." That's not a temporary mood — it's the steady state. If your fleet fetches anything from a domain you don't own, egress is a subsystem you have to design, not a client.get() call you sprinkle through a tool.

This is the view from the proxy layer: what breaks, what to log, and what to put in front of the agents.

Two kinds of fan-out, one shared bottleneck

There are two shapes of agent workload that both look fine in a notebook and both break at fleet scale:

• Breadth-first research. Thousands of agent instances each chase a slightly different query against public sources. OpenAI's biological-data work is the well-known version of this shape; the scaling problem is dedupe, not compute.
• Depth-first tool use. One agent loops over a task and calls ten external APIs per step. The scaling problem is that a single badly-behaved loop can generate more traffic than your whole test suite.

In both cases the failure is the same: every agent has its own HTTP client, its own retry policy, and no shared notion of "we have already asked this host this question." A per-agent rate limiter is not a rate limiter. It's a multiplier.

Put egress in one place — a sidecar, a proxy process, or a single service the agents call — so the rate limiter, the robots.txt cache, and the dedupe index are shared. Sandboxed runtimes for agent fleets, OpenShell being one of the recent entries, generally give you a network boundary to hang this on. Use the boundary; don't let agents dial out directly.

What a block actually looks like

Not all rejections are 403, and the ones that aren't are worse:

The soft block is the one that burns people. Your extractor returns zero rows, the agent concludes "no results," and writes a confident summary of an empty page. Check the parse yield per host — if a host that used to return 40 links per page starts returning 0 with a 200, that's a block, not a content change.

Rotating IPs or spoofing a browser User-Agent to get past this is an arms race you will lose, and it's also a contract problem: you're evading a technical measure that expresses the operator's preference. The engineering answer is to buy the API, license the data, or drop the source.

A gate you can actually drop in

Python 3.10+, httpx. This is deliberately small: one class, one shared cache, fail-closed on unknown hosts.

import time
import urllib.robotparser
from collections import defaultdict
from urllib.parse import urlparse

import httpx

class EgressGate:
    """Shared politeness + robots.txt gate. One instance per fleet, not per agent."""

    def __init__(self, user_agent: str, min_interval_s: float = 1.0):
        self.ua = user_agent
        self.min_interval = min_interval_s
        self._last_hit: dict[str, float] = defaultdict(float)
        self._robots: dict[str, tuple[float, urllib.robotparser.RobotFileParser]] = {}
        self._client = httpx.Client(
            headers={"User-Agent": user_agent},   # identify yourself, always
            timeout=httpx.Timeout(10.0, connect=5.0),
            follow_redirects=True,
        )

    def _robots_for(self, url: str) -> urllib.robotparser.RobotFileParser:
        host = urlparse(url).netloc
        cached = self._robots.get(host)
        # Cache for an hour. Re-fetching robots.txt per request is itself abuse.
        if cached and time.monotonic() - cached[0] < 3600:
            return cached[1]
        rp = urllib.robotparser.RobotFileParser()
        rp.set_url(f"https://{host}/robots.txt")
        try:
            rp.read()
        except Exception:
            rp.disallow_all = True  # fail closed: unknown host, stay out
        self._robots[host] = (time.monotonic(), rp)
        return rp

    def fetch(self, url: str) -> httpx.Response | None:
        host = urlparse(url).netloc
        if not self._robots_for(url).can_fetch(self.ua, url):
            return None  # policy skip, not an error - log it separately

        wait = self.min_interval - (time.monotonic() - self._last_hit[host])
        if wait > 0:
            time.sleep(wait)
        self._last_hit[host] = time.monotonic()

        for attempt in range(3):
            resp = self._client.get(url)
            if resp.status_code == 429:
                # Retry-After is sometimes an HTTP date; parse defensively in prod.
                delay = float(resp.headers.get("Retry-After", 2 ** attempt))
                time.sleep(min(delay, 60.0))
                continue
            break

        if resp.status_code == 403:
            raise PermissionError(f"{host} refused agent traffic")
        return resp

Two things worth calling out. min_interval is a global per-host floor, so two agents racing on the same domain queue behind each other. And the 403 raise is deliberate: a 403 should propagate up and kill the sub-task, not get swallowed by a try/except: pass three frames down.

Dedupe before you fan out

If you're running a thousand agents against a fixed corpus, the cheapest win is not making the request twice. Hash the (method, url, normalized_query) tuple, claim it in a shared store with an atomic SETNX, and let the loser wait on the winner's result rather than re-deriving it. The failure mode without this is ugly: 2,000 agents independently discovering the same 40 pages, then 2,000 polite rate limiters all hammering the same host at one request per second.

Store the fetched payload keyed by URL and content hash, with a TTL tied to how volatile the source is. Don't store bodies you aren't licensed to retain — that's a policy decision your gate should be able to enforce by host, not a judgement call at extraction time.

Log the decision, not just the response

Per fetch, emit: host, status, bytes, latency, the policy decision (allowed / robots_skip / blocked), the dedupe key, and the agent ID that asked. Then alert on two ratios: policy-skip rate per host, and parse yield per host. Both going wrong at once is the signature of a soft block. Neither appears in a status-code dashboard.

Keep the agent ID. When someone asks why your fleet hit their site 8,000 times, "which agent, doing what, under which task" is the only answer that ends the conversation.

Pitfalls

• Fetching robots.txt inside the request path. You've turned one request into two and added a per-request dependency on a host that may rate-limit it.
• A rate limiter per agent process. Ten agents, ten limiters, ten times the intended rate.
• Retrying 403. It will not become a 200, and now you've doubled the signal that you're a problem.
• Treating 200 as success. Challenge pages, consent walls, and JS shells all return 200 with no content.
• Ignoring Retry-After and using fixed 1s backoff. Fixed backoff synchronizes your retries into a thundering herd; add jitter.
• Rotating egress IPs as a fix. It works for a week and converts a rate-limit problem into a legal one.
• One User-Agent string for everything. You lose the ability to explain your traffic, and you can't be allowlisted.

Monday checklist

• [ ] All agent HTTP traffic goes through one egress component; agents have no direct network access.
• [ ] robots.txt is fetched once per host and cached with