遷移前先量度本機優先智能體與免費伺服器的部署差異

An agent that feels instant on a laptop can feel broken the moment its model call crosses a network boundary, and the postmortem usually blames the model. The regression more often comes from three numbers nobody recorded before the move: cold-start latency, payload size leaving the device, and behaviour when the remote seat is unreachable. The scenario below is illustrative rather than a customer story, but the failure shape is common enough to plan around.

This article builds a small probe that records those three numbers for any endpoint pair and turns them into a residency decision. It assumes you already have an agent that runs end to end and you are choosing where the model call happens. Nothing here depends on one vendor, and every threshold below is a starting point you should retune for your own product.

Why three measurements beat one benchmark

Public leaderboards measure model quality on fixed prompts, and they say almost nothing about the machine your agent actually runs on. Residency is a property of your process, your network path and your prompt size, so it has to be measured in place. A single latency number is also easy to game: a warm local runtime can win on the median and lose badly on the first request after a laptop wakes up.

The probe therefore records a distribution rather than a point estimate, and it pairs that distribution with two non-latency signals. The payload audit answers what content leaves the device, which matters when prompts carry repository context or customer records. The offline check answers whether the agent degrades into a queue or into an exception, and that difference decides how much of the fallback path you have to build.

Step 1: Fix one request shape for both endpoints

The comparison is only meaningful when both sides receive the same bytes, so define one request body and reuse it. Keep the system prompt and user prompt in files, and let environment variables supply the base URLs. This also makes the probe cheap to rerun whenever a runtime updates.

mkdir -p prompts
printf 'Summarise the three functions in this diff.\n' > prompts/short.txt

export LOCAL_BASE_URL="http://127.0.0.1:8080/v1/complete"
export REMOTE_BASE_URL="https://your-endpoint.example/v1/complete"

The two URLs are placeholders, and the script below does no vendor-specific shaping. If your local runtime expects a different JSON envelope, adapt build_body once and keep the rest identical. That single edit point is what makes the numbers comparable later.

Step 2: Measure first body byte, with explicit honesty about it

Time to first token is the metric developers feel, but most HTTP clients only expose time to first body byte. The probe reads exactly one byte and stops, then labels the result as an approximation. It also reports p50 and p95 separately, because tail behaviour is where remote seats and cold local runtimes diverge.

The script also counts failures instead of hiding them behind a mean. A run that times out is data, and a remote seat that fails two of seven attempts changes the architecture conversation more than a hundred milliseconds of median latency.

#!/usr/bin/env python3
"""residency_probe.py - reference implementation, not a benchmark suite.

Records first-body-byte latency, payload size, secret-pattern hits and
forced-offline behaviour for up to two endpoints. Every number it prints
is yours to collect; no vendor performance is asserted here.
"""
from __future__ import annotations

import argparse
import json
import os
import re
import socket
import statistics
import time
import urllib.error
import urllib.request

SECRET_PATTERNS = {
    "provider_key": re.compile(r"sk-[A-Za-z0-9]{16,}"),
    "bearer_value": re.compile(r"(?i)bearer[ ]+[A-Za-z0-9._-]{12,}"),
    "aws_secret": re.compile(r"(?i)aws_secret_access_key[ ]*[:=]"),
}


def build_body(prompt: str, system: str) -> bytes:
    return json.dumps({"system": system, "prompt": prompt}).encode("utf-8")


def scan_for_secrets(raw: bytes) -> list[str]:
    text = raw.decode("utf-8", errors="replace")
    hits = [name for name, pat in SECRET_PATTERNS.items() if pat.search(text)]
    for key in ("API_KEY", "ACCESS_TOKEN", "SECRET"):
        value = os.environ.get(key, "")
        if len(value) >= 12 and value in text:
            hits.append("env:" + key)
    return hits


def first_body_byte_ms(url: str, body: bytes, timeout: float) -> float:
    req = urllib.request.Request(
        url, data=body, headers={"Content-Type": "application/json"}
    )
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        resp.read(1)  # first body byte, NOT first token
    return (time.perf_counter() - start) * 1000.0


def probe(url: str, body: bytes, attempts: int, timeout: float) -> dict:
    samples, failures = [], 0
    for _ in range(attempts):
        try:
            samples.append(first_body_byte_ms(url, body, timeout))
        except (urllib.error.URLError, socket.timeout, TimeoutError, ConnectionError):
            failures += 1
    ordered = sorted(samples)

    def pct(p: float) -> float | None:
        if not ordered:
            return None
        idx = min(len(ordered) - 1, int(round(p * (len(ordered) - 1))))
        return round(ordered[idx], 1)

    return {
        "url": url,
        "attempts": attempts,
        "failures": failures,
        "p50_ms": pct(0.50),
        "p95_ms": pct(0.95),
        "mean_ms": round(statistics.fmean(samples), 1) if samples else None,
    }


def offline_behaviour(fallback) -> dict:
    """Point at the discard port, then check the fallback still returns work."""
    dead = "http://127.0.0.1:9/v1/complete"
    started = time.perf_counter()
    try:
        first_body_byte_ms(dead, b"{}", timeout=2.0)
        reached = True
    except (urllib.error.URLError, socket.timeout, TimeoutError, ConnectionError):
        reached = False
    result = fallback()
    return {
        "endpoint_reachable": reached,
        "fallback_returned": result is not None,
        "fallback_seconds": round(time.perf_counter() - started, 3),
    }


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--prompt", default="prompts/short.txt")
    ap.add_argument("--attempts", type=int, default=7)
    ap.add_argument("--timeout", type=float, default=20.0)
    ap.add_argument("--out", default="residency_report.json")
    args = ap.parse_args()

    prompt = open(args.prompt, encoding="utf-8").read().strip()
    body = build_body(prompt, system="residency-probe")
    report = {
        "payload_bytes": len(body),
        "payload_secret_hits": scan_for_secrets(body),
        "endpoints": [],
        "offline": offline_behaviour(lambda: {"queued": True}),
    }

    for name in ("LOCAL_BASE_URL", "REMOTE_BASE_URL"):
        url = os.environ.get(name)
        if not url:
            continue
        row = probe(url, body, args.attempts, args.timeout)
        row["name"] = name
        report["endpoints"].append(row)

    with open(args.out, "w", encoding="utf-8") as fh:
        json.dump(report, fh, indent=2)
    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    main()

Run it twice: once with the local runtime freshly started, and once after it has served a few requests. Those two runs are the cheapest way to see cold-start cost without building a load generator.

Step 3: Audit what actually leaves the device

The payload audit is deliberately blunt, and it is not a data-loss-prevention product. It flags obvious credential shapes and any long environment value that leaked into the serialised body, which catches the common accident of pasting a token into a system prompt. A hit is a stop sign for a remote seat and a warning for a local one, because local runtimes log too.

The byte count matters as much as the hits. A prompt carrying an entire diff can be orders of magnitude larger than a chat message, and upload time on a home connection scales with it. Record the byte count next to the latency figures, or you will attribute a bandwidth problem to model speed.

Step 4: Force the offline path on purpose

The probe points at a dead port and then calls the same fallback your agent would call. If the fallback raises instead of returning queued work, the agent needs a fallback contract before any residency change ships. If it returns queued work in a few seconds, the offline story is already credible.

This check is cheap and it prevents the most expensive migration surprise. A remote seat that disappears during a demo is survivable when work queues locally and can be resumed. It is fatal when the whole run lives in a request handler that has already returned.

A decision table you can argue with

The thresholds below are operator-tunable starting points rather than published guidance. Fill the middle columns with your own probe output and use the last two as the interpretive lens.

The table is a discussion device, not a verdict. Two teams with identical numbers can land on different rows because their data classification and support burden differ.

Where a free server genuinely wins

A recurring discussion in developer circles concerns how many intermediaries sit between a developer and a model, and residency is the concrete version of that question. The honest answer is that self-hosting is sometimes the wrong trade, especially when the local machine cold-starts constantly or has no accelerator at all. Spiky workloads, shared team artifacts and short-lived experiments all favour a remote seat that is already warm.

MonkeyCode, an open-source coding agent project, offers free model access and a free server option, which makes it a reasonable second column for the probe above. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Token allowances and server terms change, so read the project page for the current limits before you design around them, and never assume a free tier is permanent.

The useful pattern is to run the probe against a free remote seat and against your laptop on the same day with the same prompt file. That produces a defensible comparison instead of an intuition, and it costs one environment variable.

Limitations and who should not use this

The probe measures first body byte, not time to first token, and the two diverge when a server streams headers early. It is a single-machine instrument, so it is not a load test and it will not reveal behaviour under ten concurrent runs. The secret scan is a regex tripwire, not a compliance control, and it will miss bespoke credential formats.

Teams with hard data-residency obligations should keep the model call local and treat the remote column as unavailable rather than slow. Interactive products with tight latency budgets should not adopt a shared endpoint without a warm-up strategy and a measured tail. Anyone expecting deterministic throughput from a free tier should build the queue first and treat the server as an accelerator, not a guarantee.

If you want a second column without provisioning anything, point the probe at a free remote seat first and keep the local runtime as the fallback. The numbers you collect will tell you more than any migration plan written before them.