未量度的容量等同不存在：MonkeyCode 免費模型存取及伺服器選項的實測方法

At 2 a.m. on a Thursday, a demo I had rehearsed four times died on the first request, in front of six people, on a free server I had never once looked inside. The endpoint answered locally. The migration ran on my laptop. The only thing that had changed was the box, and I had no number for that box, no memory ceiling, no cold-start figure, no idea whether the disk could take an eight-megabyte write. So here is my position, and I will argue it rather than hedge it: capacity you have not measured is capacity you do not have, and a free tier is only useful to the degree that you interrogate it.

That sounds like a complaint about free infrastructure, and it is not. MonkeyCode's free model access and free server option are, as far as this article is concerned, an operator-supplied starting point: a place where real latency, real tokenization and real resource limits exist, instead of the tidy fakes I used to write for myself at midnight. The free tier is valuable precisely because it is not polite. It returns errors you did not design for, and it punishes assumptions you did not know you had made. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

But a free tier is a starting position, not a verdict, and the interesting engineering question is not "can it ship?" but "what can it prove before I spend a week on it?" I want two artifacts out of that question. The first exercises the model seam so the call fails the same way twice. The second measures the server the moment it boots, because the number you never recorded is the number that wakes you up.

The seam: make the provider answer to a contract, not to a mood

If your application calls out to a hosted model, the risk is rarely the happy reply, which works. The risk lives in the three shapes that are almost the same and not quite: a timeout that should be retried, an error that should not be, and a response that parses as JSON while violating your schema. So I keep one module with a single narrow function, and I test that function against two providers, a fake I control and the real endpoint I do not.

# app/llm.py — illustrative request shape; copy the path your provider documents
import json, os, httpx

BASE_URL = os.environ["LLM_BASE_URL"]          # real endpoint or the local fake
MODEL    = os.environ["LLM_MODEL"]
TIMEOUT  = float(os.environ.get("LLM_TIMEOUT", "20"))

class ProviderTimeout(RuntimeError): ...
class ContractViolation(RuntimeError): ...

def summarize(prompt: str) -> dict:
    try:
        r = httpx.post(
            f"{BASE_URL}/chat/completions",
            json={"model": MODEL, "messages": [{"role": "user", "content": prompt}]},
            timeout=TIMEOUT,
        )
    except httpx.TimeoutException as e:
        raise ProviderTimeout(str(e)) from e

    if r.status_code >= 500:
        raise ProviderTimeout(f"upstream {r.status_code}")
    if r.status_code >= 400:
        raise ContractViolation(f"client {r.status_code}")

    text = r.json()["choices"][0]["message"]["content"]
    try:
        data = json.loads(text)
    except json.JSONDecodeError as e:
        raise ContractViolation("non-JSON body") from e
    if not isinstance(data.get("summary"), str) or not data.get("bullets"):
        raise ContractViolation("schema mismatch")
    return data

Nothing clever, and that is the point: every provider-specific quirk is squeezed into two exception types the rest of the app already understands. Now the same test file runs twice, once against the fake and once against the real thing, with only an environment variable changing.

# tests/test_generation_seam.py
import os, pytest
from app.llm import summarize, ProviderTimeout, ContractViolation

REAL = os.environ.get("RUN_REAL_PROVIDER") == "1"

@pytest.mark.parametrize("payload,expected", [
    ({"choices": [{"message": {"content": '{"summary": "ok", "bullets": ["a"]}'}}]}, "pass"),
    ({"choices": [{"message": {"content": 'sorry, I cannot'}}]}, "violation"),
    ({"choices": [{"message": {"content": '{"summary": 1}'}}]}, "violation"),
])
def test_response_shapes_are_classified(payload, expected):
    ...  # drive the fake, assert the exception type

@pytest.mark.skipif(not REAL, reason="needs a live endpoint")
def test_live_endpoint_honours_our_timeout():
    # force a deliberately tiny timeout and confirm we get ProviderTimeout,
    # not a hung worker or a bare socket error leaking past the seam
    ...

That skipped test is the one I care about most. Running it against the free model access tells me whether my timeout budget is realistic on real infrastructure, and it tells me in twenty seconds instead of during a review. It also gives me evidence for the only claim I am willing to make: this seam behaves correctly under the failures I chose to name. It says nothing about throughput, nothing about a paid model's quality, and nothing about tomorrow, because availability terms on any free tier are the operator's to change, not mine to assume.

The box: probe it at boot, or you are guessing in public

Here is the part I got wrong at 2 a.m. I had built a beautiful provider seam and deployed it to a server whose memory ceiling I had never read, which meant I had tested my code and shipped my ignorance. So now every deployment writes one honest row about itself before it serves anybody.

# app/probe.py
import json, os, platform, sqlite3, tempfile, time

DDL = """CREATE TABLE IF NOT EXISTS runtime_probe (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    rss_mb REAL, disk_write_mb REAL, disk_write_ms REAL,
    json_ops INTEGER, python TEXT, note TEXT
)"""

def rss_mb() -> float:
    with open("/proc/self/status") as f:
        for line in f:
            if line.startswith("VmRSS:"):
                return int(line.split()[1]) / 1024
    return -1.0

def disk_probe(mb: int = 8):
    payload = b"x" * (mb * 1024 * 1024)
    t0 = time.perf_counter()
    with tempfile.NamedTemporaryFile(dir=os.environ.get("PROBE_DIR", "/tmp"), delete=True) as f:
        f.write(payload); f.flush(); os.fsync(f.fileno())
    return mb, (time.perf_counter() - t0) * 1000

def json_probe(seconds: float = 1.0) -> int:
    blob = {"k": [i for i in range(500)]}
    n, t0 = 0, time.perf_counter()
    while time.perf_counter() - t0 < seconds:
        json.dumps(blob); n += 1
    return n

def record(db_path: str, note: str = "boot") -> dict:
    mb, ms = disk_probe()
    row = {"rss_mb": rss_mb(), "disk_write_mb": mb, "disk_write_ms": ms,
           "json_ops": json_probe(), "python": platform.python_version(), "note": note}
    with sqlite3.connect(db_path) as c:
        c.execute(DDL)
        c.execute(f"INSERT INTO runtime_probe ({','.join(row)}) VALUES ({','.join('?'*len(row))})", tuple(row.values()))
    return row

Expose the latest row at GET /probe/latest and you have a permanent, boring, honest answer to "what was this thing actually running on?" Later, when someone swears the endpoint got slower, you compare two rows instead of two memories. You are not benchmarking a provider, and you must not publish these numbers as a comparison against hardware you never touched, because that is how a measurement becomes a lie.

What the free tier can prove, and what it cannot

The middle column is the whole argument. Free infrastructure is an excellent instrument for falsifying your own assumptions, and a terrible foundation for promises you make to other people. Who should not use this approach? Anyone with an uptime clause, a regulator, or a paying customer who would notice the difference between a good week and a guarantee. Who should? Anyone with a prototype and a suspicion that the prototype is lying.

So point your client at the free model access, run the seam with the live flag on, deploy the probe, and read the row it writes. Then ask yourself which layer handed off to which, and where you would find out first. Tell me which handoff in your stack is the least stable, the model call, the write, or the deploy, and paste the exact failure state or status code you see, because that is the number I want in my test file next.