以 MonkeyCode 為例，如何把免費推理服務拆分為三個平面

Free inference is not a socket you plug in. It is a plane you place in a diagram.

Sketch three planes before you write a client. One plane holds your state. One plane answers prompts. One plane runs work that outlives a single request.

If you cannot name which plane failed, you are not ready to ship on a free tier. Here is the review I would run on that sketch.

Constraints set the shape

Three constraints decide the design. The model behind a free endpoint can change without telling you. Throughput is shared, so latency is a distribution and not a number. The machine that runs your jobs is not yours to patch at 2 a.m.

A wallet mental model fails here. Budgets feel finite. Shared capacity is not yours to reason about. You need a pipeline view, not a balance.

None of those constraints are defects. They are inputs. Design against them, not around them.

MonkeyCode is the concrete case in this review. It offers free model access and a free server option, which map to the middle and last planes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The advertised allowance is 10M tokens at the time of writing. Re-read the current terms before you size a batch.

One rule keeps the flow one-way

Your ledger owns state. The model plane owns nothing. The compute plane owns nothing you cannot rebuild.

That rule turns data flow into six steps. Create a job row, assemble the prompt, hold a token estimate, capture the response, store the artifact, trigger the run. Every step writes to your side of the fence. None of them read state back from the provider.

Now the part that saves you a night. Refuse work before you spend, not after.

# budget.py — starter, no dependencies, single-flight only
from dataclasses import dataclass

@dataclass
class Ledger:
    ceiling: int
    spent: int = 0
    held: int = 0

    def hold(self, estimate: int) -> bool:
        if self.spent + self.held + estimate > self.ceiling:
            return False
        self.held = estimate
        return True

    def settle(self, actual: int) -> None:
        self.held = 0
        self.spent += actual

Call hold with a rough estimate from your prompt length. Call settle with the usage the response reports. If hold returns false, the job waits in a queue you can see.

That queue is the whole point. A refusal you can inspect beats a 429 you cannot.

Run it once with a tiny ceiling to prove the refusal path fires. You want that failure in a test, not in a demo.

Check the lane before the batch

Free endpoints move. Treat reachability as a measurement, not an assumption. Run a canary before a batch, never during one.

#!/usr/bin/env bash
set -euo pipefail
: "${MC_BASE_URL:?set MC_BASE_URL}"
: "${MC_API_KEY:?set MC_API_KEY}"

curl -sS --max-time 20 "$MC_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $MC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"reply with ok"}],"max_tokens":8}' \
  | grep -q '"content"' && echo "canary: up" || { echo "canary: down"; exit 1; }

Adjust the path to your provider's shape. The check is not the point. The habit is. Gate the batch on the canary, and a bad afternoon becomes a delayed job instead of a half-written artifact.

Three blast radii, not one

Naming failure domains is cheap. Bounding them is the work. Draw a circle around each plane and ask what leaks out of it.

The ingress radius covers prompt assembly. A bad template corrupts every job downstream. Version the template and stamp the version on each row.

The model radius covers inference. A timeout here costs one held estimate, nothing more. Keep it that way by never letting a response write directly to your system of record.

The compute radius covers the free server. Restarts are weather, not incidents. Make every job idempotent so a restart replays cleanly and the artifact lands once.

Read the table top to bottom. The last row is the one people skip. It is also the one that quietly spends your allowance while you sleep.

What I would change next

Move prompt assembly behind a versioned template first. Your ledger is only as honest as the thing that produced the estimate.

Then make the ledger durable and single-flight. The starter above assumes one writer. Real batch runners need a lock or a queue. Until then, keep concurrency at one and say so in the README.

Finally, put a fallback model behind the same interface. Do not build a router with five branches. Build one seam and change what sits behind it.

Measure queue depth weekly. If it grows, your estimate is wrong, not the model.

Who should not use this

Do not build on a free tier if you need pinned model versions in a contract. Do not send regulated data you cannot share with a third party. Do not accept cold starts if you sold an availability number. Pay for the guarantee instead.

Everyone else should start here. Sketch the planes, prove the refusal path, gate on a canary. Then move real work through it.

Try the free model access and the free server with one canary job, not your backlog. If the diagram survives that, it survives Tuesday.