在教學前測試免費方案：訓練營實驗的三個檢查點

If your bootcamp lab leans on a free tier nobody has measured, you are not grading skill. You are grading luck. That is the whole thesis of this lab: before students write a single prompt, they write a probe, and the probe sheet is what gets graded.

I have watched cohorts where two students submitted near-identical agent runs and got wildly different scores, because one ran at 09:00 and the other at 21:00. Neither of them could explain the gap. Could your rubric survive that? Mine could not, until I moved the measurement to the front of the exercise.

So here is the lab, end to end: setup, three checkpoints, stretch goals, and a rubric that a second grader can apply without arguing about vibes.

What the lab assumes

• A small cohort (4–12 people) with individual accounts on whatever free tier you are using.
• Python 3.9+ and nothing else. No SDK, no framework, no lockfile archaeology.
• One shared repo per cohort, one probes.jsonl per student, committed as-is.
• Rule zero: observed numbers belong to the observer. Nobody generalizes their measurements to the whole tier.

That last rule matters more than it looks. A probe tells you what your account saw, from your network, in your region, at that hour. It is evidence, not a specification.

Checkpoint 1 — Characterize the endpoint

Students write a probe script that does one thing: sends a request, records what happened, appends a JSON line. No retries on the first pass. No clever abstractions either — you want the raw behavior visible.

The snippet below is labelled as a starting point, not a tested artifact on your provider: it speaks an OpenAI-compatible chat-completions shape, so if your provider's API is not compatible, keep the record format and rewrite post().

# probe.py - one probe, one JSON line, stdlib only.
import json, os, time, urllib.request, datetime

BASE  = os.environ["PROBE_BASE_URL"].rstrip("/")
KEY   = os.environ["PROBE_API_KEY"]
MODEL = os.environ["PROBE_MODEL"]
PATH  = os.environ.get("PROBE_PATH", "/chat/completions")  # adjust to your provider


def post(payload, timeout=90):
    req = urllib.request.Request(
        BASE + PATH,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
        method="POST",
    )
    return urllib.request.urlopen(req, timeout=timeout)


def record(**kw):
    kw.setdefault("at", datetime.datetime.now(datetime.UTC).isoformat(timespec="seconds"))
    kw.setdefault("model", MODEL)
    line = json.dumps(kw, sort_keys=True)
    with open("probes.jsonl", "a") as f:
        f.write(line + "\n")
    print(line)


def probe(name, text, stream=False):
    payload = {"model": MODEL,
               "messages": [{"role": "user", "content": text}],
               "stream": stream}
    t0 = time.perf_counter()
    chunks, body, err = None, None, None
    try:
        r = post(payload)
        if stream:
            chunks = sum(1 for _ in r)   # crude line count, not tokens
        else:
            body = json.loads(r.read())
        ok = True
    except Exception as e:
        ok, err = False, f"{type(e).__name__}: {e}"
    ms = round((time.perf_counter() - t0) * 1000)
    record(probe=name, ok=ok, latency_ms=ms, streamed=stream,
           chunks=chunks, error=err)


if __name__ == "__main__":
    probe("c1-short", "Reply with the single word: ready")
    probe("c1-stream", "Count to five, one number per line.", stream=True)
    probe("c1-toolish", "Call the `add` tool to compute 2+2.")

Run it once per student and check the shape of the output before anything else:

python probe.py
wc -l probes.jsonl      # expect 3, not 0 and not 12

Checkpoint 1 passes when every student has three lines and can point at which of them returned ok: false and why. A student who "fixes" a failing probe by deleting it has failed the checkpoint. That is the single most common failure I see.

Checkpoint 2 — Repeat until the spread is visible

One run is an anecdote. Five runs is a distribution, and a distribution is what a rubric can point at.

for i in $(seq 1 5); do python probe.py; done

python - <<'PY'
import json, statistics
rows = [json.loads(l) for l in open("probes.jsonl")]
ok = [r["latency_ms"] for r in rows if r["probe"] == "c1-short" and r["ok"]]
print(f"n={len(ok)} min={min(ok)} median={statistics.median(ok)} max={max(ok)}")
PY

Do not paste my output into your lab notes, because I do not have your output and neither does anyone else in the room. The deliverable is your n, your spread, and one sentence about what would make you distrust it (small n, one slow hour, a laptop on hotel wifi).

Checkpoint 2 passes when the student reports n honestly and names one confounder. Extra credit for noticing that c1-toolish may fail for a boring reason: the provider never promised tool calls.

Checkpoint 3 — Break it on purpose

Happy-path probes teach nothing. Students inject three faults and record recovery behavior:

1. Bad key. PROBE_API_KEY=not-a-key python probe.py — does the client surface a clear error, or silently write an empty success?
2. Oversized input. head -c 200000 /dev/urandom | base64 -w0 > big.txt then send it. Record truncation, refusal, or a timeout — not a guess about which.
3. Mid-flight interruption. Wrap the call in timeout 3 python probe.py. Then check: did a partial record get written, or nothing at all?

The point is a taxonomy, not a fix list. "Fails loudly" and "fails silently" are different grades of engineering, and only one of them will wake you up at 3 a.m.

The rubric, built to be fair under wobble

This is the part most lab write-ups skip, so here it is as a table a second grader can reuse.

Two rules keep it honest when the free tier wobbles:

• Never reward a lucky success. If a probe passed once and failed twice, the graded artifact is the record, not the win. Success rate is part of the evidence, not a bonus.
• Grade artifacts that exist. Effort, intent, and a screenshot of a chat window are not artifacts. This is also what makes it fair: two graders reach the same score because they are reading the same JSON lines.

If your cohort is big enough for two graders, grade blind and compare. Any disagreement wider than one point on a 4-point scale means the criterion was written too vaguely — fix the rubric, not the student.

Stretch goals

1. Run the probe at three different hours and plot median latency per hour. Now you can talk about traffic dependence with evidence instead of folklore.
2. Add --repeat N and an automatic summary, so checkpoint 2 is one command.
3. Generate a Markdown fact sheet from probes.jsonl with a script. Hand-typed summaries drift from the data within a week.
4. Put the probe on a scheduled CI job. Fail the job when the record schema changes, not when latency moves — schemas are contracts, latency is weather.

Where MonkeyCode fits, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project, and the operator states that it offers free model access plus a free server option — at the time of writing, a 10M free-token allowance as part of that offer. For a bootcamp, that combination is genuinely useful for one narrow reason: checkpoints 1 and 2 need a place to run and a model to hit, and neither should require a budget approval email. Students can probe, record, and re-run on day one.

But please keep the recording discipline. Treat the advertised allowance as the campaign's claim, not as your measurement — put your n, your failures, and your spread in the repo, and re-run the probe before every cohort. Free tiers move; your fact sheet should move with them. And if you need guarantees on throughput, data handling, or uptime for a production workload, this is the wrong tool and the wrong lab — go get a contract with an SLA instead.

If you want to run the three checkpoints as written, the free model access and free server option are enough to get through them, and the repo README is the place to check current terms before you promise anything to students.