Claim Cards：以候選人驗證證據評分的 take-home 面試作業

The short version

A take-home that grades the artifact rewards plausible output, and plausible output is now cheap. A take-home that grades the evidence attached to the artifact rewards engineering judgment, and that evidence is expensive to fake. This article packages the idea as a reusable interview task: the candidate ships a claim card for every assertion the build depended on, plus a checker that re-executes each claim and compares the result to what the card recorded. Candidate time lands around three to four hours; reviewer time lands around thirty minutes.

The design goal is narrow. The task should make the difference between checked and assumed visible in a diff, so that a reviewer can grade verification discipline without re-reading an entire repository. Everything below is a template that a team can copy, trim, and rerun.

Why output-only grading breaks down

Most take-homes ask for a small service, then grade it on structure, tests, and naming. Assistants have collapsed the cost of all three, which means the signal that used to separate candidates has decayed. A candidate who never ran the test suite can still deliver a tidy repository with a green-looking CI badge.

The failure is not dishonesty in most cases; it is unexamined confidence. A generated README says the importer streams rows, the code actually buffers, and nobody measured either. Claim cards attack exactly that gap, because each card must record the command that produced the belief and the exit code that came back.

A useful side effect appears during review. Two candidates can submit near-identical code while their card sets look nothing alike, and the card set is the more predictive document.

The prompt

Hand the candidate the block below verbatim, with the clock stated in writing.

Task: implement CSV import for the attached fixture set (about 200 lines).

You may use any assistant, any model, and any editor. You must also ship:

1. src/            working implementation
2. claims.yaml     one entry per assertion the build relies on
3. notes.md        <= 30 lines: what you refused to claim, and why

Rules:
- Do not modify verify.py; the reviewer runs it unmodified.
- Every entry marked verified must carry a command and the exit code you observed.
- Assertions you could not test go in as status: unverified. That is not a penalty.
- A claim you deleted because you could not test it is worth less than a claim you
  labelled unverified and explained.

The third rule carries most of the weight. Candidates arriving from output-only take-homes tend to either over-claim or silently drop the hard parts, and both behaviours are visible in the file.

The artifact: a claim card schema and a re-executable checker

Three fields make a card useful: the assertion in one sentence, the method that tests it, and the result the candidate actually saw. Anything softer turns the file into prose.

# claims.yaml (schema is the candidate's contract)
- id: C-01
  claim: "CSV files with a UTF-8 BOM import without an offset error"
  method: "python -m pytest tests/test_import.py -k bom -q"
  recorded_exit: 0
  status: verified
  evidence: "1 passed in 0.41s"

- id: C-02
  claim: "The importer streams rows instead of buffering the file"
  method: "no measurement performed"
  recorded_exit: null
  status: unverified
  evidence: "row count memory profile not taken"

The checker stays deliberately small, because a large checker becomes a second implementation the candidate has to debug. It re-runs each verified method, compares exit codes, and flags drift; it never executes a command the reviewer has not already seen in the file.

# verify.py — reference implementation, roughly 60 lines
import subprocess
import sys

import yaml


def run(cmd: str, timeout: int = 120) -> tuple[int, str]:
    proc = subprocess.run(
        cmd, shell=True, capture_output=True, text=True, timeout=timeout
    )
    return proc.returncode, (proc.stdout + proc.stderr)[-300:].strip()


def main(path: str = "claims.yaml") -> int:
    cards = yaml.safe_load(open(path, encoding="utf-8"))
    drifted = 0
    for card in cards:
        cid, status = card["id"], card.get("status")
        if status != "verified":
            print(f"{cid}  SKIP     {status}")
            continue
        code, tail = run(card["method"])
        ok = code == card["recorded_exit"]
        drifted += 0 if ok else 1
        print(f"{cid}  {'REPRODUCED' if ok else 'DRIFT':<10} exit={code}  {tail[:80]}")
    print(f"\ndrifted cards: {drifted}/{len(cards)}")
    return 1 if drifted else 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:]))

The reference above is a labelled starting point rather than a hardened tool. It has no sandboxing, no per-card timeout budget, and no protection against a candidate shipping a conftest.py that lies. Reviewers who accept untrusted code should run it in a disposable container and pin the checker by hash before sending the prompt.

The rubric

Five dimensions, weighted so that a broken implementation with honest cards still outscores a polished implementation with fabricated ones.

A reviewer can apply this table in about ten minutes per submission. The 20-point honesty axis is the one that consistently separates senior candidates from fluent ones, because fluent candidates have the most to gain from over-claiming.

A sample solution, excerpted

A strong submission on the CSV task typically ships five to eight cards, not thirty. One representative excerpt shows the behaviour worth hiring for: the candidate discovered that a buffering assumption did not hold, downgraded the card, then added a small measurement instead of arguing.

- id: C-02
  claim: "The importer streams rows instead of buffering the file"
  method: "python -m pytest tests/test_streaming.py -k peak_rss -q"
  recorded_exit: 1
  status: unverified
  evidence: "peak RSS grew linearly with fixture size; assumption failed"

- id: C-02b
  claim: "The importer buffers the whole file, acceptable below 50 MB"
  method: "python -m pytest tests/test_streaming.py -k peak_rss -q"
  recorded_exit: 1
  status: unverified
  evidence: "threshold stated from fixture sizes, not from a measured limit"

A card that records a failing exit code alongside status: unverified is a stronger signal than a passing suite, because it proves the candidate distinguishes observation from conclusion.

Calibrating the task with free model access and a free server

Interview tasks decay when the rubric has never seen a weak submission. MonkeyCode's free model access makes calibration cheap: generate three or four candidate attempts against the same prompt, keep them as an internal reference set, and score them with the table above before the first real interview. Because the free endpoint is enough for short generation runs, a team can refresh the reference set whenever the prompt changes without a procurement step. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free server option covers the second half of calibration. The checker needs a clean checkout to be meaningful, and a disposable preview environment is the natural place for it; a reviewer wires verify.py into that environment, runs the reference attempts, and records which ones drift. The workflow that matters is not the host itself but the habit of never trusting a card set that has only ever run on the candidate's laptop.

Two guardrails belong in the setup. Reference attempts stay internal, since shipping them to candidates converts the task into pattern matching. And the checker runs against reviewer-controlled infrastructure only, because a candidate-supplied Makefile plus a shell-executing checker is an arbitrary-code path into shared compute.

Limitations and who should not use this

The task costs more to author than a feature ticket, and it grades written reasoning, which disadvantages candidates who write English slowly. It also rewards candidates who already work in verification-heavy environments, so it should be paired with a live discussion where the reasoning is spoken rather than written.

Teams hiring for pure implementation speed should not use it, because the rubric spends 45 points on documentation behaviour. Teams without a disposable runner should not use the checker either; a claim-card task with an unrunnable checker teaches candidates that evidence is decorative.

Common failure modes

1. Cards as a changelog. Entries describe what was built instead of what was tested, and the checker reproduces nothing.
2. Trivially passing methods. python -c "import src" gets marked verified and inflates the reproducibility score.
3. Silent deletion. Hard claims vanish from claims.yaml with no note, which is worse than an honest unverified.
4. Checker drift. The candidate patches verify.py to skip a failing card, which the hash pin catches immediately.
5. Rubric inflation. Reviewers award 2 on coverage for a large card set whose methods do not rerun on a clean checkout.
6. Prompt leakage. An earlier candidate posts the fixture set publicly, and the calibration value collapses.

Closing note

The test of a take-home is whether it still separates candidates after assistants got good. Claim cards survive that shift because they grade a behaviour the assistant cannot perform on the candidate's behalf: deciding what deserves to be called true. Teams that want to try the calibration loop can start with the free model access and free server option, then keep whichever reference attempts still earn their place in the rubric.