教學：用事件記錄檢查審核者是否查看審核卡片

Last month I sat in on a review session that lasted eleven minutes.

Twelve agent actions got approved. Nine went through in under two seconds.

Nobody was careless. That is the point. The card was simply easy to click past.

So what did we actually approve? The support lead owned that decision. A refund had gone out against a ledger nobody opened. It was reversible for twenty-four hours, which is the only reason it became a lesson instead of an incident.

This tutorial does not improve the card. It builds the instrument that tells you the card failed. We go from empty folder to a running audit, with a verification step at each stage.

Step 0: Decide what "read" means before you log anything

You cannot measure comprehension. You can measure three observable proxies.

Dwell time is the first. Evidence expansion is the second. Editing before approving is the third.

None of them is proof. A fast approval can be correct. A slow one can be lazy. Write that on the wall before you start.

Ask the harder question early. Which missing evidence should stop approval here? And which extra panel would only add noise?

Step 1: A log format small enough to keep

One JSON event per line. No schema registry, no vendor SDK.

{"session":"pilot-01","review_id":"r-01","event":"card_open","t":0}
{"session":"pilot-01","review_id":"r-01","event":"evidence_expand","panel":"source_ledger","t":1840}
{"session":"pilot-01","review_id":"r-01","event":"decision","verdict":"approve","t":2310}

Three event names. card_open, evidence_expand, decision. The panel field is what makes the later report readable.

Keep t relative to the session, not wall clock. You are measuring intervals, not surveillance timestamps.

Step 2: Wire the card, gently

Here is the sketch I use. It is a sketch, not production code, and I have not run this exact block in your app.

// sketch — point the sink at your own collector
function logEvent(reviewId, event, extra = {}) {
  const payload = JSON.stringify({
    session: SESSION_ID,
    review_id: reviewId,
    event,
    t: Math.round(performance.now()),
    ...extra,
  });
  navigator.sendBeacon("/events", payload);
}

card.addEventListener("open", () => logEvent(REVIEW_ID, "card_open"));

for (const panel of document.querySelectorAll("[data-evidence-panel]")) {
  panel.addEventListener("toggle", () => {
    if (panel.open) {
      logEvent(REVIEW_ID, "evidence_expand", { panel: panel.dataset.evidencePanel });
    }
  });
}

approveButton.addEventListener("click", () =>
  logEvent(REVIEW_ID, "decision", { verdict: "approve" })
);

Use real <details> elements, not clickable divs. Keyboard users get expansion for free. Screen reader users hear the panel state.

And do not put a countdown on the card. A timer turns measurement into pacing. You will get longer dwell times and worse decisions.

Step 3: Get pilot material without writing twelve cards by hand

You need reviews with varied risk. Hand-writing them takes an afternoon you do not have.

Here is where I reached for MonkeyCode. The operator offers free model access, so drafting card text costs me nothing but review time. I wrote the ground truth myself, because model output is material, not truth.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two caveats, stated plainly. First, the operator describes a free token allowance — reported to me as ten million tokens — and I have not personally verified the current figure or its conditions. Second, free access is an availability claim, not a permanence promise. Check the terms before you build a workflow around them.

My split was simple. The model drafted the visible card text: the action summary, the stated reason, the risk label. I wrote the hidden answer key by hand.

That answer key matters. Without it you can measure speed but never correctness.

Step 4: Run the session somewhere two people can reach

Twelve reviews. Two reviewers. One shared build.

I did not want to host anything, so I used the free server option to run a single small Python process that accepts /events and appends to a JSONL file. One process, one port, one file. If your environment gives you different start mechanics, keep the shape and change the command.

python3 collector.py --port 8080 --out pilot-01.jsonl

Tell both reviewers what you log, before they start. Expansion events and decision times. Aggregate reporting only. No individual scores, ever.

Consent is not paperwork here. It is the difference between a study and a sting.

Step 5: Audit the log with the standard library

Save this as review_audit.py. No dependencies.

#!/usr/bin/env python3
import json
import sys
from collections import defaultdict

READ_MS = 3000  # your threshold, not a universal constant

def load(path):
    events = []
    with open(path) as f:
        for line in f:
            line = line.strip()
            if line:
                events.append(json.loads(line))
    return events

def per_review(events):
    groups = defaultdict(list)
    for e in events:
        groups[e["review_id"]].append(e)
    for rid in groups:
        groups[rid].sort(key=lambda e: e["t"])
    return groups

def audit(groups):
    rows = []
    for rid, evs in groups.items():
        opened = next((e["t"] for e in evs if e["event"] == "card_open"), None)
        decided = next((e for e in evs if e["event"] == "decision"), None)
        if opened is None or decided is None:
            rows.append({"review_id": rid, "status": "incomplete"})
            continue
        expanded = [e for e in evs if e["event"] == "evidence_expand" and e["t"] <= decided["t"]]
        dwell = decided["t"] - opened
        rows.append({
            "review_id": rid,
            "status": "ok",
            "dwell_ms": dwell,
            "panels_opened": len({e["panel"] for e in expanded}),
            "verdict": decided["verdict"],
            "flag": dwell < READ_MS and not expanded,
        })
    return rows

def main(path):
    rows = audit(per_review(load(path)))
    done = [r for r in rows if r["status"] == "ok"]
    if not done:
        print("no complete reviews found")
        return
    dwells = sorted(r["dwell_ms"] for r in done)
    mid = dwells[len(dwells) // 2]
    flagged = [r for r in done if r["flag"]]
    opened = sum(1 for r in done if r["panels_opened"])
    print(f"reviews: {len(rows)}  decisions: {len(done)}  incomplete: {len(rows) - len(done)}")
    print(f"median dwell ms: {mid}")
    print(f"reviews with at least one evidence panel opened: {opened}/{len(done)}")
    print(f"flagged as possible rubber-stamp: {len(flagged)}/{len(done)}")
    for r in flagged:
        print(f"  {r['review_id']}  dwell={r['dwell_ms']}ms  panels=0  verdict={r['verdict']}")

if __name__ == "__main__":
    main(sys.argv[1])

Note what the flag means. Short dwell and no expansion. One condition alone is a rumour.

Step 6: Verify the script against a fixture

The numbers below come from a fixture I wrote by hand. They are not from a real session, and you should not read them as a result.

{"session":"fixture","review_id":"r-01","event":"card_open","t":0}
{"session":"fixture","review_id":"r-01","event":"evidence_expand","panel":"source_ledger","t":1840}
{"session":"fixture","review_id":"r-01","event":"decision","verdict":"approve","t":2310}
{"session":"fixture","review_id":"r-02","event":"card_open","t":0}
{"session":"fixture","review_id":"r-02","event":"decision","verdict":"approve","t":900}
{"session":"fixture","review_id":"r-03","event":"card_open","t":0}
{"session":"fixture","review_id":"r-03","event":"evidence_expand","panel":"missing_fields","t":400}
{"session":"fixture","review_id":"r-03","event":"decision","verdict":"edit","t":6200}

python3 review_audit.py fixtures/pilot-01.jsonl

reviews: 3  decisions: 3  incomplete: 0
median dwell ms: 2310
reviews with at least one evidence panel opened: 2/3
flagged as possible rubber-stamp: 1/3
  r-02  dwell=900ms  panels=0  verdict=approve

Now break it on purpose. Add a review that opens the card, expands nothing, and decides at 9000ms. The script must not flag it, because the dwell is long.

Then delete one decision line and run again. The incomplete count must rise.

Two checks. If either fails, your log format drifted and every later number is fiction.

Step 7: Agree on stop conditions before the real session

Write these down with the decision owner, not after the data arrives.

The last row is the one teams skip. Show people a speed metric and they will beat it.

What this instrument cannot tell you

Twelve reviews is a smoke test, not a study. You cannot compute significance on it.

Dwell time is a proxy with a known failure: reading is invisible. A reviewer can read carefully in 2,400ms. A reviewer can stare at a card for nine seconds and think about lunch.

The fixture proves the script runs. It says nothing about your reviewers.

And accessibility is a real constraint here. Panels must be keyboard reachable, labelled, and untimed. If your instrumentation makes the card slower for assistive tech users, you are measuring your own bug.

Who should not use this

Do not run this with fewer than five reviewers. You will be able to identify individuals from the log, and that turns a study into a performance review.

Skip it if you cannot act on the result. A number nobody changes anything for is just anxiety with a dashboard.

If your context is regulated, get the logging reviewed before you collect a single event.

Where I would start

Twelve reviews. Two people. One afternoon.

If you want to try this without paying for model calls, the free model access is enough to draft the card text. Draft twelve, then write the answer key yourself.

Start with twelve, not twelve hundred. The first flagged review will teach you more than a quarter of aggregate data.

What would you need to see before you let that approval through?