Perch 的 70 段錄音藥名盲測

Perch ran a blind 70 clip test built to catch invented medications, against a commercial veterinary AI scribe and against itself, with every miss on both sides published in the open.

Drafted with AI assistance from our test records. Every result below is in the public ledger linked at the end.

The failure mode

An AI scribe that turns a spoken exam into a written chart usually runs two steps in sequence. First a speech recognition layer turns audio into text. Second a language model layer reads that text and writes the structured note: the SOAP, the DAP, the medication list.

The first step is weak on its own. Drug names are unusual words, said quickly, often as a brand name next to its own generic in the same breath. Raw speech recognition misses a meaningful share of them.

The second step is where the real work happens, and where the real risk starts. A language model reading a rough transcript can recover a drug name the recognizer garbled, by matching it against context, such as the dose and the species. That is genuinely useful work. It is also the same mechanism that can produce a drug name nobody said. A model able to infer the right drug from a mangled word can just as easily infer a wrong one that merely sounds plausible, because the same inference step produces both outcomes.

We built a test set to find out whether this happens: on our own scribe, Perch, and on a commercial veterinary AI scribe already in clinical use. We are publishing what we found on both sides, at the same weight.

How the test set was built

Perch built the set as 70 short clips: 45 ordinary dictations that mirror how a vet talks through a visit, and 25 traps built to bait a scribe into charting a drug that was never said. A trap might be an offhand phone note that happens to contain a word that sounds like a brand name, with no clinical content in it at all.

The clips were written by someone who never saw the matching code on either side, so nothing in the corpus was tuned to pass. Each clip was rendered once, with a synthetic voice, so no clip has more than one reading.

We ran both products against the same 70 clips. We ran the commercial scribe the way any customer would run it, and scored Perch on the same clips through its own production pipeline.

Scoring rules

Both products are scored the same way. A drug actually said in a dictation counts as recalled if it appears correctly in the output text, or is charted in the note. A false offer is a drug name that ends up charted or offered in the note when the clinician never said it. There is no partial credit and no benefit of the doubt for either side.

The 45 dictation clips carried 84 drug mentions between them. That is the base every clip level result in this piece is measured against.

What happened on the run

Every example below comes from a clip read by a synthetic voice, not a real clinician, in a test we ran and scored ourselves. Real speech runs harder than a synthetic voice does. This is a reproducible shared test set, not an accuracy claim about either product.

One dictation read: "Started on Baytril, enrofloxacin, 5 mg per kg IV once," a brand name followed by its own generic, said together the way vets often say it to keep a record unambiguous. The commercial scribe's transcript rendered this as "batril and roflexacin." Its note generation step then charted both as separate medications: Baytril, and Roflexacin. Roflexacin does not exist. The chart, as generated, doubled the antibiotic dose under a name that is not a real drug. Perch, scored on the same clip, charted enrofloxacin and missed the Baytril brand, in both of its configurations: a miss, but no second drug.

A trap clip carried a different shape of risk. The dictation was an offhand phone note about "the next guard shift schedule change," nothing clinical. The same product's transcript wrote "the NexGard shift schedule change," and the brand name for a flea and tick product carried through into the note's narrative text, where no medication had been discussed at all.

Perch's own record is not clean either, and it belongs at the same weight as the two failures above. On a separate trap clip from Perch's own testing, a cat is described obliquely as being on Bexacat. An earlier build of Perch charted Zycortal, a real but wrong drug, from the garbled audio. The build running now declines to chart anything on that same clip. And one configuration we evaluated while building the current matcher, and never shipped, charted ProHeart from a misheard "pretreated"; this test set caught it before it reached a clinic. We name our own build and the fix here, and leave the comparison at that.

Credit belongs on the other side too. One dictation contained a badly garbled reading of atenolol, rendered by the recognition step as something close to "8 nilal." The commercial scribe's repair layer recovered the correct drug name from that garble. One of Perch's two configurations missed it on the same clip. That repair layer did real work across the run, recovering several other names a raw transcript alone would have lost.

The same repair layer, while recovering names, also quietly dropped the brand half of more than one dictated brand and generic pair: Apoquel twice, plus Clavamox, Rimadyl and Cerenia each lost their brand name in the finished note, leaving only the generic behind. Nobody flagged the loss, since the note read as complete either way. Perch, scored on those same five pairs, charted both halves of each. Its one dropped brand in the run was Baytril, on the clip above.

A few smaller errors from the same run are worth naming plainly. The commercial scribe's chart spelled a dictated Cosequin as "Kosquin," close enough to pass a quick read but wrong on the page. Entyce, an appetite stimulant, defeated every system in the test, Perch included, and the commercial scribe's note put the ordinary word "Entice" next to its medication list. On two other clips its note stated a diagnosis the clinician never dictated, and a surgery date nobody said, detail added on top of what was actually spoken.

Across the 25 trap clips, Perch charted nothing that was not said. The commercial scribe charted no drug from the ordinary word bait either. Its two trap misses were NexGard in the narrative and an unstated generic name, phenylpropanolamine, added next to a correctly charted Proin.

Why Perch refuses instead of guessing

Perch's rule for an unresolved drug name is to leave it out of the note rather than fill in a best guess. If the audio does not clearly match a known term, that line stays blank for the vet to fill in on review, rather than being written with a name that only sounds right.

A blank line gets checked before a note is signed. A wrong but plausible drug name can read as complete and get missed entirely. That asymmetry is the reason for the design. Across both of Perch's configurations in this run, the count of invented drug names charted was zero. The count of real drug names it declined to guess, and left blank instead, is a cost of the same design, accepted on purpose.

Limits

Every clip in this set is synthetic speech, generated by text to speech voices, one voice per clip. This is our own test, run and scored by the team that built Perch, not an outside body. We expect real clinician speech, with background noise and cross talk, to run harder for any scribe, ours included.

We ran the commercial scribe through its own documented upload flow, at its default settings, on a trial account, generating one note per case from its stock template, with no manual edits and no clinic drug list uploaded on either side. That is an out of box comparison, not a claim about what either product does once tuned for a specific clinic.

How to run this yourself

The clips, the written dictations, the scoring script, and the full ledger of what each product charted on every clip, including every miss on both sides, are published at https://github.com/perchvet/perch-trapset. The rule is the one described above: a drug counts as recalled if it appears or is charted, a false offer is anything charted that nobody said. Any scribe, including a future build of Perch, can be scored against the same 70 clips the same way.

Mohammed Usama is a cofounder of Talcoe LLC, which makes Perch, a veterinary AI scribe. Questions about the test set: mo@talcoe.com.