作者建立 1,000 條合成釣魚偵測基準，測試兩款開放模型的表現
I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored
作者建立包含 1,000 條合成記錄的釣魚偵測基準 RSB-1000，並以零樣本方式測試兩款開放模型。Qwen 3.8-27B 的召回率為 98.9%、誤報率為 11.6%；GPT-OSS 20B 的召回率為 94.0%、誤報率為 23.1%。文章指出，誤報率令模型不適合作為自主攔截工具，建議先由 AI 篩查，再交由人手覆核。
I built a 1,000-record synthetic benchmark for phishing detection — here's what 2 open models scored
The problem
I run a YouTube channel where we test AI claims with real experiments. For an episode on phishing detection, I went looking for published scam/phishing-detection scores for open-weight models.
There were none. Zero. For any of them.
So we built the benchmark ourselves.
RSB-1000: what it is
- 1,000 records, fully synthetic (every brand, domain, name, and phone number is invented)
- 780 scam / 220 legitimate — the legitimate set includes deliberately tricky edge cases (urgent-but-real notifications, lookalike-but-legit senders)
- Phishing bucket: 200 records = 140 classic + 60 adversary-in-the-middle (AiTM) — the AiTM subset uses AI-polished language, lookalike domains, and MFA-reauth urgency. This is the subset that breaks naive filters.
- Each record:
id,category,subcategory,difficulty(easy/medium/hard),text,ground_truth, plustellson scam records andsafe_actionon all records - Frozen October 2026, SHA256-pinned, CC-BY-4.0
How we scored the models
Zero-shot classifier prompt, no fine-tuning, no cherry-picking: each model labeled every record as scam or legitimate, and we computed recall and false positive rate against ground truth.
Results
| Model | Recall | False positive rate |
|---|---|---|
| Qwen 3.8-27B | 98.9% | 11.6% |
| GPT-OSS 20B | 94.0% | 23.1% |
Both catch nearly everything. The false positive rate is the real story: at 23%, nearly 1 in 4 legitimate emails gets flagged — unusable as an autonomous blocker. The practical setup is AI as first-pass triage with human review, not a sole filter.
The AiTM subset was the hardest for both models, which tracks: AI-written lures kill the old "look for bad grammar" heuristic stone dead.
Reproduce it
The repo has the dataset, a stdlib-only eval script (eval.py — feed it a predictions JSONL, get recall/FPR/accuracy), and the full methodology:
- GitHub: https://github.com/RevealWire/rsb-1000
- Hugging Face: https://huggingface.co/datasets/RevealWire/rsb-1000
If you run other models against it, I'd genuinely love to see the numbers — the whole point is that nobody had published any.
Disclosure: I'm the creator of Reveal Wire, the YouTube channel this benchmark was built for. We never script the result — the evidence writes the ending.
來源：DEV Community · dev.to