Gemini 作為第二個模型，實際能抓出甚麼？七篇文章、四十條審閲意見的量化分析

Author: Guangyi Zhao

1. Problem and background

This article series runs under a standing rule: every draft gets reviewed by a different model before it can ship. The reviewer is told to hunt for factual errors, logical leaps, and overclaims; every comment must be resolved, one by one, before the piece is final. After seven days and seven articles, a fair question is whether the ritual earns its keep. What does the second model catch that a careful author re-read would not? And how does it fail when it fails?

No new apparatus is needed to answer that. The review records themselves are the dataset: every comment, every disposition, and every failed API call is on disk in the project's review/ directory. This piece does two things with that record. First, it codes all forty comments from the seven articles and counts what kinds of problems a cross-model review actually produces. Second, it runs a planted-error probe: a short passage with six defects buried in it, sent to the reviewer with the key withheld, to measure recall against a known denominator — something the historical record alone can never give you.

One clarification up front, because the title invites the wrong reading: of the forty comments, 29 came from the cross-model reviewer and 11 from the human three-role fallback used on the two nights the model never answered. The two sets are counted separately throughout. The headline result: the model reviewed five of seven articles and produced 29 comments, 27 of which changed the text. The other two were a false alarm of its own and a style note I kept anyway. What it catches is concentrated — not typos, but numbers that don't reconcile, causal claims that outrun their evidence, and conclusions stated without boundaries. The catch is availability: getting those five reviews took 13 API calls, of which 6 died with HTTP 503 and 2 with 429 quota exhaustion. It is a good reviewer who calls in sick a lot.

2. Method

Dataset A is the complete review log of the seven articles published October 13–18, 2026: seven review records, plus the raw reviewer receipts where they exist (JSON receipts for three successful reviews — Sentinel, Vuln-Scanner, Bloom; the Wiper receipt file was lost to a capture bug and its comments and usage figures survive in its record; the REA review survives as record text; two failure receipts document the nights the model never responded). The reviewing configuration was Gemini throughout — gemini-flash-latest as primary, with one article falling back to gemini-3.8-flash after two 503s. I coded each comment on two axes: category (numbers/definitions, causality/concepts, overclaim/generalization, scope qualification, method disclosure, structure/cross-references, style) and disposition (changed the text, verified as correct on re-check, rejected as reviewer error, already covered elsewhere, kept by author). Counting was done by a script, not by hand; the script and its JSON output ship with this article's data files.

Dataset B is the probe: a roughly 330-word excerpt about adding a cache to a diagnostics gateway, with six planted defects — a breakdown whose parts sum to 45 ms against a stated 41 ms total, a causal attribution asserted without control conditions, a boundless "no side effects, at any scale" conclusion, a stale cross-reference, a conclusion "supported" by an unverifiable internal newsletter metric that measures something else, and a success rate of 100% in the same breath as two timed-out failures. Only the passage went to the reviewer; the answer key did not. One design flaw disclosed in advance: the cross-reference defect carries an editorial note in the text admitting the numbering is stale, which half-cues that item. Discount it accordingly.

Also disclosed: the historical coding counts only the reviewer's comments. The single largest catch of the week — a draft that claimed "we tried this, it doesn't work" about an experiment that had never been run — was caught by the author's own pre-submission check, not by the reviewer. It is discussed in Section 4 but not credited to the reviewer.

3. Measurements

3.1 Availability: 13 calls, 5 reviews

The call log, article by article: REA failed once with 503, then succeeded on retry. Web-Vuln-Scanner failed twice with 503 and succeeded only after switching to the fallback model. AI-Node-Sentinel succeeded on the first call but a day late — drafted on the 14th, reviewed and published on the 15th. Wiper and Bloom-Counter succeeded first try. The remaining two never got through: Go-Diagnoser hit a 429 quota wall, and Emotion-Guard burned four calls in one evening (three 503s, one 429) before I gave up and ran the human fallback. Totals: 5 of 13 calls succeeded (38%); of the 8 failures, 6 were 503 and 2 were 429. Five of seven articles received a cross-model review; only four received it the same day. This is what a free-tier quota buys you: a 429 that says "come back in about 22 hours" and a 503 that means the provider is busy are both failures that retrying for a few minutes will not fix.

3.2 The probe: six planted, six caught, one unplanted

All six planted defects were identified explicitly. The arithmetic error came back with the sum worked out. The success-rate contradiction was quoted back in adjacent fragments. The causal claim was faulted for ignoring second-order effects — downstream relief, network and GC behavior — and for asserting complete attribution without controls. The overclaim was quoted word by word. The newsletter evidence was flagged, though filed under causality rather than sourcing: ticket volume moves with business cycles and rosters, so it cannot certify "stability." The cross-reference was caught too, subject to the half-cue disclosed in Section 2.

More interesting than the perfect score is the seventh finding, which I did not plant. The probe states an 87% cache hit rate and also books the full 19 ms origin fetch into the average per-request breakdown. The reviewer objected: if only 13% of requests reach the origin, that line should enter the average weighted (≈2.5 ms), not at full value — the metric definition contradicts the hit rate. On re-checking, the reviewer is right; the breakdown described a full request path while claiming to be an average. Planting defects, I had accidentally planted a real one. Final probe score: 6 of 6 on the key, plus 1 valid finding outside it.

3.3 What the 29 comments look like

By category: causality/concepts 6, overclaim/generalization 6, numbers/definitions 5, method disclosure 5, structure/cross-references 3, scope qualification 3, style 1. The top three families are 17 of 29 (59%).

Representative catches. Numbers: in the Bloom-Counter piece I had summarized an overexposure condition as essentially unaffected; the reviewer pointed out that its 15 false positives (precision 0.933) were bleached leaf fragments masquerading as buds — brightening manufactures targets while darkening loses them, two different failure modes my summary had flattened into one. In the Web-Vuln-Scanner piece, a target's 15 findings were itemized as 14 in my text; the reviewer counted. Causality: in Bloom again, my explanation of an over-count contradicted my own limitations section — adhesion blamed in one place, a "conservative estimator" claimed in another; the reviewer laid the two passages side by side. Overclaim: "not a bottleneck at any realistic scale" in the Sentinel piece was flagged as evidence-free extrapolation and cut down to the measured scale. The most instructive comment was method-related, in the Wiper piece: the reviewer said the rounding convention behind the quoted prices was undocumented, so a reader recomputing from the printed numbers would not balance. I re-ran all 8 records under both pricing branches — 16 combinations — and every total balanced to the cent. That comment's product was not a corrected number but a paragraph documenting the convention and the reconciliation.

Dispositions: 27 of 29 changed the article (including the Wiper comment, resolved by documenting the re-check). One was rejected: the reviewer called the Sentinel draft's date, 2026-10-14, a "future date typo." The draft was written on the 14th and reviewed on the 15th; the review prompt carried no current date, and the model's knowledge ends before 2026, so a true date read as future. Section 4 returns to this. One was kept: a style objection to a turn of phrase in the REA piece, noted for the record; I preserved the first-person register.

3.4 The fallback group: what human three-role review produces

The two fallback articles generated 11 comments: numbers 5, method 2, scope 2, overclaim 1, style 1. Their disposition profile is the mirror image of the model's: only 5 changed the text, while 5 were "verified correct on re-check" and 1 was already covered by the limitations section. The Emotion-Guard reviewer (also me, wearing a different hat) recomputed every probability in the draft — the teacher-spike figure, 1 − 0.95⁶ ≈ 26.5%, checks against the text's "more than a quarter" — and reconciled every number against the raw data file. That is audit confirmation, not fault-finding, and the reason is structural: in the fallback, reviewer and author share one brain. It is good at re-deriving its own arithmetic and poor at stepping outside its own framing. The Go-Diagnoser fallback did catch an unsupported attribution and one "tests pass" claim that had never actually been run — both are evidence-state problems, neither required an outside frame.

3.5 Cost

The direct cost is small. The four successful reviews with complete usage records: Sentinel 5,095 prompt tokens, Vuln-Scanner 5,365, Bloom 5,149, Wiper 24,372 (that receipt folds the model's internal thinking tokens into its totals, so it is not strictly comparable to the other three), with responses of 1,820, 1,764, 1,974, and 1,605 tokens — all under two thousand. The REA review's usage was not preserved, the one gap in the ledger. The probe cost 443 prompt and 1,009 response tokens. The real cost of the pipeline is not tokens but latency and uncertainty: retries against 503s, a model swap, a review that landed a day late, and two evenings of human fallback labor.

4. Analysis

Four judgments survive the data.

First, the reviewer's highest-value skill is reconciliation, not judgment. Roughly ten of the 29 comments turn on numbers, definitions, or count completeness, and nearly all were right. Reconciliation needs no theory of the whole article — it needs the stamina to collect every instance of the same quantity scattered across sections and compare them. That is exactly where a single author fatigues: the same figure appears in an abstract, a body paragraph, and a table, gets edited in one place, and drifts in the other two.

Second, it is more sensitive to unbounded language than its author is. Overclaim plus scope comments form the largest bloc at 9. The reviewer has no sunk cost in a pretty sentence. The probe reproduced the pattern in miniature: "no side effects" was quoted back and condemned in the same reflexive way "at any realistic scale" had been a few days earlier.

Third, its false alarms are context-shaped, not confabulation-shaped. The week's single rejection traces to a missing date in the prompt plus a knowledge cutoff that predates the draft's year. The defenses are procedural, not rhetorical: resolve every comment individually rather than accepting them wholesale, record rejections, and hand the reviewer the context it cannot guess. For this article's own review I injected the current date into the prompt; the failure mode did not recur — one observation, not a cure.

Fourth, the fallback path is part of the main path. Two of seven articles ended up there, so "what happens when the model doesn't answer" is not an edge case; it is a weekly event. The fallback produces a different substance (verification over reframing), and the ledger says so in plain language, so readers can discount the review strength of those two articles correctly. The one unforgivable version of this pipeline is a vague methods line that lets a fallback review masquerade as a cross-model one.

And the question from Section 1 — can author self-check substitute? Within this workflow — one author, unaided re-reads, no checklist scripts — the evidence says the two catch different species of error. The week's worst defect (a never-run experiment reported as run) was invisible to the reviewer: on the page it read perfectly. The reconciliation errors the reviewer caught had survived two of my own read-throughs. Bolting extraction scripts and checklists onto self-check would move that boundary; this dataset doesn't measure that configuration.

5. Limitations and open problems

First, the sample is seven articles, one reviewer family (one primary model, one fallback variant), one author, and one coder — me — with no second coder to check the category boundaries; the script guarantees arithmetic, not coding. Second, the probe is a single short passage with a defect density no real draft has, and one half-cued item; 6-of-6 says these defect types don't slip past it in a short text, and says little about recall in a 5,000-word article where defects hide deeper. Third, the fallback and model groups reviewed different drafts, so the contrast in Section 3.4 mixes reviewer effects with manuscript effects; it is not a controlled comparison. Fourth, the availability figures are bound to a free-tier quota and one particular week of provider load; 5-of-13 will move with a paid tier or a different week, and every conclusion here is scoped to this configuration. Fifth, nothing here measures the endpoint that ultimately matters — whether reviewed articles earn more readers or more trust. Comments changed texts; the rest of that chain has no data in this study.

6. References

• Review records for the seven articles: review/2026-10-13-rea-gemini-review.md, review/2026-10-14-sentinel-review.md, review/2026-10-15-vuln-scanner-review.md, review/2026-10-16-wiper-review.md, review/2026-10-17-bloom-review.md, review/2026-10-18-emotion-guard-review.md; raw Gemini receipts in the same directory (*-gemini-raw.json).
• This round's data: data/2026-10-19-cross-review-bench.json (script: data/2026-10-19-cross-review-bench.py, re-runnable); probe text, answer key, and reviewer output: data/2026-10-19-probe.txt, review/2026-10-19-probe-gemini-raw.json.
• The pipeline's standing rule: the project's writing standard, quality-gate section (cross-model review and honest fallback labeling).

Author: Guangyi Zhao. The pipeline, records, and experiments described here are the author's own clean-room work; the experimental material consists of the series' own manuscripts and a synthetic probe passage. No client information is involved.