我的 AI 可見度評分是 29.2：這個數字掩蓋了甚麼

The daily run of my AI citation tracker finished on 21 October with a blended score of 29.2. On its own that reads like a mediocre week. Split by question type it is something else entirely, and the split is the only part of it I would trust.

How the number is built

Every prompt goes to two web-grounded models, Gemini and OpenAI's search model. Each answer is scored on a 0 to 3 scale for how well it identifies and describes the entity I track. The blended score is points earned over points possible. On 21 October that was 28 out of 96: 16 prompts, two models, 32 answers, three points each.

What was underneath

Same run, broken out by question bucket. n is the number of answers in the bucket.

So the 29.2 is mostly a statement about the mix. The bucket where the entity does well is 6 of 32 answers. The genre recommendation bucket alone is 12 of 32, more than a third of the denominator.

Three reasons I stopped reading the blended number

Bucket weights decide the headline. If I added six more direct questions tomorrow and they scored like today's, the blended score would go from 28/96 to 44/114, which is 38.6. Nothing about how the models treat the entity would have changed. Only my question list would have.

Small buckets are coarse. With n=4 and three points per answer, a single answer is worth 25 percentage points of its bucket. A bucket that goes from 0 to 25 has moved by one answer. With n=2 one answer is 50 points. Any chart that shows these as smooth lines is drawing noise.

The model gap is smaller than it looks. Gemini scored 35.4 (17 of 48), OpenAI search 22.9 (11 of 48). Twelve and a half points sounds like a ranking. In absolute terms it is six points, two fully correct answers, on one day. I would not publish a model comparison on that.

What I report instead

Points, possible points and n for every bucket, and never a percentage without the n next to it.

I also keep two questions apart that the blended score mixes together. The direct bucket answers "does the model know who this is when asked". The genre buckets answer "does the model think of him without being asked". Those are different problems with different fixes, and an average of the two answers neither.

The honest reading of 21 October is short: asked by name, both models mostly get it right. Asked an open genre question, they almost never bring him up. That second line is the one worth tracking week to week, and it needs more prompts per bucket before a change in it means anything.

Limits

One day, 16 prompts per model, two models. Anything that differs by one or two answers is inside the noise. The tracker runs daily, so the next step is pooling a week per bucket before comparing anything.

Code (MIT): https://github.com/marintkael/marin-research-tools?utm_source=devto
Open dataset: https://huggingface.co/datasets/marintkael/ai-citation-fidelity?utm_source=devto
Research overview: https://marin-t-kael.de/en/research?utm_source=devto