如何把對 AI 輸出的直覺轉化為評分量表
How I Turn Gut Feelings About AI Output Into a Scoring Rubric
自由 AI 評估員 Tony Sterlin Cantave 分享一套把對 AI 輸出的主觀直覺轉成可重複評分標準的方法。流程包括把觀察拆成獨立維度、為 1 至 5 分各級設定具體錨點、先寫評分理由再給分,並用 Python 腳本比較兩次評分的一致性。若某個維度反覆出現兩分差距,他會檢查並改寫該維度的錨點。
Part of my work as a freelance AI evaluator is scoring model output: images, video, audio and user interface designs, each judged against a rubric. The hardest lesson I learned early is that my first reaction to an output is almost never useful on its own. "This looks off" or "this is pretty good" feels like a judgment, but nobody can train a model on it, and I cannot even repeat it reliably the next day.
What works is turning that gut feeling into criteria someone else could apply and get the same answer. This post is the process I use when I write or interpret a rubric, plus a short script I use to check my own consistency.
Step 1: Name what the reaction is actually about
When an output bothers me, I stop and write down every specific thing I noticed before I score anything. For a generated interface screen that might be:
- the button text is cut off
- two labels say the same thing
- the color contrast on the secondary text is weak
- the layout does not match the prompt, which asked for a sidebar
Each line is a candidate criterion. Usually the list falls into a few groups: did it follow the instructions, is it technically correct, and is it usable or pleasant. Those groups become rubric dimensions.
Step 2: Separate dimensions that move independently
A common mistake is one score that mixes everything. An image can follow the prompt perfectly and still have obvious artifacts. Another can be beautiful and ignore half the request. If both get a 3 out of 5, the score hides what went wrong.
I keep dimensions separate when they can fail separately. For most visual tasks I end up with something like:
- Instruction following: does it include what was asked, and nothing that contradicts it?
- Technical quality: artifacts, distortions, broken text, audio glitches, layout bugs.
- Usability or clarity: could a real person read it, use it, or understand it quickly?
Step 3: Anchor every score level
A scale of 1 to 5 means nothing until each number has a description. The anchors are where consistency comes from. Here is an example for instruction following:
- 5: every requested element is present and correct
- 4: all major elements present, one minor detail missed
- 3: one major element missing or wrong
- 2: several major elements missing
- 1: the output does not address the request
The test I use for an anchor: if two careful people read it and look at the same output, would they pick the same number? If the honest answer is "maybe", the anchor needs a concrete example.
Step 4: Write the reason before the number
I write one or two sentences of justification first and pick the score second. When I do it the other way around, I catch myself writing reasons that defend a number I already chose. Writing the reason first also makes disagreements easy to resolve, because the reviewer can see exactly which observation drove the score.
A good justification names the evidence: "The prompt asked for three pricing cards; only two appear, so instruction following is a 3." A weak one restates the score: "Mostly follows the prompt."
Step 5: Check your own consistency with a small script
Rubrics drift. After a long session, my 4 can quietly become someone else's 3. One way I catch this is to rescore a few items I already scored earlier, without looking at my first scores, and compare. A tiny Python script makes the comparison quick:
import csv
from collections import defaultdict
def load_scores(path):
scores = {}
with open(path, newline="", encoding="utf8") as f:
for row in csv.DictReader(f):
key = (row["item_id"], row["dimension"])
scores[key] = int(row["score"])
return scores
first = load_scores("first_pass.csv")
second = load_scores("second_pass.csv")
gaps = defaultdict(list)
for key, score in second.items():
if key in first:
gap = len(range(min(score, first[key]), max(score, first[key])))
gaps[key[1]].append(gap)
for dimension, values in sorted(gaps.items()):
exact = sum(1 for v in values if v == 0)
print(f"{dimension:22} items {len(values):3} exact match {exact:3} max gap {max(values)}")
The output shows, for each dimension, how often my second pass matched my first exactly and the biggest gap. If one dimension keeps showing gaps of two points, the problem is usually the anchors for that dimension, not my attention. That is a signal to rewrite the anchor descriptions with clearer examples.
What this taught me beyond AI work
Before software, I spent years in delivery and logistics operations and in manufacturing, where quality checks only worked when the standard was written down and specific. A vague instruction like "make sure the parts look good" never holds up across shifts. A rule that names exactly what to reject, and shows an example, does. Scoring AI output is the same problem in a new setting: the value is not in having taste, it is in writing your taste down so clearly that someone else can apply it.
If you evaluate model output, test prompts, or review any kind of generated content, try this on your next batch. List what you noticed, split it into independent dimensions, anchor every level, write the reason first, and rescore a sample later. Your scores will become something a team, or a model, can actually learn from.
Originally published at tonycantave.com by Tony Sterlin Cantave, a Connecticut software developer.
來源:DEV Community · dev.to