以模型自身機率打破重排序器平分:12 題中 11 題排名第一,每題需時 95 秒
Broke the Re-Ranker's Ties With Its Own Probabilities: 11 of 12 First, at 95 Seconds a Question
這項重排序測試使用模型對分數 token 的機率打破平分,讓正確文章在 12 題中的 11 題排名第一,並修正兩題原本因平分而排名下降的結果。gemma3:12b 每題平均需時 94.8 秒;把評分範圍擴至 0–100 並沒有消除平分。模型自身機率只能重新排列同分項目,無法修正沒有平分的錯誤評分。
Entry 20 ended with two right answers demoted by ties. gemma3:12b gave the labeled article and a rival the same score out of 10, and my tie-break fell back to embedding order, which put the rival first. It also left a cost question open, because the script didn't time anything. This entry tests two ways to break the ties and times every scoring call.
I decided the three variants before running anything:
- A: Entry 20's setup unchanged, 0 to 10 with ties kept in embedding order. If it didn't reproduce Entry 20's numbers, that would be the story instead.
- B: the same prompt, asking for 0 to 100.
- C: the same integer scores as A, but ties broken by the model's own uncertainty. Ollama can return the probability of each possible next token, so for every reply I took the probabilities the model gave the digits 0 to 9 at the score position and computed an expected score. A chunk scored 9 with most of the remaining probability on 8 ranks below one scored 9 with the rest on 10. C can only reorder chunks that already share a score.
Same twelve questions and strict labels as Entries 18 to 20, top 20 chunks per question, temperature 0, two runs each.
~/mcp-env/bin/python3 eval_rerank_v2.py
A reproduced Entry 20 exactly: the same four questions moved, to the same positions, and the summary row matched to two decimals. The collection was still 741 chunks. So the new rows are comparable to the old ones.
The same eight questions had the right article first in every column, so here are the four that moved:
| Question | Embedding only | A, 0-10 | B, 0-100 | C, 0-10 + tie-break |
|---|---|---|---|---|
| 6, switch LLM providers | 1 | 2 | 2 | 1 |
| 7, AI agent reading the cluster | 8 | 5 | 2 | 3 |
| 10, incident triage with a workflow tool | 1 | 2 | 1 | 1 |
| 12, managed or self-hosted | 7 | 1 | 1 | 1 |
| Setting | Top 1 | Top 3 | Top 5 | MRR | Chunks from right article in top 5 |
|---|---|---|---|---|---|
| embedding only | 0.83 | 0.83 | 0.83 | 0.86 | 3.17 |
| A, 0-10 | 0.75 | 0.92 | 1.00 | 0.85 | 3.58 |
| B, 0-100 | 0.83 | 1.00 | 1.00 | 0.92 | 3.75 |
| C, 0-10 + tie-break | 0.92 | 1.00 | 1.00 | 0.94 | 3.50 |
That table is with no cap. With a cap of one chunk per article, every setting's top-5 hit rate stays the same or rises to 1.00 and the chunk count drops to 1.00, as in Entries 19 and 20, so the cap still adds nothing once the re-ranker is in place.
The finer scale didn't remove the ties, because the model doesn't use it. Across 240 replies on the 0 to 100 scale, gemma3:12b used only ten values: 0, 10, 20, 30, 40, 60, 70, 75, 85 and 90. It never used 50, 80 or 100, or anything that wasn't a multiple of five. On the 0 to 10 scale it used nine values and never 5 or 10. A 101-point scale with ten values in use is no finer than the 11-point one. Question 6 shows it: three chunks scored 90, the rival first, exactly as they'd tied at 9 before. B still won back question 10 and moved question 7 to second, so the wider scale changed some judgments, but not by spreading them out. On average 1.92 chunks shared the top score under B, against 2.75 under A, and in four questions the tied chunks came from more than one article (five under A).
C used information the integer threw away. For question 6, the four chunks tied at 9 had expected scores of 8.96, 8.82, 8.73 and 7.10, and the highest belonged to the labeled article. That fixed question 6 and question 10 without changing a single score, and C ended with the right article first on 11 of 12 questions. The one miss is question 7, the agent question that has never come first in any setup in this series. The 12B scored an n8n workflow chunk 8 there, ahead of everything else, with no tie involved.
Then the cost. To time it cleanly I ran a separate single pass per model: the plain 0-10 prompt, no probabilities requested, twenty chunks per question.
| Scorer | Seconds per question (20 calls) | Range | All 12 questions |
|---|---|---|---|
llama3.2:1b |
4.4 | 4.2 to 4.6 | 52 s |
llama3.2:3b |
13.6 | 12.4 to 15.6 | 163 s |
gemma3:12b |
94.8 | 69.8 to 136.9 | 19 minutes |
The only scorer that helped in Entry 20, and the one that makes the tie-break work here, costs about a minute and a half per question on this Mac mini (Apple M4, 16 GB). The 1B is twenty times faster, and Entry 20 showed it barely changes anything. That's not a price for an interactive search box. It might be fine for a batch job or a nightly report.
Some limits on all of this. Twelve questions means one question is eight points: C's 0.92 against B's 0.83 is question 6 and nothing else. C's tie-break comes from the same model's judgment, so it can't fix a confident wrong score like question 7's, only ties. The timings come from one pass, and the variant runs were slightly faster per call because the second run of each question repeated identical prompts, which Ollama partly caches. Other light work ran on the machine at the same time, including a few container commands near the end, which may explain some of the 12B's spread. The embedding lookup itself wasn't timed; it's small next to twenty model calls. And the labels are still strict and mine.
So the ties were mostly a tie-break problem, not a scale problem. Asking for a finer scale got round numbers back. Reading the model's own uncertainty, already available in the same call, closed both of Entry 20's demotions and left one miss. The open questions are whether this holds on questions written after seeing these results, and what the agent question's label should be. That's the next entry.
來源:DEV Community · dev.to