IT 工單路由實作札記：以 0.60 棄答閾值平衡錯誤與人工成本

The number that decided this project was 0.60, and it isn't in any model card.

I spent six weeks on an IT ticket router — roughly 4,000 tickets a day, eleven queues, all of it triaged by hand. The demo took an afternoon. The part that actually shipped took the other five weeks and nine days, and almost none of it was modelling.

Here's the field note.

The demo that fooled me

Prompted LLM, ticket body in, queue name out, few-shot with eight examples per queue. On a 2,000-ticket holdout it hit 94%. The service desk lead smiled. My manager smiled. I smiled, which in retrospect was the least justified of the three.

Then I asked the only question that matters: what happens on the other 6%?

Silence. Because the answer was "they get routed wrong, silently, and someone finds out two days later when the SLA is already red."

A 94% classifier with no abstention isn't a 94% system. It's a 100% system that's wrong 6% of the time. Those are very different products.

The pricing conversation

I sat down with the service desk lead with a blank spreadsheet and one question: what does a wrong route cost you, relative to a clean one?

She thought about it and said five times. Rework, the SLA clock restarting, the context switch for whoever picks it up second, and the occasional ticket that bounces through three queues before someone just calls the user.

I wrote 5x in cell B2. That cell is a guess. It's a guess from the person who actually eats the cost, which makes it a good guess, but it's still a guess — and it is now the single most load-bearing number in the entire system. More than the accuracy. More than the model. If she'd said 2x, the design changes. If she'd said 10x, it changes the other way.

That conversation is the engineering work. Everything after it is arithmetic.

The confidence problem

So I went to build abstention on top of the LLM. And it didn't work, for a reason I should have seen coming.

Ask a prompted model how confident it is and you get a sentence. "Confidence: high." On tickets it got right, it said high. On tickets it got wrong, it also said high. I pulled token logprobs over the label tokens and they were uncalibrated in the way that matters — a 0.9 meant "this is the most likely token," not "this is right 90% of the time." I tried a verbalized 1-10 scale, then a self-consistency sample of five, then a second model as a judge. Each one cost more per ticket and each one gave me a score I couldn't put a threshold on without lying to myself.

You can calibrate an LLM's outputs. But by the time you've built the calibration layer, the held-out set, and the threshold sweep, you have built a router. The LLM is now a component inside it, not the thing itself.

What I actually shipped

TF-IDF, word 1-2 grams plus character 3-5 grams, into a logistic regression. Isotonic calibration fit on a held-out slice, checked against a reliability diagram because I don't trust myself. Inference is about 8ms on one core. No GPU. The whole model is a few megabytes and it fits in the same container as the API.

Then the sweep. For each threshold from 0.30 to 0.95, compute total daily cost: inference, plus misroutes at 5x, plus the human cost of everything that abstains. Plot it. It's a U-curve, and the bottom of the U is the only number I cared about.

At 0.60, on my traffic:

• 78% auto-routed, 99.1% precision on that slice
• 22% abstained to a human
• ~31 misroutes a day instead of ~240
• total cost roughly a third of the always-route LLM version

The LLM version was more accurate overall. It lost on cost anyway, because accuracy is the wrong objective when errors are five times more expensive than abstentions. That's the whole trick and it took me an embarrassingly long time to internalise it.

The abstain path is not "give up"

The first version of abstention just dumped the ticket into a human queue. That's not abstention, that's a shrug with extra steps, and the humans noticed immediately.

What fixed it: the abstained ticket arrives with the top three candidate queues and a one-line summary of what the ticket is actually asking for. Triage time on those dropped from about 90 seconds to about 40. That single change moved the cost curve more than any modelling I did, because it made abstention cheap. Cheap abstention is what lets you set the threshold low and be honest about uncertainty.

The LLM still earns its keep — it writes those summaries and ranks the candidates on the 22% that abstains. That's about 880 calls a day. It's a good job for it. It's just not the router.

Things I'd tell someone starting this

Write the price down, in the config, with a name next to it. My threshold lives in a YAML file with a comment: # 0.60 — agreed with service desk, misroute = 5x triage, reviewed quarterly. When someone asks why we abstain on a fifth of the traffic, I don't have to reconstruct an argument from memory. And when the price changes, the threshold is a one-line diff, not a retrain.

Calibrate, then verify you calibrated. Logistic regression on TF-IDF is roughly calibrated out of the box, which is most of why I picked it. Boosted trees are not. LLM logprobs are definitely not. Isotonic regression on a held-out slice, then look at the reliability diagram with your own eyes.

Watch the abstain rate like a canary. Mine sits at 22%. If it drifts to 35% next month, something upstream changed — a new product line, a ticket template rewrite, a queue that got renamed. That drift signal is free and it caught a broken intake form before anyone filed a bug.

Don't let the threshold be a vibe. I've seen "0.7 feels right" in more codebases than I'd like. It's not a hyperparameter. It's a business decision wearing a float costume.

I don't think this generalises to everything. If your errors are cheap and symmetric, route everything and skip the abstention machinery entirely — you'll save yourself a month. But if a wrong answer costs meaningfully more than a handoff, the threshold is the product, and the model is a detail you get to pick afterwards.

Mine is 0.60. It'll probably move next quarter. That's fine. That's the job.