AI 客服工單分流示範：以準確率目標設定門檻，讓每 100 張工單中約 76 張毋須人工處理

When a company says "let's automate support with AI", the first question is usually: "what percentage of tickets can the model handle?" That is the wrong question. The right one is: what share of wrong answers are we willing to send to customers? The automation percentage falls out of that answer, not the other way around.

Here is a small demo that makes the idea concrete. The data is public and the numbers are test numbers, not a client result.

What the demo does

It takes a customer message such as "why didn't my top-up go through?" and answers three questions: which of 77 categories it belongs to, how confident the model is, and what to do next: handle it automatically or hand it to a human.

The data is the public Banking77 set: 10,003 messages for training and 3,080 held out for testing. The model is deliberately plain: word and character TF-IDF features, logistic regression, calibrated probabilities. No large language model, no GPU.

The confidence threshold

For every ticket the model produces a confidence between 0 and 1. Above the threshold, the ticket is handled automatically. Below it, the ticket goes to a person.

You don't pick the threshold by eye. First you fix the target: "accuracy on automatically handled tickets must be at least 97%". Then, on a separate validation split, you find the lowest threshold that still meets that target. Lower threshold means more automation and more errors, so the target decides where you land.

The final check runs on the test set the model has never seen, exactly once. Otherwise the numbers look nicer than real life.

What came out (test data)

• 75.9% of tickets handled automatically.
• 97.7% accuracy on those, about 17 to 18 wrong answers per 1,000 tickets.
• If everything were automated, accuracy would be 90.2%.
• The remaining 24% went to a human.

The gap between 97.7% and 90.2% is the whole point of "human in the loop": the person takes the most doubtful cases.

Where it goes wrong

The model mostly confuses neighbouring categories: "top-up pending" vs "top-up failed", "card payment not recognised" vs "card compromised". That is normal. Such pairs are worth sorting out with the client, on the client's own labelled tickets.

What to take away

1. Start from a quality target, not from a desired automation percentage.
2. Pick the threshold on one split, verify on another.
3. Always keep a path to a human.
4. Measure on the client's own data: a public benchmark shows the method, not your result.

The code and numbers are open: ai-ticket-triage-demo. More demos and contacts are in my portfolio mini-app: @Kotenev01_bot.

I write about practical, measurable AI adoption for business: demos on test data, no invented case studies. Telegram channel: @kotenev01_en