Leadolovets 的 LLM 篩選器空回應源於訂閲檢查短路

We build Leadolovets, a service that reads open Telegram chats and pulls out buying intent: "who does stretch ceilings", "looking for an outsourced accountant", "need post-renovation cleaning". Keyword search finds thousands of those messages. Almost none of them are real requests.

Across the whole dataset we have 542,795 keyword matches. Real requests among them: 14,774. So 97.3% of keyword matches are junk: someone else's ad, a retelling of a past experience, a price discussion, a complaint about a contractor. Roughly 36 useless messages per one real request.

That is why there is an LLM filter between keyword search and the customer. It reads every message and decides whether it is demand. Which raises the question almost nobody in this niche answers with numbers: how many real requests does the filter throw away along with the junk?

The bug that was not a bug

Three days before, we had a confident conclusion in our notes: the model fails under parallel load, 118 empty responses out of 225 calls. The logs showed no errors at all, just empty bodies.

There was no bug.

Our llm.chat_json calls a billing guard on its first line. For an account with an expired subscription that guard returns early and chat_json hands back {} immediately, without ever calling the model. That is a deliberate product rule: an expired account does not get AI. Downstream, the batch filter saw an empty response, logged "response without results: {}" and marked every message as not passing.

Our test harness picked accounts by whether they had historical requests, ignoring whether their subscription was still alive. Most of them had expired months ago.

The tell is in the timing. The refusal arrived in 0.0 seconds. A real model call takes 4 to 9 seconds. If your "model failure" is instant, it is not the model, it is your own code returning early.

After fixing the harness we measured the real thing: 12 parallel calls in a row, 12 successes, 8 to 9 seconds each; real system prompts up to 15,896 characters, answered; batches of 1, 3, 6, 9 and 12 messages, answered. No failures.

The second way we poisoned our own measurement

With the load question closed we went back to the actual question: what does the filter cost us in lost requests?

The first sample included our own accounts, both staff and admin. In those, we and our team click "contact" indiscriminately just to see how the thing behaves.

So the label "good request" ended up attached to a neighbour complaining about renovation noise, a request to share a unit economics spreadsheet, and a UGC blogger's ad. The model rejected all of them correctly, and our harness counted every rejection as a loss. Until we dropped our own accounts from the sample, the loss figure was twice as bad as reality.

The honest numbers

Final sample: 8 paying customer accounts with live subscriptions, 100 requests, labelled by behaviour rather than by opinion. Good means a human replied to it or picked it up manually. Junk means the customer themselves clicked "not relevant".

We then read those seven losses by hand. Five were correct rejections and our labelling had simply been generous: the renovation noise complaint, the spreadsheet request, the retold story. The real loss is closer to 5%.

We did not touch the filter prompt afterwards. It is already detailed, with seller/hirer/buyer roles and a seven-step check, and the measurement noise on a sample this size is larger than the expected gain. Tuning a prompt against noise is how you ship a regression and call it an improvement.

What transfers to any product with an LLM inside

Until you know who is in your sample, you are measuring your labels, not your model. Drop your own accounts and every interaction where somebody clicked as an operator rather than as a customer. Do that before you draw any conclusion about quality.

Watch the latency distribution, not just the error rate. An instant refusal is a short-circuit somewhere in your own stack. Ours was a billing guard; yours might be a cache, a rate limiter or a validation layer. Errors get logged. Early returns usually do not.

Price your two error types separately. In lead generation they are not symmetric. Letting junk through costs the customer a minute of reading. Losing a request costs them money. A filter that cuts 95% of junk at the price of roughly 5% of real demand is worth running at 8.3 to 1, but that ratio has to stay on a dashboard instead of being tuned once and forgotten.

In short

• 97.3% of keyword matches in Telegram chats are not real requests.
• Our LLM filter cuts 95% of junk and loses 18% of good requests under strict labelling, closer to 5% in practice.
• Ratio: 8.3 junk cut per one good request lost.
• Our own accounts in the sample doubled the apparent loss rate.
• An instant empty model response is almost never the model.

The service in question: Leadolovets. Five day trial, no card.

If you run an LLM filter in production: have you measured what it loses, or only what it catches?