搜尋路由不一定需要聊天模型：Jev 較 Gemini 3.5 Flash-Lite 快約 2 倍、便宜約 7 倍

We recently added record search to Sapling RCM, our medical billing app. You press ⌘K, type something, hit Enter, and get results. The query might be a clinic name, patient name, claim number, payment ID, or something else. The results can come from eight different tables.

Names are blurred.

Before hitting the database, we run the search text through a small model. Its job is to figure out what kind of text the user entered, so we can search only the tables that make sense.

I tried two models for this: Jev and Gemini 3.5 Flash-Lite. Then I measured how they compared on speed, cost, and missed results.

Here's what the router does, how I tested it, and what I found.

TL;DR

• Jev's median routing call took 466 ms. Gemini took 900 ms.
• Jev was about 7x cheaper: roughly $23 per million searches vs. $170.
• Jev missed results on 3 of 27 searches. Gemini missed on 6 of 30.

Ask what the text is, not which table to search

One way to build this is to ask the model, "Which tables should I search?"

We don't do that.

Instead, we ask the model what kind of text the user entered, and then map that to tables in our own code.

For example:

We have six kinds of text like this.

This ended up being useful for a few reasons:

• Each question is simple. "Could this be a person's name?" is easier than "Which of these eight tables should I search?"
• The model never needs to know our database schema. If we add another table, we update the mapping in our code instead of changing the prompt.
• A table can belong to multiple kinds. Claims, for example, can be found through a patient name or a claim number. The model doesn't need to know that relationship.

Router 1: Jev

Jev, from TypeSafe and served through OpenRouter, isn't a chat model.

You give it some state and a set of typed questions, and it returns a probability for each question. There's no generated text to parse and no reasoning tokens.

const response = await fetch('https://openrouter.ai/api/alpha/decisions', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'typesafe/jev-1.14',
    state: { query },
    questions: {
      claim_number: {
        type: 'noul',
        instructions:
          'Could this search text be all or part of a claim number? Claim numbers look like CLM-2026-00847.',
      },
      // one question per kind
    },
    provider: { zdr: true },
  }),
  signal: AbortSignal.timeout(2_500),
});
// { answers: { claim_number: { noul: 0.97 }, person_name: { noul: 0.02 }, ... } }

A few details in this request turned out to matter:

• Put your actual ID formats in the question. Otherwise, the model has to guess what your IDs look like.
• Pin the model version. Using something like ~latest means your routing behavior can change without you noticing.
• provider: { zdr: true } tells OpenRouter to use providers with zero data retention.

For each kind, we search its tables when the score is 0.3 or higher. If nothing reaches 0.3, we take the highest scoring kind.

The nice thing about getting probabilities is that the threshold is ours.

If we're missing results, we can lower it and search more tables. If we're getting too many irrelevant results, we can raise it. We don't need to change the prompt.

Router 2: Gemini 3.5 Flash-Lite

For Gemini, I gave it the same questions as a list in the system prompt, along with a JSON schema for the response. Temperature is set to 0.

const response = await vertex.models.generateContent({
  model: 'gemini-3.5-flash-lite',
  contents: query,
  config: {
    systemInstruction:
      'Decide which kinds of text this search could be. Return every kind that could apply.\n' +
      '- claim_number: Could this be all or part of a claim number? ...',
    temperature: 0,
    responseMimeType: 'application/json',
    responseJsonSchema: {
      type: 'object',
      properties: {
        kinds: {
          type: 'array',
          items: { type: 'string', enum: ['person_name', 'claim_number', 'payment_id' /* ... */] },
        },
      },
      required: ['kinds'],
    },
    abortSignal: AbortSignal.timeout(2_500),
  },
});
// {"kinds":["claim_number"]}

The schema makes the response easy to parse.

The downside is that Gemini gives us a list rather than scores. There's no equivalent of "use everything above 0.3," so there's no threshold to tune.

Don't let the router break search

The router is an extra network call in the search path, so eventually it's going to fail.

That's why every step has a fallback:

1. Ask Jev.
2. If Jev fails for any reason missing API key, timeout, bad status, malformed response, etc. ask Gemini.
3. If Gemini fails too, skip routing and search every table the user has access to.
4. Otherwise, search only the tables for the selected kinds, in parallel.
5. If those searches return nothing, search the remaining tables before showing "No results." The router can be wrong.

The last step is important. Routing is an optimization. It shouldn't change whether search works at all.

Trace every call, but be careful with search text

Every routing call creates an OpenTelemetry span with OpenInference attributes, so Phoenix displays it as an LLM call:

tracer.startActiveSpan(
  'search_routing',
  {
    attributes: {
      'openinference.span.kind': 'LLM',
      'llm.provider': 'openrouter',
      'llm.model_name': 'typesafe/jev-1.14',
      'input.value': JSON.stringify(body),
    },
  },
  async (span) => {
    // ...call the model...
    span.setAttributes({
      'output.value': JSON.stringify(answers),
      'llm.token_count.prompt': usage.input_tokens,
      'llm.token_count.completion': usage.output_tokens,
      'llm.cost.total': usage.cost,
    });
  },
);

These spans made it easy to compare the two routers call by call.

I also ran into two things worth mentioning.

First, Phoenix displays any cost below one cent as <$0.01. A Jev call costs only a few millionths of a dollar, so every call looked identical in the UI. If you care about the actual number, look at the raw llm.cost.total attribute.

Second, our Phoenix instance runs inside our own infrastructure, so the spans currently contain the actual search text. That's convenient for debugging, but it also means that search text can end up in your trace system.

If you're using a SaaS trace provider, think about that before sending the raw input. In a healthcare app, the search text might be a patient name.

Error logs can have the same problem.

Some ORMs, in our case, Drizzle, include query parameters in database error messages. Since our query parameter is the user's search text, that can end up in the logs too.

When a query fails, we log the database error code rather than the full error message.

How I measured the two routers

Speed and cost are straightforward. Accuracy is a little harder because we need to know what the correct answer actually was.

I kept it simple: for every test query, I also searched all eight tables. If a router skipped a table that contained a match, I counted that search as a miss.

The test queries included:

• a clinic name
• two claim numbers
• a payment ID and the bare prefix ERA-
• the bare prefix of a batch number, DISB-
• a provider ID (NPI)
• a short word: pay
• a junk string: a_b%
• a common surname

Each query ran three times through our real search service.

The surname only went to Gemini. We don't have a BAA with OpenRouter yet, so patient names don't go through it. That's why Jev has 27 calls while Gemini has 30.

Results

For this test, Jev was roughly 2x faster at the median and 2.5x faster at p95. It was also about 7x cheaper and had fewer misses.

Jev costs $0.042 per million input tokens, and output is free. Our questions are short, so each call ends up costing almost nothing.

Gemini's actual cost may be slightly higher than what's shown in the table because our span didn't include thinking tokens.

For our use case, Jev was the better fit of the two.

The biggest difference wasn't just that it was faster and cheaper. The probability output gives us something we can tune. That's useful for a router because we don't really need the model to "answer" the user. We just need a good signal for deciding where to search.

If you need a small classifier in a hot path, that's a pretty good reason not to reach for a chat model by default.