Ankur Jindal 開發可用自訂提示詞並排比較 LLM 的工具

Benchmarks and leaderboards are useful, but they answer a different question from the one I usually have.

They tell you how a model did on someone else's test. When I need a model for a real use case, I want to know how it handles my prompt: the one my team will send every day, with our tone, our format rules and our edge cases.

The quickest way I know to answer that is to try the prompt on a few models and read the answers. So I built a small tool that makes that loop fast:

1. Type one prompt.
2. See up to three models answer it side by side, with the same settings.
3. Swap a model out, run it again, and keep going until one clearly fits.

An LLM judge scores each answer and explains why, which helps when the answers are long. It's a proof of concept built with FastAPI and React, and the code is on GitHub: jindalankuraws/ai-evals.

This post walks through how it works.

What the tool does

• Three model slots. Each slot takes any model from any configured provider. A run needs at least two.
• Four providers. OpenRouter, OpenAI, Anthropic, and any server that speaks the OpenAI chat-completions API, such as Ollama, vLLM or LM Studio. You can mix them in one run, including the same model from two providers.
• One prompt, one set of settings. An optional system prompt, temperature, top_p and max tokens apply to every model in the run.
• Live streaming. All answers stream into their panes at the same time.
• A judge. After the answers finish, a separate model scores them on five metrics, shows its reasons, and highlights the strongest answer.
• Swap and repeat. Remove a model from its slot, pick another, and run again.

Architecture overview

Browser (React)
   │  POST /api/run  (prompt, settings, models, judge)
   ▼
FastAPI backend
   ├── settings check      every filled-in setting must be supported by every model
   ├── fan-out             one async task per model
   │      ├── OpenRouter adapter
   │      ├── OpenAI adapter
   │      ├── Anthropic adapter
   │      └── OpenAI-compatible adapter
   ├── stream              all answers return on one server-sent events response
   └── judge               POST /api/runs/{id}/evaluate, once the stream ends

The backend is Python with FastAPI and httpx. The frontend is React, TypeScript, Vite and Tailwind CSS. Runs are kept in memory, which is enough for a tool you use interactively.

Same prompt, same settings

The settings every model receives are one small Pydantic model. Anything left empty is not sent at all, so each model uses its own default.

class SharedParams(BaseModel):
    """Inference params applied to every model in a run; anything unset uses the model's default."""

    temperature: Optional[float] = Field(None, ge=0, le=2)
    top_p: Optional[float] = Field(None, gt=0, le=1)
    max_tokens: Optional[int] = Field(None, ge=1)

Before a run starts, the backend checks each filled-in setting against what each model says it supports. If one model can't take a setting, the run is rejected with a message naming that model, rather than silently dropping the setting for some models and not others.

for field in set_params:
    key = PARAM_TO_CATALOG_KEY[field]
    if key not in supported:
        problems.append(
            "%s does not support '%s'. Clear the param for all models, "
            "or remove the model." % (model_id, key)
        )

Where the support list comes from depends on the provider. OpenRouter publishes supported_parameters for every model in its catalog, so that check is exact. OpenAI and Anthropic don't list sampling parameters per model, so the app assumes they are supported. That assumption is the first item in the limitations section below.

Every model has a composite id: provider::model, for example openrouter::anthropic/claude-haiku-4.5 or anthropic::claude-haiku-4.5. The backend splits the id to choose the adapter, and each adapter only ever sees the bare model name.

Streaming several models at once

Each model runs in its own asyncio task. Every task pushes its events into one shared queue, tagged with the model id, and the route forwards them to the browser as they arrive. Whichever model produces a token first shows it first.

async def consume(model_id: str) -> None:
    try:
        adapter = adapter_for(model_id)
        _, bare_model = parse_composite(model_id)
        async for ev in adapter.stream_completion(bare_model, prompt, system_prompt, params):
            await queue.put(TaggedEvent(model_id, ev))
    except Exception as exc:  # never let one model's crash kill the fan-out
        await queue.put(TaggedEvent(model_id, StreamEvent(kind="error", error=str(exc))))
    finally:
        await queue.put(None)  # sentinel: this model is finished

Two details matter here:

• One model failing doesn't stop the others. A bad key, an unknown model or a dropped connection becomes an error event for that model's pane, and the other answers keep streaming.
• The None sentinel lets the loop count finished models. When every task has sent one, the stream closes and the judge can start.

On the browser side, the run is a POST with a JSON body. The browser's built-in EventSource only takes a URL and a withCredentials flag, with no way to set the method or send a body (MDN). So the frontend reads the response with fetch and a stream reader, and splits the server-sent events itself.

The judge

Reading three long answers carefully takes time, so a separate model scores them. Each of the five metrics has a weight and a 1–5 scale, with written descriptions of what a 1 and a 5 look like:

There is one judge call per metric, and each call sees every answer, so the scores are relative to each other. Scores are scaled to 0–1 and combined using the weights. When the top two finish within 0.15 of each other, one extra head-to-head call decides between them.

LLM judges have known biases. The MT-Bench paper describes position bias (favouring an answer because of where it appears) and self-enhancement bias (favouring answers that resemble the judge's own) (Zheng et al., 2023). The tool handles these in three ways.

1. The judge never sees model names. Answers are labelled A, B and C, and the mapping back to models stays on the server.

2. The order is shuffled on every call. Letters are assigned in a random order once per run. Then each metric call shows the answers in a fresh random order, so no model is always first.

rng = random.Random(seed)
completed = [r for r in record.results.values() if r.text and not r.error]
rng.shuffle(completed)
...
letters = {r.model_id: LETTERS[i] for i, r in enumerate(completed)}
responses = [(letters[r.model_id], r.text) for r in completed]
...
_call_metric(adapter, record, m, rng.sample(responses, len(responses)), expected_letters, params)

The optional seed is there so tests get the same order every time.

3. The judge can't belong to a compared model's family. The family of a model is its vendor prefix on OpenRouter, or the provider for direct OpenAI and Anthropic models. So openai::gpt-4.1-mini and openrouter::openai/gpt-4o count as the same family.

def _family(model_id: str) -> str:
    provider, model = parse_composite(model_id)
    if "/" in model:
        return model.split("/", 1)[0]
    if provider == "openai-compatible":
        return model
    return provider

The judge's reply must be JSON with a score and a reason for every letter. The backend asks for a strict JSON schema where the provider supports one. It then validates the reply with Pydantic and checks that every letter appears exactly once. A reply that fails gets one retry, and a metric that still fails is left out. Its weight is shared across the metrics that did score, and the UI says which metric is missing.

Provider differences worth knowing

Four providers means four slightly different APIs. These details affected the design:

• Anthropic requires max_tokens on every request. When the user leaves Max tokens empty, the app sends the model's own maximum from Anthropic's model list, or 8,192 tokens if the list doesn't give one. That way Claude isn't cut short while other models run without a limit.
• Newer models reject a custom temperature. Anthropic's deprecations page says Claude 4.7 and later return an error for a non-default temperature, top_p or top_k (Anthropic). OpenAI's guide for its latest models says to drop temperature and top_p unless reasoning effort is none (OpenAI). The judge asks for temperature=0 for more consistent scores, and retries without it if the model refuses:

async def _complete(adapter, model_id, user, system, params, schema):
    """Call the judge, retrying once without temperature if the model rejects it."""
    raw = await adapter.complete(model_id, user, system, params, schema)
    if raw.error and params.temperature is not None and "temperature" in raw.error.lower():
        params = SharedParams()
        raw = await adapter.complete(model_id, user, system, params, schema)
    return raw, params

• Errors can arrive partway through a stream. Anthropic can send an error event mid-stream, for example when it is overloaded (Anthropic). OpenRouter does the same with an error field in a chunk (OpenRouter). Both adapters treat that as a failed answer, so a half-finished answer is never judged as if it were complete.

Try it

You need at least one provider key, or a local OpenAI-compatible server.

# backend
cd backend
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # add at least one provider key
uvicorn app.main:app --reload --port 8000

# frontend, in a second terminal
cd frontend
npm install
npm run dev

Then open http://localhost:5173, fill two or three slots, and type your prompt.

The code is at github.com/jindalankuraws/ai-evals. When you pick a model for a real workload, what decides it for you: your own test prompts, a judge model's scores, or a person reading the answers?