tokio-prompt-orchestrator 基準測試揭示併發瓶頸，2.0.3 將 20 個不同提示詞處理時間降至 3.0 s

Originally published at tensorust.com.

TL;DR: On October 17, 2026 I put my open-source Rust LLM proxy, tokio-prompt-orchestrator, between the official OpenAI SDK and a fake provider, and measured what it actually does. 200 identical requests fired at once became 1 upstream call. During a 60 second provider outage, the circuit breaker cut the calls that hit the dead API from 89 to 5. And the same test found a bug I had shipped: the server sends one upstream call at a time, so 20 different prompts sent together took 30.2 s instead of 1.5 s. That bug and two smaller ones are fixed in 2.0.3, out on crates.io on October 18: the same 20 prompts now take 3.0 s at the default settings and 1.5 s with --concurrency 32.

Short answer for the search. An LLM proxy is a small server that sits between your app and the model API. Your client keeps its code and only changes its base URL. The proxy can then merge identical requests into one paid call, stop calling a provider that is failing (a circuit breaker), cap spend, and log every call. On real chatbot traffic, exact deduplication only catches about 1.5% of calls (measured below on 172,418 real conversation turns), so its value is in retry storms, double clicks, agent loops and fan-out jobs, not in halving a normal chat bill.

The crate is MIT licensed: crates.io, docs.rs, and the rest of my Rust crates are on tensorust.com/open-source/. I wrote it, so read the numbers with that in mind; every script and raw result file is listed at the end so you can rerun them. The three problems this test found are fixed in tokio-prompt-orchestrator 2.0.3 (commit 1f251b8); the before and after numbers are both below.

200 identical requests from the openai-node SDK, provider latency 1.5 s. Orchestrator v2.0.1 release binary, Oct 17 2026, 7:14 PM ET.

The setup: one SDK, two base URLs

I wanted the comparison a reader would actually face: same app code, with and without the proxy. So every request in this test comes from the official openai-node SDK (version 4.104.0) with its defaults untouched. That matters, because those defaults are aggressive: a 10 minute timeout and 2 automatic retries with exponential backoff on 408, 409, 429 and every 5xx. The Python SDK ships the same defaults.

The only thing that changes between the two runs is one line:

// run.mjs (trimmed). The "direct" run talks to the provider, the "proxy" run to the orchestrator.
const baseURL = (target === 'direct' ? MOCK : PROXY) + '/v1';
const client = new OpenAI({ baseURL, apiKey: 'local' });

const { data, response } = await client.chat.completions
  .create({ model: 'gpt-4o-mini', messages: [{ role: 'user', content: prompt }] })
  .withResponse();
rows.push({ start, end: Date.now() - T0, ok: true, dedup: response.headers.get('x-orchestrator-dedup') });

The provider is a 60-line Node server I wrote for the test. It speaks both the OpenAI chat format (for the direct run) and the llama.cpp /completion format (what the orchestrator calls with --provider llama). It answers after a fixed delay, and it has two failure modes I can flip mid-run: fast 503s, and hang, where it accepts the connection and never answers. Most importantly, it counts every call that reaches it. That count is the number your provider bills you for, and the number that hammers a provider while it is trying to recover.

Why a fake provider instead of a real one? Because I can't make OpenAI go down on cue, and I didn't want to pay for 600 calls to measure a counter. The latency and failure shapes are simple on purpose. The proxy is the real thing: the v2.0.1 Windows release binary, downloaded the same way a reader would, SHA-256 46a995fa...43d5, started with:

LLAMA_CPP_URL=http://127.0.0.1:18081 orchestrator --provider llama --port 18080

Test 1: 200 identical requests at once became 1 paid call

The first scenario is a stampede: the same prompt, 200 times, all in flight at the same moment. In real apps this is a cache miss on a popular page, a queue consumer that got the same job 200 times, a user mashing a button, or an agent framework that retries a slow call while the first one is still running. The provider takes 1.5 s per answer.

Direct: 200 requests, 200 upstream calls. Through the proxy: 1 upstream call. The response headers tell you what happened to each request: 1 was a miss (it made the call) and 199 were joined (they waited on that call). The slowest answer took 1.58 s either way, so the merge cost nothing in latency.

Here is the code that does it, from src/web_api/openai_compat.rs, trimmed:

/// Answer `prompt`, sharing identical in-flight or recent requests.
async fn complete(state: &Arc<AppState>, request_id: &str, session: String, prompt: String)
    -> Result<(String, DedupOutcome), ApiError>
{
    let key = hex::encode(Sha256::digest(prompt.as_bytes()));

    for _ in 0..MAX_JOIN_ATTEMPTS {
        let (found, similar) = state.openai.dedup.check_and_register_semantic(&key, &prompt).await;
        match found {
            DeduplicationResult::Cached(text) => return Ok((text, DedupOutcome::Cached)),
            DeduplicationResult::InProgress => match state.openai.dedup.wait_for_result(&key).await {
                Some(text) if !is_cancelled_result(&text) => return Ok((text, DedupOutcome::Joined)),
                _ => { /* the call we joined failed or was cancelled: retry the loop */ continue; }
            },
            DeduplicationResult::New(token) => {
                // Run the upstream call in its own task so a client that
                // disconnects does not cancel it for everyone who joined.
                let task = tokio::spawn(async move { /* run_pipeline, then dedup.complete(token, text) */ });
                return match task.await { /* ... */ };
            }
        }
    }
    // ...
}

Two details in there are worth stealing for your own code, in any language. First, the key is a hash of the whole flattened conversation, not just the last message, so two users who typed the same words into different chats never share an answer. Second, the upstream call runs in its own spawned task. If the first caller hangs up, the 199 who joined still get their answer. Without that, one impatient client would cancel the call for everyone. Go programmers will recognise the pattern as singleflight.

Finished answers stay reusable for a window (300 s by default, ORCHESTRATOR_DEDUP_SECS to change it), stored in a moka cache with a size limit so a flood of different prompts can't grow memory forever.

How much does dedup save on real traffic? 1.5%

A stampede is the best case. I wanted the honest number for normal traffic, so I replayed real chatbot logs. WildChat-1M from the Allen Institute is a public dataset of real conversations people had with a free ChatGPT front end, with timestamps (paper). I took the first parquet shard: 59,857 conversations, 172,418 user turns, 25.3 days of traffic from April to May 2023.

For every turn I rebuilt the key the proxy would use (a SHA-256 of the whole conversation so far) and asked: was this exact request already sent recently?

WildChat-1M shard 0000 (59,857 conversations, Apr to May 2023), replayed with the proxy dedup key, Oct 17 2026.

At the default 5 minute window, 2,568 of 172,418 calls (1.49%) were exact repeats. Stretching the window to 24 hours gets 4.39%, and the whole 25 days only 4.77%. For a normal chat app, exact dedup is a small win, not a bill cutter. Anyone who tells you a cache in front of a chatbot halves your spend should show you their logs.

The breakdown is the useful part. Within 5 minutes, 91% of repeats came from the same user (same hashed IP) sending the same thing again: refreshes, resubmits, "it didn't answer, let me try again". And the most repeated requests were not humans at all. Near the top of the list: the system prompt AutoGPT used in 2023, that begins "CONSTRAINTS: 1. ~4000 word limit for short term memory", repeated 76 times within 5 minute windows, and scripted "fill in the blank with either John or Mary" test prompts. Agent loops and scripts are where duplicates live. The two most repeated human openers were "你好" (hello, 93 times) and "hi" (45).

So the rule I'd use: turn dedup on everywhere (it costs nothing on a miss), but expect real savings only where machines send the requests. If you need the reworded-question case, that's semantic dedup, which is a harder problem; I wrote up why embeddings alone will give users the wrong answer in I stopped paying twice for the same LLM answer.

Test 2: the provider goes down, and the breaker stops calling it

Next, an outage. One request every 2 seconds, each with a different prompt (so dedup can't help), for 150 seconds. The provider is healthy for 15 s, answers every call with a 503 from 15 s to 75 s, then recovers.

One request every 2 s, a different prompt each; provider answers 503 from 15 s to 75 s. Real run, Oct 17 2026, 7:16 to 7:21 PM ET.

Direct: 89 calls hit the dead provider in 60 seconds. That is 30 requests, each tried 3 times, because the SDK retries every 5xx twice. Your app tripled its load on a provider that was already falling over. This is the retry storm the AWS Builders' Library article on retries warns about: retries help with a blip and make a real outage worse.

Through the proxy: 5 calls. After 5 consecutive failures the circuit breaker opened, and every later request was refused inside the proxy without touching the provider. That's a 94% cut in calls to a failing API, with no configuration at all.

The breaker is the classic pattern Martin Fowler described in 2014: closed (calls go through), open (calls are refused), half-open (let traffic through to test recovery). The core, from src/enhanced/circuit_breaker.rs, trimmed:

pub async fn call<F, Fut, T, E>(&self, f: F) -> Result<T, CircuitBreakerError<E>>
where F: FnOnce() -> Fut, Fut: std::future::Future<Output = Result<T, E>>,
{
    {
        let mut state = self.state.write().await;
        if let CircuitStatus::Open = state.status {
            // Exponential backoff: each failed half-open probe doubles the
            // wait, capped at 64x the base timeout.
            if let Some(last_failure) = state.last_failure_time {
                let backoff_factor = 1u32 << state.probe_failures.min(6);
                if last_failure.elapsed() >= self.config.timeout * backoff_factor {
                    state.status = CircuitStatus::HalfOpen;
                    state.recent_results.clear();
                    state.failures = 0;
                } else {
                    return Err(CircuitBreakerError::Open); // refused, provider not called
                }
            }
        }
    }
    let result = f().await;
    match &result { Ok(_) => self.record_success().await, Err(_) => self.record_failure().await }
    result.map_err(CircuitBreakerError::Failed)
}

And the settings the server uses, from src/stages.rs:

// Threshold: 5 consecutive failures open the circuit.
// Success rate: 80% of recent requests must succeed to close.
// Timeout: 60 seconds before half-open probe.
let circuit_breaker = CircuitBreaker::new(5, 0.8, std::time::Duration::from_secs(60));

Two honest caveats from the same run. First, your users waited just as long for their error: about 1.3 s both ways. The proxy refused instantly, but the SDK saw a 503 and retried it twice with its 0.5 s and 1 s backoff. The proxy doesn't send Retry-After or x-should-retry: false on a breaker refusal, and the OpenAI SDKs obey x-should-retry when a server sends it. That's a one-line fix on my side, and it would turn those 1.3 s errors into near-instant ones.

Fixed in 2.0.3: a refusal from the open breaker now carries x-should-retry: false and a Retry-After, so the OpenAI SDK stops retrying it. In the same outage run, refused requests came back in about 1 ms instead of about 1.35 s, and overall p95 fell from 1,459 ms to 314 ms. Calls reaching the dead provider stayed at 5.

Second, recovery lags. The breaker opened at 18.6 s and doesn't look again until 60 s after the last failure, so it reopened traffic at 78.8 s. The provider had been healthy since 75 s; the request sent at 76 s still got a 503. In this run the lag was under 4 seconds because the timing happened to line up. In the worst case it is a full minute of refusing a provider that's already fine. That's the trade: fewer calls to a sick API, slower return to a healthy one.

Test 3: a hung provider beats every timeout default

Outages aren't always clean 503s. The nastier version is a provider that accepts your connection and then says nothing. Same load, same 60 second window, but this time the fake provider hangs.

Same load; the provider accepts connections and never answers. Real run, Oct 17 2026, 7:21 to 7:25 PM ET.

Nobody failed fast. Direct, the p95 wait was 53.7 s; the first request in the outage waited 59.7 s, until the provider dropped the stuck connections at 75 s and the SDK's retry went through. With the SDK's 10 minute timeout, a provider that never dropped them would have held every request for 10 minutes, times 3 attempts.

Through the proxy, the p95 was 53.6 s, the same. Only 2 calls reached the dead provider (versus 30 direct), but the breaker never opened. The orchestrator's llama.cpp worker times out after 30 s, and the breaker needs 5 failures, so it would have needed 150 s of hanging to trip. The outage was over before that.

Fixed in 2.0.3, which adds --timeout and counts a timeout as a breaker failure. That was a second bug: before 2.0.3 the pipeline's timeout wrapped the breaker call, so a hung call the pipeline cancelled never counted as a failure at all. With --timeout 5 the breaker opened about 13 s into the hang; the slowest wait fell from 66.3 s to 11.4 s and the median wait for requests sent during the hang from 34.9 s to 1 ms. The price: those requests now fail fast with a 503 instead of succeeding a minute later, which is the point of a breaker. And 9 calls reached the hung provider instead of 2, because the old single worker sat on one hung call at a time (direct was still 30). Keep the timeout short: at the default 30 s the breaker opens too, but late (median wait during the hang 30.4 s), and recovery is slow, with the first success at 136.9 s for an outage that ended at 75 s, because a failed half-open probe doubles the 60 s open period. That period isn't a flag yet.

The lesson is the most useful thing in this whole article: a circuit breaker can only be as fast as your timeout. 5 failures at a 30 s timeout is 2.5 minutes of hanging before it helps. If your model normally answers in 5 s, set the timeout near your real p99 (say 15 to 20 s), not the default 30 s, 120 s or 10 minutes. Then a hang becomes a failure the breaker can count.

That graph also has a second shape hiding in it. Look at the proxy's tick marks after 75 s: a dense block of calls, one after another, and requests still waiting until 86 s. That's the queue draining one call at a time. Which brings me to the bug.

The bug I found: one upstream call at a time

To check concurrency, I sent 20 different prompts at once, with the provider taking 1.5 s each. Dedup can't help here; it's pure throughput.

20 distinct prompts, 1.5 s provider latency, Oct 17 2026, 7:14 PM ET.

Direct: every answer in 1.5 s. Through the proxy: a staircase, 1.5 s, 3.0 s, 4.5 s ... 30.2 s. The proxy made exactly the same 20 calls, but strictly one after another. The orchestrator binary serialises every upstream call.

Fixed in 2.0.3: the server now runs up to 16 upstream calls at once (--concurrency), so the same 20 different prompts finish in 3.0 s at the default (16, then a second round of 4) and in 1.5 s with --concurrency 32, the same as calling the provider directly. Identical prompts in flight still share one call: the 200-request herd is still 1 upstream call (1 miss, 199 joined, slowest answer 1.60 s).

The cause is in src/stages.rs. The inference stage is one Tokio task that takes a request off its channel and awaits the model before it takes the next one:

async fn inference_stage(mut rx: mpsc::Receiver<AssembleOutput>, /* ... */) {
    loop {
        let assemble_output = tokio::select! {
            biased;
            _ = cancel.cancelled() => break,
            item = rx.recv() => match item { Some(req) => req, None => break },
        };
        // ...
        let infer_fut = breaker.call(|| async move { retry.run(|| w.infer(&prompt)).await });
        // the loop waits here, so the next request can't start until this one ends
        let cb_result = match tokio::time::timeout(effective_timeout, infer_fut).await { /* ... */ };
        // ...
    }
}

The library already has the fix: spawn_pipeline_with_config runs a pool of these workers on one shared channel, sized by inference_workers in the pipeline config. But it defaults to 1 "for backward compatibility", and the orchestrator server binary goes through the simpler spawn_pipeline_with, which always uses a single worker. So every user of the server got a one-lane road and nobody, me included, had measured it. The demo in the README (12 requests, 3 model calls) never caught it, because it was built to show dedup, and dedup hides the problem.

This is the part of benchmarking your own code that stings and pays. The dedup and breaker numbers above were what I expected. The staircase was not, and it's the number a stranger would have hit in their first hour. The fix is a worker pool (or a bounded Tokio semaphore) in the server path, with a sensible default like 16 in-flight calls and a --concurrency flag. That is what 2.0.3 does. On an older version, if you use the crate as a library, build the pipeline with spawn_pipeline_with_config and set inference_workers to the number of calls you want in flight.

All the numbers in one place

Real output of the load generator, Oct 17 2026, 7:14 to 7:25 PM ET.

Direct and 2.0.1: Oct 17, 2026, 7:14 to 7:25 PM ET. 2.0.3: same harness, mock and SDK defaults, a release build of commit 1f251b8, 7:48 to 7:57 PM ET; that commit is the 2.0.3 on crates.io.

Do this in your project today

You don't need my crate for most of this. Here's what I'd change in any app that calls an LLM API, in order of payoff:

1. Set a real timeout. The SDK default is 10 minutes. Measure your p99 latency for the model you use and set the timeout a bit above it. In Node: new OpenAI({ timeout: 20_000 }); in Python: OpenAI(timeout=20.0). This one setting decides whether an outage costs your users 20 seconds or 10 minutes.
2. Know what your retries do in an outage. 2 retries means 3 times the load on a struggling provider. Keep them for blips, but put something in front that notices a real outage and stops calling (a breaker), and make it say "don't retry" when it refuses.
3. Merge identical in-flight calls. If anything in your system can send the same request twice at once (queues, agents, retries, cron jobs, cache misses), a singleflight map keyed on a hash of the full conversation turns N calls into 1. It's 30 lines in most languages and it never costs anything on a miss.
4. Don't expect a cache to halve a chat bill. 1.5% exact repeats on real chat traffic. Spend your effort on prompt caching at the provider (Anthropic, OpenAI), smaller models for easy calls, and shorter prompts. Price the options with the free LLM API cost calculator.
5. Load test your proxy with different prompts, not the same one. Same-prompt tests flatter any proxy with dedup. My own demo did exactly that and hid a 20x slowdown.

If you'd rather not write it yourself: cargo install --locked tokio-prompt-orchestrator --features web-api,tantivy (that gets 2.0.3 with the fixes; the prebuilt release binaries are still 2.0.1 as I write this), then point your client at it with OPENAI_BASE_URL=http://127.0.0.1:8080/v1. It speaks the OpenAI and Anthropic APIs, so the official SDKs work unchanged. Other proxies in this space, if you want to compare: LiteLLM (Python) does routing and budgets across many providers. Run the same three tests on whatever you pick. And if your problem is the replies themselves, not the transport, see LLM JSON output: one setting dropped valid JSON from 75% to 4%.

How to rerun this

Everything ran on one Windows 11 PC on October 17, 2026 between 7:14 and 7:25 PM ET, with the 2.0.3 reruns from 7:48 to 7:57 PM ET (the WildChat pass took about a minute after a 230 MB download). The scripts: mock.mjs (the fake provider, about 60 lines), run.mjs (the load generator: node run.mjs herd|concurrent|outage503|outagehang direct|proxy out.json) and wildchat_dups.py (the replay). Every request's start, end, status and dedup header, and every call the provider received, are in the JSON results; the charts are drawn straight from them.

What would change these numbers: a real provider's latency spread (the fake one is constant), streaming responses, and your own traffic mix. The dedup and breaker counts are mechanical and will hold. The 1.5% is one public dataset from 2023; I'd love to see the same measurement on a production agent's logs, where I'd bet the repeat rate is far higher.

Which brings me to the question I can't answer from here: what share of your LLM calls are exact repeats? If you log request hashes, check yours and tell me. I'll add real numbers from readers to this page.

I build AI features into apps for a living, with the boring reliability parts (timeouts, retries, budgets, evals) included. Get AI built into your app.