Sherlock Holmes：語音智能體預熱為何沒有加快首次回覆

This is one case from the Sherlock Holmes series. Each write-up is a production problem me had to take apart. feel free to checkout other blogs.

This one did not start as a latency mystery. It started as a bill.

We prewarm a voice agent before the call connects. You send the model a request early, and you open the speech sockets early, so that if a person picks up, the first reply is already on its way. You pay a little up front, and the caller is not supposed to sit in silence while everything wakes up. That is the whole idea, and for a long time we treated it as obviously worth it.

Then we looked at where the LLM money was actually going. Almost 60% of the voice agent's model cost was the prewarm. Not the turns where someone was talking to the agent. The request we fired in case they might.

So the first question was not whether prewarm was fast. It was who we were warming up for.

The calls were not conversations

Once we split a day's calls open, they did not look like the conversations we thought we were paying to speed up. About 28% ended before the agent had played a single message. Another 63% got as far as the greeting and then nothing came back. The caller never spoke. Most of what was left was over inside one or two turns. Voicemail, no answer, or someone who picked up and cut the line straight away.

We had already paid the model to get ready for calls that never became a conversation.

Retries made that impossible to miss. Out of 48,103 calls that day, 32,244 were first attempts and 15,859 were retries, about a third. Of those retries, 15,143 failed, voicemail or no answer. About 95% of the time a retry was the same dead call again. If you warm every one of those, you are paying a model request on a pile of calls that are very unlikely to have a person waiting on the other end.

That was a small enough change to try. We left prewarm on for ordinary calls and turned it off for retries. After that we were simply not paying for the voicemail, the no-answer, and the calls that ended the moment they connected.

Once the change had been out for a bit, we went back and analysed the calls. Ordinary calls still had the prewarm. Retries did not. The cost side looked the way we hoped. Then one thing in the latency did not.

Both groups were almost the same.

That felt fishy. The prewarm was supposed to be why the first reply came out faster, so the retries, running without it, should have been the slow ones. They were not. I did not want to shrug at a chart, so I dug into the calls behind it.

What came out was a p50 of 1,235 ms with prewarm off and 1,163 ms with it on. About 72 ms. p95 was 2,118 ms against 2,149 ms. We watched the same split over the next days, prewarm completed on one side and prewarm skipped on the other, and the two lines still sat together.

Production traffic moves a lot of things at once, so I did not want to stop there. I took the production prewarm and the first-turn request, put them in a small script, and ran both ways on a VM. One run let the prime finish. The other skipped it. The speed came back about the same. Not much in it either way.

That is the part that took a while to sit right, because the cache was doing its job. On the GPT-5.1 prewarm replies, about 98% hit the cache, and around 70% of the prompt tokens were billed as cached. The prompt was warm. The caller still waited. We went looking for anyone else who had measured this, and Vlad Iliescu's notes on prompt caching with Azure OpenAI landed in the same place. He cached the prompt and still did not see a real jump in speed, because the reply is generated again every time. A cache hit only means the model did not have to read the prompt again. The reply is new on every call, so it still has to be generated, token by token, and that is the wait the caller feels. You can have a high cache rate, and you can spend real money keeping that prefix warm, and the first sentence still arrives when the generation finishes. The prewarm was not a head start they could hear. It was a bill for a prefix.

Once that stayed true past a single day, we stopped treating it as a retry tweak and turned the LLM prewarm off for every call.

The speech path was telling the same story

Taking the model prewarm off is what finally pushed us into the rest of the code. Speech to text and text to speech were being warmed too, and we had left them alone because a socket you open early sounds like a socket you will not have to open later. Worth reading the code before you believe that.

Speech to text went first, and it fell apart as soon as we followed the socket. While the phone is still ringing we open a second Deepgram connection, and when the person answers we close it. The call is not listening on that connection. LiveKit has already opened the socket the audio actually flows through, so the one we opened for prewarm never receives the caller's voice. We open it, the call connects, we close it, and nothing they said has passed through it. It might still warm a DNS lookup or a TLS handshake. On bulk calling, the call before this one has usually done that already. There is no audio path being reused. We had been warming a socket that could not hear.

Text to speech is the place the idea is actually real, which is why it is worth not throwing the whole thing out here. For some providers, the warmup opens a WebSocket and leaves that connection ready. When the agent speaks the first line, it uses the connection that is already open instead of dialing a new one. That is a real head start, and it is what these providers do: Cartesia, ElevenLabs, Deepgram TTS, Sarvam, Fish Audio, and Soniox.

Two of them do not do that. With Inworld, the warmup only sets aside an empty pool. The WebSocket is still opened when the first line is generated, which is the moment you were hoping to have already got past. With OpenAI, the warmup sends a light HTTP request and keeps no connection at all. By the time the agent has something to say, both of them are starting cold. We had been counting that warmup as done, and for those two nothing was being held open for the first line.

Where it landed

Follow it back and the three findings are one story. We were paying to get ready for calls nobody answered, the model warmup was not making the first reply faster, and two of the speech warmups were not holding anything the call could use.

So the LLM prewarm stays off. The speech-to-text prewarm comes out with it, because a second socket that never hears the caller is not a warmup. Text-to-speech prewarm stays, but only where the provider really does keep a connection waiting. Where the warmup only sets aside an empty pool, or pings and hangs up, we stop doing it.

That is the part I would want you to take from this. The bill was mostly for calls that had already ended.

Have you hit the same thing? A warmup, or a cached prompt, that looked fine on your side while the person on the call still waited. If you have, I would like to hear how it showed up for you.