跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-06AI 評分41

Hiding Tool Latency in On-Device Cascaded Voice Agent through Speculative Execution

Hiding Tool Latency in On-Device Cascaded Voice Agent through Speculative Execution

AI 導讀

一種面向裝置端級聯語音智能體的推測式工具執行方法,會根據部分 ASR 假設預測工具呼叫,並在用戶仍在説話時啟動執行,以降低端到端回應延遲。在完整實作的 Android 語音助理實測中,首次音訊輸出的中位時間由 5.79 秒降至 4.60 秒,標準差由 3.49 秒降至 2.81 秒。

正文

Published on Oct 6

Authors:

,

,

,

,

,

,

Abstract

Tool-augmented speech assistants typically serialize automatic speech recognition, large language model inference, and external tool execution. As a result, tool latency is incurred only after the user has finished speaking and the LLM has identified the required tool calls. We present speculative tool execution for on-device cascaded voice agents, which predicts tool requests from partial ASR hypotheses and initiates tool execution while speech is still being received, thereby reducing end-to-end response latency. Our approach introduces a Predictor module that anticipates tool calls during speech recognition, executes them speculatively, and caches the results. The cached outputs are then injected into the LLM prompt, enabling faster responses. Additionally, to mitigate errors caused by user self-corrections during speech, we employ a rule-based validation mechanism that selectively injects only valid cached results. As a final safeguard, the LLM retains the ability to issue tool calls directly, ensuring that the latency of our framework is upper-bounded by the baseline serial execution pipeline in the worst case. We evaluate our method using live measurements from a fully implemented Android voice assistant. Our approach reduces the median time-to-first-audio from 5.79,s to 4.60,s and decreases the standard deviation from 3.49,s to 2.81,s, resulting in more predictable response latency.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.07641

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.07641 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.07641 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.07641 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co