語音對語音 AI 如何聽懂人聲並生成回應？

Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

You might have heard of audio-to-audio AI models. But have you wondered how they work behind the scenes?

An audio-to-audio model listens to your voice and responds with a voice. Unlike a traditional speech pipeline, it does not necessarily need to convert your speech into text before generating a response.

Instead, it can represent sound as a sequence of small codes. The model processes these codes to understand the input and generate a response, then converts the output codes back into audio.

Let's break down how this works.

1. Sound Is Just Numbers

Consider a microphone.

When you speak, sound waves travel through the air. The microphone captures these vibrations and converts them into an electrical signal. An audio system then samples that signal many times per second, storing each measurement as a number.

For example, audio sampled at 24,000 samples per second contains 24,000 numerical samples for every second of audio.

That means a 10-second voice message contains 240,000 samples.

That's a lot of numbers for a model to process directly. More importantly, the raw waveform contains far more detail than the model needs to represent speech efficiently.

We need a way to compress the sound while preserving the information that matters.

2. The Compression Tool: A Neural Audio Codec

A codec is a system that encodes data into a representation and decodes it back. Think of it as a tool for compressing and reconstructing information.

A neural audio codec does this for sound using neural networks. Examples include SoundStream, EnCodec, and Mimi.

A neural audio codec has two main components:

• Encoder: Converts the audio waveform into a compact sequence of codes.
• Decoder: Uses those codes to reconstruct an audio waveform.

Think of a song and its sheet music. The recording contains a huge amount of information, while the sheet music provides a more compact representation of what needs to be played. A musician can use it to recreate the song, although the performance may not be identical to the original recording.

Audio codes serve a somewhat similar purpose. They represent the sound in a form that a model can process more efficiently.

3. Turning Sound Into Frames

The encoder does not necessarily represent the entire recording as one giant code. Instead, it divides the audio into small time steps called frames.

Consider a codec that produces 12.5 frames per second. Each frame corresponds to approximately 80 milliseconds of audio.

The encoder converts each frame into a compact representation. The model can then process these representations in sequence, much like a language model processes tokens.

But there is another detail: a frame can contain more than one code.

4. Why Each Frame Can Have a Stack of Codes

Some neural audio codecs use a technique called Residual Vector Quantization (RVQ).

The name sounds complicated, but the basic idea is straightforward: represent the audio using several layers of codes, with each layer refining the representation provided by the previous layers.

Imagine approximating the number 7.34:

• Code 1: Represents approximately 7.
• Code 2: Adds a correction of approximately 0.3.
• Code 3: Adds another correction of approximately 0.04.

Together, these values represent 7.34 more accurately than the first approximation alone.

RVQ works with learned codebooks and vector representations rather than literally adding decimal numbers like this. The example illustrates the idea of progressively refining a representation.

Each frame can therefore contain a stack of codes, with each code contributing additional information about the audio.

5. How Much Does the Representation Shrink?

Let's use a simplified example for one second of audio.

With these assumptions, the audio is represented by approximately 100 discrete codes instead of 24,000 waveform samples.

That is a reduction of roughly 240 to 1 in the number of represented values.

The decoder can use the resulting codes to reconstruct speech that preserves much of the original audio's important characteristics.

6. Speech Contains Meaning and Sound Details

When someone speaks, the audio carries more than the words themselves.

There are two useful ways to think about this information:

• Semantic information: What is being communicated. For example, "Please close the door."
• Acoustic information: How the message sounds, including voice characteristics, intonation, speaking rate, pauses, and emotional expression.

A voice assistant needs to handle both. Understanding the words is not enough if the goal is to produce natural, expressive speech. Sound details matter too.

Different audio models organize this information in different ways. For example:

• Separate representations: A system may use distinct representations for semantic information and acoustic details.
• Layered representations: Some codec designs arrange multiple code sequences so that different layers capture different levels of detail.

However, these are not universal rules. In an RVQ codec, the first code is not automatically a semantic representation, and later codes are not necessarily dedicated to emotion or voice characteristics. That behavior depends on the model's architecture and training.

The key idea is that a model must represent enough information to understand the input and generate the kind of audio response it is designed to produce.

7. How an Audio-to-Audio Model Responds

Once the input speech has been encoded into codes, the main model can process them and generate codes for its response.

The overall process has three main stages.

Step 1: Listen

The audio encoder converts the incoming waveform into a sequence of audio codes.

Step 2: Predict

The main model processes the input representation and generates a sequence of output codes. Depending on the architecture, it may also use text tokens, separate audio representations, or other intermediate information.

Like an autoregressive language model, an audio model may generate tokens one step at a time. Other architectures can organize generation differently.

Step 3: Speak

The audio decoder converts the generated codes back into a waveform, producing the model's spoken response.

The simplified pipeline looks like this:

Input waveform → Encoder → Audio codes → Model generates response codes → Decoder → Output waveform

The model does not have to treat audio as raw numbers throughout the entire process. The codec gives it a compact representation that is more suitable for modeling.

8. Why Not Convert Speech to Text First?

A traditional speech-based assistant might use a three-stage pipeline:

Speech → Speech recognition → Text model → Text-to-speech → Spoken response

This approach works well for many applications. However, converting speech into text can discard information that is not represented in the transcript.

For example:

• Emotion: A transcript may preserve the words but lose much of the excitement, frustration, or uncertainty in the speaker's voice.
• Timing: Pauses, speaking rhythm, and interruptions may not be fully represented in the text.
• Non-speech sounds: Laughter, sighs, and other vocal sounds may be omitted or represented only with additional labels.

An audio-to-audio model can preserve and process more of this information when its architecture and training support it. That can help it produce responses with more natural timing and vocal expression.

It may also reduce the need for separate speech-recognition and speech-synthesis stages, potentially lowering latency. The actual speed advantage depends on the model and the system in which it runs.

Conclusion

Audio-to-audio AI models do not have to process an entire sound waveform directly. A neural audio codec converts the waveform into compact codes, a model processes those codes to generate a response, and a decoder reconstructs the output as sound.

The important idea is that audio codes give AI models a compact representation of sound that they can learn to understand and generate.

This is one of the building blocks behind more natural voice interactions, where an AI system can respond to spoken input without relying entirely on a speech-to-text-to-speech pipeline.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:

• Understand risky changes faster.
• Fix problems in those changes, with human attention directed to the work that needs it.
• Enforce your engineering standards.
• Learn from production outcomes, such as incidents, rollbacks, and queue wait times.
• Improve the review process over time and make it more efficient.
• Connect to the major engineering tools that your team already uses.

Auto-Approve: Ship the Safe 50% Without a Human

Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.

demo.mp4

LiveReview auto-approving a low-risk change while…

Click below to try LiveReview with your codebase: