Alex Smola 訪談：Boson AI 語音智能體的企業部署與設計

Alex Smola is co-founder of Boson AI, where he works on conversational voice AI and multimodal systems. His career spans academic research at Carnegie Mellon University and large-scale AI initiatives at AWS. He brings that combination of research and product experience to building assistants that can understand conversational cues, handle interruptions, and carry out useful work.

Boson AI develops foundation models and tools for voice, audio, and multimodal interaction. Its Higgs Realtime model supports live speech-to-speech conversations, while its broader platform includes expressive speech generation, avatars, and enterprise voice agents. In this interview, Smola discusses conversational intelligence, multilingual performance, inference economics, and the safeguards needed to move voice agents into real business workflows.

Your career has taken you from academic research at Carnegie Mellon University to leading large-scale AI initiatives at AWS, and now to co-founding Boson AI. What inspired you to focus on conversational voice AI, and what fundamental limitations in human-AI interaction are you hoping to overcome?

Humans communicate through much more than words. You can hear uncertainty in someone’s voice or see that they haven’t understood an explanation. A small acknowledgement can tell you to keep talking. We pick up on these cues without thinking, but teaching computers to recognize them is still a work in progress.

That is what makes this area so interesting to me. We now have systems with considerable intelligence, but communicating with them can still feel awkward. I want people to be able to explain what they need in their own words, without having to figure out how to talk to the software.

At AWS, I learned to pay close attention to what customers were trying to do and whether the product helped them do it. We’ve brought that approach to Boson, with the freedom to move quickly as the research advances.

Voice and video give us more to work with. If someone looks confused or hesitates, the assistant has an opportunity to respond before the conversation goes off course. Getting those everyday interactions right is a big part of our ambition.

Boson AI recently introduced Higgs Realtime, designed to enable natural, real-time speech-to-speech interactions. What makes this approach fundamentally different from traditional systems that combine speech recognition, language models, and text-to-speech, and why does that distinction matter for the future of AI assistants?

In a traditional pipeline, speech recognition turns your words into text, a language model generates an answer, and a speech model reads it aloud. Each step depends on the one before it, which can make the conversation feel slow and disjointed.

Higgs Realtime is an audio-native speech-to-speech model, built for live interaction. Its training addresses conversational behavior, including turn-taking and responding to changing requests. That’s very different from putting a voice on a text chatbot.

Think about when you ask someone a difficult question. They might say, “Let me look that up,” while they work on it. You know they’ve heard you and are working on it. An AI assistant needs to do the same while it retrieves information or uses a tool, so you’re not left wondering whether it understood you or stopped working.

That becomes even more important with longer tasks. The assistant needs to keep you informed as it works and respond if you ask a question or change your mind.

One of the biggest challenges in conversational AI is knowing when to speak, when to listen, and how to handle interruptions or changes in direction. How is Boson AI addressing these challenges, and what technical breakthroughs are still needed before AI conversations feel genuinely natural?

An acknowledgment and an interruption can sound quite similar but they need different responses. If someone says “yeah” while you’re talking, they may want you to keep going. If they correct a digit in their phone number, you need to stop and use the right number. An AI assistant should be able to tell the difference.

Higgs Realtime uses the meaning of what someone says to help decide whether they’ve finished speaking or are taking a pause. Responding quickly is important, but so is knowing how to respond.

Getting that right consistently is still quite difficult. People don’t speak in neat turns and the same words can mean different things depending on the conversation. The assistant has to understand what changed and remember what it was doing.

Boson AI has developed benchmarks such as IHBench to measure how voice agents recover from interruptions. What have these evaluations revealed about the weaknesses of today’s leading voice models, and how should the industry rethink the way it measures conversational intelligence?

We built IHBench to test what happens after someone interrupts a voice agent. If you correct something halfway through a booking, the agent should take the correction and carry on from the right point. You should not have to start again, or sit through information it has already given you. The results varied a lot depending on the model and the type of interruption. One finding we did not especially enjoy publishing: the closed-weight systems generally handled interruptions better, and they held up better the deeper into a conversation the interruption landed.

That is a real gap in the open ecosystem, and we would rather measure it in public than pretend it isn’t there. It is also most of the reason the benchmark exists. You cannot close a gap that nobody has agreed how to measure. We published that benchmark with no model of our own in it. When we did put one in, Higgs Realtime scored 71.2 on recovery quality, ahead of GPT Realtime 1.5 at 65.5, GPT Realtime 2 at 62.4 and Gemini 3.1 Flash Live at 61.1. So the gap is closable, and an open-weight model closed it.

I would rather show that on a test we designed to be hard and published before we had anything to show on it. More broadly, I think the field measures the wrong things. A voice can sound completely natural and still fail at the job, because the test that matters is whether the agent finishes the task. If you have to keep reminding it what you were doing, how good the voice sounds stops mattering very quickly.

There is a significant difference between an impressive voice AI demonstration and an autonomous agent capable of handling real customer interactions. What separates the two, and which capabilities must improve before enterprises can confidently deploy voice agents in business-critical workflows?

When was the last time you enjoyed calling your cable company? For most people the answer is never. You get handed from one person to the next, and each time you explain your name, your date of birth and your problem again, so that the next person can ask you for all of it again. So take the fourth call about the same problem. You want the agent to know what happened on the first three. If restarting the router hasn’t worked, perhaps it is time to send a technician, and you should not have to make that case yourself. For the agent to do that, it needs your account history and access to the system that books the visit.

It also has to cope when the booking fails, and know when to hand you to a person who can actually fix it. Those are the situations a business needs to test before it puts an agent on customer calls. Then there is the cost of running it. A system that handles a demo beautifully can be far too expensive to run for thousands of calls a day, so efficiency is part of being ready for business rather than a separate concern.

We have been running this exact scenario with a global insurance company for several months. It is not a trial sitting alongside the real system. It handles live customer conversations, inbound and outbound, next to their human sales agents, in their contact centre. Thousands of concurrent calls, at roughly a third of the cost of a human agent.

The part I find more interesting than the cost is that it improved in place. Their own evaluation scored our conversation quality at 30 when we started. Five months later it was at 80, which is the score they use for a qualified human telemarketer. GPT-4o Realtime scored 55 on the same evaluation. We did not tune our way to that in a lab. The system got better against the traffic it was actually handling, which is the only kind of improvement a contact centre cares about.

As voice agents move beyond answering questions to executing tasks, accessing customer information, and interacting with enterprise systems, how should developers balance autonomy with human oversight? What safeguards are essential to establishing long-term trust in these systems?

I’d start by asking what you’re trusting the agent to do. Letting it look up an account balance is very different from allowing it to move money or cancel a service. The permissions should reflect that, and the systems it connects to need to enforce them. You can’t leave all of that to an instruction in a prompt.

People should also know what they’re agreeing to. Before an agent makes a change that would be tough to undo, it should explain the consequences and ask for confirmation. That way when it gets stuck, getting a person involved should be straightforward.

Things will still go wrong. When they do, you need to be able to see what the agent did and where it stopped, so someone can put it right. A voice that sounds confident can make us assume a system knows what it’s doing but we need to earn that confidence through how reliably it behaves.

Higgs Realtime supports more than 100 languages, including the ability to switch languages during a conversation. What are the biggest technical challenges in delivering consistent conversational intelligence across different languages, accents, and communication styles, particularly when understanding emotional context and user intent?

Language coverage sounds like a list, and it isn’t. A while ago a customer needed Thai. We had partial coverage, and the first thing we discovered was that we knew absolutely nothing about Thai proper, starting with the fact that written Thai essentially has no punctuation. That breaks the sentence-boundary assumptions most speech systems are built on. We could have deferred the market. Instead we found computer scientists who spoke Thai and rebuilt the handling properly.

About six weeks later we were in Bangkok demonstrating it. During the demo I could not tell how it was going — I was listening to people test a system in a language I don’t speak — and afterwards they said it was good, which was a considerable relief. Conventions differ as much as vocabulary does. In Japanese, listeners make small acknowledgments while you are speaking to show they are following along. If the assistant stops every time to hand over the turn, it destroys the rhythm of the conversation.

The words can be perfect and the conversation still fails. And how you say something matters as much as what you say. You can use exactly the same words affectionately or sarcastically, and the appropriate response is completely different. Holding on to tone and context while the language changes mid-sentence is a large part of what makes this interesting to me.

Companies such as OpenAI and Google are investing heavily in real-time multimodal AI. How can a specialized company like Boson AI compete against organizations with significantly greater computational resources, and where do you see the greatest opportunities for differentiation in model architecture, performance, and inference economics?

Having more compute does not automatically mean you are using it well. In a live conversation you can spend a great deal of money running a very powerful model just to process someone saying “hmm,” or “yeah, probably.” That model will use a lot of its intelligence listening to filler. Brains are expensive, they eat a lot of energy, and so does inference. In the end the customer pays for it.

Humans solve this by dividing attention. You can be in a coffee shop having a slightly boring conversation while the back of your mind is working out what to cook for dinner. The front of your mind carries the conversation. Then the other person asks you something genuinely difficult and you are fully immersed, spending everything on it. We build conversational systems the same way.

An inexpensive model handles the front of the conversation, and more expensive reasoning and tool calls come in when the task actually requires them. The other half of it is that we bought our compute rather than renting it. The arithmetic is not complicated: take the hourly cost of a cloud instance, multiply by the hours in a year, and you land close to what the hardware costs to buy outright. We worked out the break-even point component by component and built from there. It was harder than that makes it sound and not everything went perfectly. But it takes brute-force scaling off the table, which means every gain has to come from better data and better architecture. It also turned out to protect us: components we were quoted at under $100,000 cost roughly twice that six months later. We can move quickly because we are a small team working on one thing. A researcher can take an idea, test it, and get it into the product themselves. And we benefit from the open research community and contribute back to it, which lets a small team do rather more than its size would suggest.

Your research on ProactBench explores an important dimension of conversational intelligence: recognizing what users need even when they haven’t explicitly asked for it. How do you envision voice agents evolving from reactive assistants into proactive collaborators, and where should the boundaries lie between anticipating user needs and overstepping?

Let’s say you’re making a doctor’s appointment and early in the conversation you mention that you can’t drive. A good assistant would remember that and suggest a remote appointment or help with transport. You shouldn’t have to repeat that information.

That’s what we’re studying with ProactBench – whether an assistant can use what you’ve told it to anticipate your needs and offer help you haven’t directly asked for.

Of course, you may have already arranged a ride to your appointment. The assistant should offer the option and let you decide, rather than assume it knows your plans. I’d like these systems to save people effort but keep them in control of what happens.

Looking ahead five years, do you believe voice will become the primary interface through which people interact with AI and software? If so, how might that transformation reshape traditional applications, enterprise software, and the role of screens and graphical user interfaces?

I expect we’ll talk to computers much more, often with video as well. When you can see someone looking puzzled, you know you might need to explain something differently. Our audio-video demo shows the beginnings of that kind of interaction, with AI responding to what it sees as well as what it hears.

I still prefer a keyboard for writing code, and if I’m comparing information, I may want it on a screen where I can look through it at my own pace.

For business software, I can imagine starting a task by explaining what I need, then reviewing the result on screen and making changes directly. Over the next five years, I think moving between those ways of working will become much easier. You could talk when it’s convenient and use the screen when you need to be precise.

Thank you for the great interview, readers who wish to explore conversational voice AI and enterprise voice agents should visit Boson AI.