Unity 開發經驗帶給語音智能體開發的五種可沿用習慣與三項未能沿用之處

I spent five-plus years in Unity before I built my first AI voice agent. Freelance game work through university, then AR for fire response at ARCortex, clinical XR on a Vive Focus 3 at Nystag, a cross-platform C# SDK at Geonode, and a sports-tech platform as CTO at RAQTS.

When I started taking voice agent work, the assumption from every direction was that I was starting over. Unity was a charming line on a CV, not preparation.

That turned out to be wrong in a specific way. Very little of what I use now is knowledge about Unity. Almost all of it is knowledge about building things that run in real time, in front of a human who will not wait, on top of sensors that lie. That describes an AR headset on a fire ground, and it describes a phone call. The phone call turned out to be the easier of the two.

Here is the honest split: five habits that carried over almost intact, and three that did not carry over at all. The second list is where the expensive lessons were.

What transferred

1. The frame budget is the latency budget

Unity teaches you that time is a hard resource before it teaches you anything else. At 60 fps you have about 16 ms per frame. On a standalone VR headset it is closer to 11 ms, which I wrote about in Standalone VR Gives You 11 Milliseconds a Frame.

The discipline that produces is not "make it fast". It is knowing exactly what sits on the critical path and what can be pushed off it. You profile instead of guessing, you move work off the main thread, you spread expensive work across frames, and you stop trusting averages, because the player only ever feels the worst frame.

A production voice agent needs exactly those habits. A caller notices the slowest turn, not the median one. The work has the same shape: find what actually blocks the response, move CRM writes and notifications out of the call into automation that runs afterwards, cover an unavoidable wait with speech instead of silence, and measure the tail. I broke that down for phone calls in A Faster Model Will Not Fix Your Slow Voice Agent. Reading it back, it is the frame budget in different units. 16 ms became roughly 800 ms. The psychology is the same: past a threshold, the thing stops feeling alive.

AR made this sharper. On the ERIS XR platform at ARCortex, a label that lags the world is not a slightly late label. It is wrong about where something is.

2. Everything real time is a state machine, and the bugs live in the transitions

Game developers live in state machines: character controllers, UI flows, animation graphs. Experience teaches you that bugs almost never live inside a state. They live in the transitions, especially the ones nobody wrote down. The player who opens the inventory mid-jump while a cutscene loads is the one who finds the gap.

Designing a conversation flow is the same job. The nodes are easy. What decides whether an agent survives a real caller is the exit conditions. What counts as done? What happens when the caller answers a question you did not ask? Where do they go when the whole thing falls apart?

I now write the exits before the happy path, which is a habit I took straight from character controllers. It is the core of Your Voice Agent Needs Three Exits, Not One.

3. Sensors lie, so design around the lie

I did not expect this one to carry over so directly. Location-based AR runs on GPS and a magnetometer, and both are wrong in ways that are neither random nor constant. In geospatial AR, heading error, not position error, was usually what broke the illusion. At Nystag the whole product was a measurement taken from an imperfect eye-tracking signal that a clinician would later base a judgement on.

Speech recognition is the same kind of input. It is a noisy sensor with structured failure modes, and you can learn the structure. It fails worst on the values that carry no redundancy: names, postcodes, registration numbers, email addresses. Whole sentences survive. (I compared the lecture and phone-call versions of this problem in Transcribing a Lecture and Transcribing a Caller Are Not the Same Problem.)

Once you accept that every reading comes with an implied confidence, you build differently:

• validate the shape downstream instead of trusting the raw value
• read back the values that matter, in small chunks
• store confidence alongside the value
• give the repair path an end, so it cannot loop forever

AR taught me not to trust a number just because a device produced it. A transcript is a number a device produced.

4. Playtesting is adversarial QA

Anyone who has watched a stranger play their game for the first time has learned something no test suite tells you: users do not follow the path. They walk into the wall, mash every button during the tutorial, and answer the question you did not ask.

Callers are players. They interrupt, give you two facts at once, go quiet, and ask something completely outside the agent's job in the middle of a booking. Game work builds the instinct to test with the difficult user, not the cooperative one. That is why the pipeline I built at Fortell AI, described in How I use Claude Code and Comet to build and test AI voice agents in a day, aims at difficult calls instead of clean ones. An agent that handles the ideal caller proves nothing. So does a game that only survives its own developer playing it.

5. Shipping through a gate that does not care about you

Store releases teach a particular kind of production discipline. There is a submission, a review, a rejection, a build that has to be reproducible, and a rollback plan you should have thought about before you needed one. Shipping the Unity Lumo platform and the mobile apps through the App Store and Play Console at RAQTS, and building the Unity and Windows CI/CD around the Repocket SDK at Geonode, taught me that "it works on my machine" describes where it works. It is not a defence.

Voice agents have gatekeepers too, and they are less forgiving than they look: carriers, A2P registration, recording-consent rules, healthcare and privacy constraints. The rules are different and the posture is the same. Something outside your control can block you from reaching users, so you plan for it in week one, not the day before launch.

What did not transfer

You cannot replay a phone call

Games are deterministic enough to re-run. Same seed, same inputs, same bug, as many times as you need. That was the biggest thing I lost. A live call cannot be repeated: a different caller, different background noise, and a model that will not produce the same output twice. You cannot step through it.

The fix was to build observability up front, because there is no debugger: transcripts, recordings, structured post-call data, the full tool-call trace. You are doing forensics after the fact, not stepping frames, so the job becomes capturing enough evidence at the time to reconstruct what happened later. I went deeper on that in There Is No Repro for a Phone Call.

You can see a game. You cannot see an agent.

Almost every visual bug in Unity announces itself. You look at the scene and something is in the wrong place.

Voice agent failures are silent, and they look like success. The call completes, the transcript reads fine, and the customer is never contacted because the email address was captured wrong. Nothing errors. I had to deliberately replace looking with reading, and to build the habit of reading failed calls from start to finish instead of sampling random ones.

The engine is yours. The model is not.

Unity is a dependency you can reason about, decompile, work around, and eventually understand completely. A language model is a dependency whose behaviour changes without a changelog, and you cannot patch it.

You do not fix it, you route around it. Keep the agent thin. Put anything that must be correct into deterministic automation downstream instead of trusting the model to remember it. Limit what it is allowed to say wherever the wording carries risk. That architectural caution is the biggest real change in how I build, and none of it came from games.

Should a Unity developer make this switch?

If you want to, yes, and faster than you think. The parts of AI engineering that take people longest are not the prompts. They are latency intuition, state design, tolerance for unreliable input, adversarial testing, and release discipline. A game developer already has all five. What you are missing is a stack, and a stack takes a few weeks.

What I would not do is treat it as a clean break. When I get hired for agent work, the reason is rarely the model layer. That is the commodity part. It is that the system around the model holds up, and that came from years of building things that had to run at speed in front of a person. It is the same conclusion I reached in Four ICPC Habits That Survived Production: the tools changed four times, and the underlying muscle never did.

If you made a jump like this, from games, embedded, or anything else real time, into AI work: which habit carried over for you, and which one did you have to drop?

Originally published on nabeelbaghoor.com.