NVIDIA Nemotron 3.5 Bolt E4B：在 CPU 上運行離線語音智能體並調用工具

NVIDIA released Nemotron 3.5 Bolt E4B on October 23. The pitch is a small model tuned for agents: it calls tools, follows instructions, and is small enough for laptops, Jetsons and other edge hardware. The model card's benchmarks are tested on H100s, RTX 5090s and DGX Spark. I wanted to see what happens without any of those.

So I took the official 4-bit GGUF, ran it on a plain 8-vCPU Xeon VM with 16 GB of RAM and no GPU, and built a small voice assistant around it. It listens to a WAV file, picks a tool, calls it, and speaks the answer back. Everything below is from my own runs. I've kept the raw CSVs for every number.

What Bolt E4B actually is

A few facts from the model card and the files themselves, because the name hides some of them:

• Architecture: Gemma 4 E4B, post-trained by NVIDIA with reinforcement learning (GRPO via NeMo RL) for tool use, instruction following and multi-turn chat. About 8B total parameters and about 4B "effective" ones.
• License: OpenMDW-1.1 on NVIDIA's release. The base Gemma 4 weights are Apache 2.0.
• Context: 64K tokens. Text in, text out. There's no audio in the model, so a voice agent still needs separate speech-to-text and text-to-speech.
• Weights: the official repo is nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB, with a single Q4_K_M GGUF. NVIDIA also ships a BF16 checkpoint for vLLM and a DFlash speculative-decoding drafter aimed at GPUs.
• Recommended serving: llama.cpp with --jinja, sampling at temperature 1.0 and top_p 0.95, and a chat-template switch for "thinking" on or off.

The first surprise: the "4GB" file is 6.18 GB on disk. I opened the GGUF and summed the tensors:

The per-layer embeddings are lookup tables: each token only reads a few rows. That's what makes the "4GB" label fair at run time, as you'll see in the memory section. You still need about 6.2 GB of disk space and download time, though.

The setup

• Hardware: 8 vCPU Intel Xeon (AVX-512, VNNI), 16 GB RAM, no swap, no GPU.
• Runtime: llama.cpp CPU build b11913, using llama-server as an OpenAI-compatible endpoint.
• Speech-to-text: faster-whisper, base.en and small.en, int8 on CPU.
• Text-to-speech: Piper with the en_US-lessac-medium voice.

The server command was nothing special:

llama-server -m Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf --alias nemotron-bolt \
  --host 127.0.0.1 --port 8000 -c 8192 -np 1 -t 8 --jinja --metrics

The model loaded in 2.5 to 6 seconds. My first sanity check was 17 × 23, and it answered 391.

Raw speed on 8 CPU cores

llama-bench, 8 threads, 3 repetitions:

About 11 tokens per second of generation is slow next to a GPU, but it's enough for an assistant whose replies are one or two sentences. Prefill speed matters more than I expected. My system prompt plus five tool schemas came to about 500 tokens, and the very first request took 6.4 seconds before anything came back. After that, llama.cpp's prompt cache kept the prefix warm, and only the new user message needed processing.

Test 1: tool calling, 20 prompts × 3 runs

I defined five mock tools, the kind a voice assistant would have:

• get_weather(city)
• calculate(expression)
• create_event(title, date, time, duration_min)
• set_timer(minutes, label)
• control_light(room, state, brightness)

The system prompt gave the date (Tuesday 2026-10-27, IST) and asked for short spoken replies. I wrote 20 prompts: three per tool, plus five where the right move is not to call any tool. Those were a joke, a definition, a "thanks", a trivia question, and "order me a pizza", which no tool can do.

Each response was scored on two things: did it pick the right tool (or correctly pick none), and were the arguments exactly right?

Thinking off (the mode I'd use for voice), three full runs:

Some results I liked:

• "Set a timer for 45 seconds" became minutes: 0.75.
• "An hour and a half" became duration_min: 90.
• "Dim the bedroom to 30%" became state: "on", brightness: 30. It didn't treat "dim" as "off".
• "18% GST on 4999" became 4999 * 0.18.
• None of the five no-tool prompts triggered a tool. For the pizza request it said plainly that it couldn't order food, rather than forcing the request into create_event.

The miss was the same in all three runs. "Put sprint planning on my calendar for Friday at 11 for an hour" got the right title, time and duration, but the date was 2026-11-06, a week too late. With today being Tuesday the 27th, "Friday" should be the 30th. Relative-date math is where a 4B model shows its size.

Thinking on, one run of the same 20 prompts:

With reasoning on, it got Friday right (2026-10-30). That cost about 350 characters of reasoning and 17.6 seconds for that one prompt. The first token still arrives fast, but it's a reasoning token rather than an answer. Overall that's roughly 5x slower per turn for one extra correct argument out of 20. For a voice assistant on CPU, I'd keep thinking off and handle dates in code: pass the model a resolved calendar ("Friday = 2026-10-30") in the prompt, or let the tool parse "Friday" itself.

One llama.cpp detail if you're measuring this yourself: tool-call output streams in late, once the call has been parsed. So TTFT for a tool call is close to the time the whole call is ready. For plain-text replies, the first token showed up in 0.36 to 0.6 seconds.

Test 2: a full offline voice turn

Next I wired up the whole loop:

1. Piper speaks the user's request into a WAV file, as a stand-in for a microphone.
2. faster-whisper transcribes it.
3. Bolt picks a tool (thinking off).
4. The mock tool runs.
5. Bolt turns the tool result into a one-sentence reply.
6. Piper speaks the reply into another WAV.

Six utterances, timed stage by stage. Medians:

The LLM is nearly all of the turn time. Piper is basically free: it made a 4-second reply in 70 to 100 ms. Whisper is cheap too. That means most of the latency budget goes to two model calls: one to pick the tool and one to phrase the answer.

What the transcripts showed:

• base.en heard "Pune" as "Pooner". Bolt still called get_weather(city="Pune") correctly.
• base.en turned "Tell me a one-line joke about compilers" into "Tell me what you want to do." Bolt sensibly listed its abilities, but the joke never happened. small.en got the sentence right, and Bolt told the joke.
• "Book a dentist appointment tomorrow at four thirty p m" became date: 2026-10-28, time: 16:30, which is correct.
• The 18%-of-4,999 answer was "about 900" in one run and "899.82" in another. At temperature 1.0 the wording varies, so if exact numbers matter, put them in your reply template.

My takeaway: small.en is worth its extra half second, because recognition errors cost far more than ASR latency. Since the model handles fuzzy input well, I'd spend the speed budget on the ASR side.

Memory: where the "4GB" name comes from

On Linux, llama.cpp memory-maps the GGUF, so top can show a scary number. Right after load, RSS was about 9.6 GB, because the file pages had just been read. Once the system had been running for a while, the server process looked like this:

So about 4 GB of RAM for the model, plus about 0.7 GB for speech. The large embedding tables mostly stay on disk until a token needs a row. That matches the name, and it's a key reason this model fits small devices. Just plan for the 6.2 GB download and the disk space.

Limitations, honestly

• CPU only. I didn't test the DFlash speculative drafter or any GPU, so these numbers are a floor, not a ceiling. NVIDIA's GPU numbers will look very different.
• Small test set. 20 prompts × 3 runs is a smoke test, not BFCL. For a fuller picture, NVIDIA reports 67.3 on BFCL-v3 and 74.1 on ToolSandbox for this Q4_K_M build.
• Single tool call per turn. I didn't test parallel or chained tool calls.
• Synthetic audio. My "microphone" was Piper's own voice, which is clean speech. Real rooms, accents and background noise will make Whisper's job harder.
• Dates. With thinking off, it misread "Friday" in every run. Resolve dates in code.
• Variance. My first run happened while other jobs were finishing, and its median turn was 3.6 s against 2.1 s in the clean third run. Run your own benchmarks more than once.

Would I use it?

For a local assistant that turns speech into a handful of well-defined actions (timers, lights, calendar, lookups), yes. On a CPU-only VM it picked the right tool 60 out of 60 times, filled the arguments correctly 95% of the time, and completed a whole spoken turn in about four and a half seconds, with no network calls and about 4.7 GB of RAM including speech.

Where I'd be careful: anything that needs arithmetic on dates or times, long open-ended answers at 11 tokens per second, and noisy audio with a tiny Whisper model. Each of those has a simple fix: resolve dates in code, keep replies short, and use small.en.

Next I want to run the same loop on a Jetson-class board with the DFlash drafter and see how far that 4.5 seconds drops.

Originally published on Medium.