[影片] 在 16 GB MacBookPro 上執行 Nemotron Dory 1.0 4B

This is the 5th episode in my series about local AI, in which I test models that can fit in a 16GB MacBook Pro. Today: Nemotron Dory 1.0, a 4B looped model from NVIDIA. The middle layers can run 1, 2, or 4 times at inference time, which means the model is slightly slower, but has a better output. I tested the 8-bit MLX quantized version, which amounts for around 4 GB, on my 16GB MacBook Pro.

Because the architecture is still very new, there is no native support in MLX Serve yet, so I had to test it it through the community dory-mlx package in the terminal. Surprisingly, that turned out to be useful, because you can watch the model’s internal reasoning in real time as it loops and revises.

On the memory side it behaved surprisingly well. Unlike Ternary Bonsai 2, which forced me to shut almost everything else down, Dory left enough headroom for other apps – didn’t use any, btw, but it was nice to see the space.

Speed wise, it wasn’t spectacular, though. A simple coding task (a word-counting utility that also had to write to disk) took around five minutes of back-and-forth thinking. The final code was sound, though the CLI interface cut the output short — which I believe to be a limitation of the current tooling rather than the model itself.

For pure text generation the results were also solid. The model stayed on topic, structured its answers, and didn’t overcomplicate things. I used my standard “what is a dolphin?” prompt.

All in all, Nemotron Dory 1.0 4B stays in a similar usability range to something like Gemma 4B, with the added advantage of being able to dial the amount of “thinking” up or down.

There’s an open pull request for proper Dory support in MLX-LM. Once that gets integrated, tool use and normal serving should become much smoother, and the output won’t be cut, like in my testing. Until then, the CLI route is usable if you’re patient and just want a small, competent local model that doesn’t eats out all your 16GB memory.

If you’re looking for a model that actually fits and runs on a base MacBook Pro without constant swapping, Dory is one of the more interesting recent additions.