我重新測試 MoE 卸載基準：部分結論改變

A follow-up to my original 16GB GPU experiment, this time with consistent quantization, repeated benchmarks, and a few unexpected results.

I recently published an article about experimenting with Mixture-of-Experts (MoE) offloading on my laptop.

I tested five different models, compared traditional layer offloading against selective expert offloading, and came away with some pretty interesting results.

Some models performed better with selective offloading. Others were considerably faster using traditional layer splitting.

And one model, LFM2.5, performed so badly with selective offloading that I really wanted to figure out what was going on.

Well, the article got some great feedback.

People pointed out things I hadn't properly accounted for, questioned some of my assumptions, and suggested better ways to test what I was seeing.

Which, honestly, is exactly the kind of feedback I love getting.

So naturally, I went right back down the rabbit hole.

And this time, some of the results were very different.

First, what needed fixing?

If you haven't read the original article, you can find it here:

MoE Offloading Isn't Always Faster: I Tested 5 Models on a 16GB GPU

In that experiment, I was testing whether selective MoE offloading could improve performance when running models that were too large to fit entirely in my GPU's VRAM.

The idea is to keep certain operations running on the GPU while offloading selected expert weights and their computation to the CPU.

I used llama.cpp with the --n-cpu-moe option and compared it against traditional layer splitting using -ngl.

But there were a couple of problems with my original testing.

The models weren't all using the same quantization.

I had Phi-3.5 running at Q4_K_M, Apex at Q5_K_M, Qwen at Q6_K, and Gemma and LFM at Q8_0.

That made it difficult to know whether the performance differences were related to the model architectures, quantization, or something else.

A couple of people brought this up in the comments, particularly regarding LFM's terrible performance.

I also wasn't doing enough repeated measurements.

Another person pointed out that laptop thermals, background activity, and ordinary measurement noise could influence the results. He suggested interleaving the configurations rather than running one strategy completely before moving to the other.

That was a really good point.

My original benchmarks were useful for spotting interesting behavior, but some of the smaller differences weren't necessarily meaningful.

So I decided to do another round of testing.

The new test setup

I'm still using the same laptop:

• GPU: NVIDIA RTX 4090 Laptop GPU, 16GB VRAM

• CPU: Intel Core i9-14900HX, 24 cores / 32 threads

• RAM: 32GB DDR5

• OS: CachyOS / Arch Linux

• Inference engine: llama.cpp with CUDA and Flash Attention enabled

But I changed several things about the testing process.

All five models were now running at Q4_K_M.

I also updated llama.cpp, adjusted the offloading configurations to accommodate the smaller quantized models, and kept enough free VRAM to avoid pushing the GPU right to its memory limit.

For each model, I used the same prompt:

"What are 3 practical benefits of using Arch Linux?

Each configuration received a warmup request before measurement.

I then ran three repeated measurements per strategy in an interleaved order, with short cooldowns between runs.

Instead of relying on a single result, I used the median generation speed.

This doesn't make the experiment perfectly controlled. In particular, changing the quantization, offload settings, and llama.cpp build means I can't attribute every difference to quantization alone.

But it does give me a much better picture of how these models behave under a consistent Q4_K_M setup.

The new results

Here's what happened with the initial 100-token Q4_K_M tests:

Well, that was interesting.

In my original tests, traditional splitting was faster on Qwen, LFM, and Gemma.

Now selective offloading was ahead on Qwen and LFM, and Gemma was essentially tied.

Phi-3.5 had also gone in the opposite direction.

But I wasn't quite ready to declare those results final.

Some of the individual runs had pretty significant variation, including one unusually fast Apex result and one unusually slow LFM result.

So I extended the testing to 512-token generations on a few of the models.

And that's where things got even more interesting.

Qwen: Quantization really does seem to matter

This was probably the result I was most interested in investigating.

In my original tests, Qwen3.5-35B at Q6_K performed better using traditional layer splitting.

But Apex-35B, which uses a related 256-expert MoE architecture, performed better with selective offloading at Q5_K_M.

I wondered whether the quantization difference was contributing to that behavior.

The new Q4_K_M results certainly suggested it might be.

So I went a little further.

I tested the same Qwen3.5-35B checkpoint at Q4_K_M and Q6_K, using the same llama.cpp build and longer 512-token generations.

Here's what I found:

At Q6_K, the two strategies were practically tied.

At Q4_K_M, selective MoE offloading was 29.5% faster.

That's a substantial difference.

And unlike the original Apex-versus-Qwen comparison, this time I was comparing different quantizations of the same model, on the same inference engine build.

I still can't say quantization was the only factor, because the layer placement also needed adjusting to fit the different model sizes.

But this is much stronger evidence that the combination of quantization and memory placement has a major influence on selective offloading performance.

It also changed how I think about fine-grained MoE architectures.

Originally, I suspected that models with 256 routed experts might be particularly problematic for CPU offloading.

These results suggest it's not nearly that simple.

A 256-expert model can perform quite well with selective offloading, depending on the configuration.

LFM: The biggest surprise needed another look

If you read my first article, you might remember LFM2.5.

That was the model that managed a rather miserable 2.11 tokens/sec with selective offloading at Q8_0.

Traditional splitting achieved 9.03 tok/s.

More than four times faster.

It was such a dramatic difference that several people commented on it.

Mudassir suggested that the heavier 8-bit expert weights might be putting too much pressure on CPU memory bandwidth.

I suspected something similar, but didn't have the evidence to separate that from the architecture itself.

Well, switching to Q4_K_M produced a massive improvement.

The initial 100-token run reached 35.44 tok/s with selective MoE offloading.

That's approximately 16.8 times the original result.

Traditional splitting also improved considerably, reaching 32.96 tok/s.

But there was a problem.

One of the selective runs dropped all the way to 11.23 tok/s, while the other two were above 35 tok/s.

That's a pretty ridiculous amount of variation.

So I repeated the experiment using longer 512-token generations.

This time, the results looked quite different:

Essentially a tie.

The weird 11.23 tok/s dip didn't reappear in those longer runs, but both configurations were considerably slower than their original 100-token medians.

That raises questions about workload length, sustained performance, and what might be changing during generation.

What I can say is that LFM clearly isn't inherently terrible at selective expert offloading.

Under the new Q4_K_M configuration, it performed reasonably well.

But whether the improvement came primarily from quantization, different layer placement, changes in llama.cpp, or a combination of all three still needs more investigation.

And the longer test also reminded me of something important.

A configuration that looks faster over 100 tokens might not have the same advantage over 512.

That's something I'll definitely keep in mind when benchmarking models in the future.

Then I tried speculative decoding with Gemma 4

Because apparently five models and two offloading strategies weren't enough to keep me entertained.

I also wanted to see whether Multi-Token Prediction (MTP) speculative decoding could improve performance when using selective MoE offloading.

Gemma 4 26B-A4B has a separate MTP drafter that can propose multiple tokens for the main model to verify.

The idea is that generating and verifying several tokens together can sometimes be faster than generating them individually.

In theory, that sounded promising.

In practice, my results were not particularly encouraging.

Without MTP, selective offloading reached:

13.58 tokens/sec

With the MTP drafter enabled:

7.24 tokens/sec

So instead of speeding things up, it reduced throughput by approximately 46.7%.

I wanted to understand why.

During a longer 512-token test, I captured the drafter telemetry:

draft acceptance = 0.11678103 accepted / 882 generatedmean len = 1.35

Only 11.68% of the proposed draft tokens were accepted.

That's a pretty awful acceptance rate.

The longer MTP run also measured just 6.03 tok/s.

Speculative decoding has overhead. If the drafter isn't making predictions that the target model can accept often enough, the extra work can outweigh whatever speedup you hoped to achieve.

The low acceptance rate gives me a much more concrete reason to investigate the slowdown than simply assuming CPU/GPU synchronization was responsible.

I still want to check whether the drafter configuration or some other compatibility issue contributed to the poor acceptance rate.

So I wouldn't conclude from this that speculative decoding and CPU offloading are fundamentally incompatible.

Just that this particular configuration definitely didn't help.

Something else the repeat testing revealed: Noise

One of Riccardo's main suggestions was to account for measurement variation.

And after seeing these results, I'm glad I did.

For example, the three Apex selective offloading runs were:

9.21 tok/s9.70 tok/s25.50 tok/s

That last run was affected by early end-of-sequence generation, making it a poor comparison with the other runs.

LFM also produced that unusually slow 11.23 tok/s result.

And Gemma's traditional offloading runs varied from 8.98 to 16.85 tok/s.

Those are not small differences.

Interleaving the configurations helped make the testing process fairer, but it didn't magically remove every source of variation.

Using medians helped reduce the influence of individual outliers, although three measurements still aren't enough to establish a particularly reliable performance distribution.

The longer tests were especially useful because they showed that some of the apparent advantages became much smaller when the workload changed.

Going forward, I think longer generations and more repeated measurements are going to be standard practice for my experiments.

It's a little more work, but clearly worth it.

So what did I actually learn?

I went into the original experiment wanting to find out whether selective MoE offloading was a better way to run large models on a GPU with limited VRAM.

My first conclusion was that it depended on the model.

That still seems true, but now I'd add quite a bit more to it.

Quantization can change the outcome.

Qwen went from traditional splitting performing slightly better at Q6_K to selective offloading gaining a substantial advantage at Q4_K_M.

Offloading configuration matters.

Two configurations using the same model can place very different amounts of work on the CPU and GPU. The amount of VRAM occupied doesn't tell the whole story.

Short benchmarks can be misleading.

LFM's apparent 7.5% selective advantage in the 100-token tests practically disappeared over the longer generations.

Speculative decoding isn't a free speed boost.

Gemma's MTP drafter had a very low acceptance rate and made generation considerably slower under my tested setup.

And perhaps most importantly, a measurement and an explanation aren't the same thing.

I can measure that something runs faster or slower.

Explaining exactly why sometimes requires considerably more testing.

That distinction is becoming more obvious the deeper I get into this.

What's next?

I'm not quite finished experimenting.

I'd still like to test more quantization levels on the exact same checkpoint, ideally with more carefully controlled offloading configurations.

I'm also interested in monitoring actual CPU memory bandwidth and GPU behavior during inference.

That might help answer some of the questions about where these slowdowns are coming from instead of relying on educated guesses.

And eventually, I still want to repeat some of these tests on a machine with 64GB of RAM.

I'm especially curious how much more flexibility that provides for keeping expert weights in system memory without running into swap pressure.

But for now, I think I've learned quite a bit from these two rounds of testing.

Final thoughts

One of my favorite things about sharing experiments publicly is that sometimes people notice things you missed.

I thought my original results were interesting, and I still do.

But several people asked good questions about the methodology, the quantization differences, and whether the measurements were reliable enough to support some of my conclusions.

They were right to ask.

And when I went back and tested those things, some of my conclusions changed.

I'm glad they did.

I'd much rather discover that my original explanation wasn't quite right than keep repeating it because it sounded reasonable.

That's half the fun of experimenting with this stuff.

You start with one question, find an interesting result, and suddenly you're investigating three completely different things you hadn't even considered.

And occasionally you realize you've spent an entire weekend benchmarking language models instead of doing literally anything else.

You know, perfectly normal behavior.

A big thank you to everyone who took the time to read the original article and offer thoughtful feedback. It genuinely helped make this second round of experiments better.

And if anyone has suggestions for what I should test next, particularly around quantization, CPU memory bandwidth, or speculative decoding, I'd love to hear them.

Apparently I haven't spent enough time staring at tokens-per-second measurements yet.