EAGLE-4 發佈版本無法在 Mac 運行，作者改測 MTP：Qwen3.8-27B 編碼快 92%，Ornith-1.5-35B-A3B 文字生成慢 19%

A release announcement for EAGLE-4, a speculative decoding method, reported up to 3.4x faster generation for Qwen3.8-27B on four NVIDIA B300 GPUs. I run local models on a Mac Studio, and I wanted to know one thing: does it make my models faster?

The short answer: EAGLE-4 as released does not run on a Mac. But my runtime already has a related technique, MTP (multi-token prediction), and I can measure that. The result depends strongly on the model:

• On a dense 27B model, MTP made code generation 92% faster (thinking off, one request).
• On a 35B mixture-of-experts (MoE) model, MTP made prose generation 19% slower (thinking off, one request).
• With MTP on, the output text changed at temperature 0 in 7 of 8 test cases. I did not measure quality yet.

This post covers speed only. All numbers below come from my own runs, except the EAGLE-4 numbers, which come from the EAGLE-4 README.

What EAGLE-4 is, and why I did not test it

Speculative decoding works like this. A small, fast drafter proposes several next tokens. The large model checks all of them in one forward pass and keeps the tokens that match its own choice. If most drafts are correct, you get several tokens for the cost of about one large-model pass.

EAGLE-4 is a trained drafter for this job. Facts from the official repository (github.com/eagle-hq/eagle-4, package eagle-harness):

• The drafter is EAGLE-Corp/Qwen3.8-27B-EAGLE4: 3B parameters, BF16.
• It runs inside SGLang, vLLM and TokenSpeed (experimental).
• The quick start requires Linux, four NVIDIA GPUs with a CUDA 13 driver and about 70 GB of disk. A separate guide covers AMD Instinct MI355X.
• The README does not mention Metal, MLX, llama.cpp or Apple Silicon.
• The license is source-available, not OSI-approved. Read LICENSE and NOTICE in the repository for the terms.

The README reports these speedups for Qwen3.8-27B at temperature 0. Each value is EAGLE-4 throughput divided by the same engine without speculation:

Two notes. First, the gain falls as concurrency rises, as expected: a fuller batch leaves less idle compute for verification. Second, a news post quoted "3.59x vs vanilla MTP". I did not find that figure in the README. The README baseline is the same engine without speculation, not MTP.

Pointers to related work on Apple Silicon (I did not verify these results):

• An EAGLE-3 prototype for MLX: mlx-lm discussion #890.
• MTPLX (github.com/youssofal/MTPLX) and dflash-mlx (github.com/bstnxbt/dflash-mlx).
• A post on aiweekly.co (aiweekly.co) says MTP makes M1 to M3 chips up to 25% slower. This report is anecdotal, and other reports disagree with it.

I found no EAGLE-4 result on a Mac or on MLX anywhere. So I tested the technique that I can run.

What I tested

Some new models ship with an MTP head: a small extra layer, trained with the model, that predicts tokens further ahead. A runtime can use this head as the drafter. You do not need a separate draft model. My runtime, oMLX, calls this feature "Lightning MTP" and turns it on per model with mtp_enabled.

Hardware and software

• Mac Studio, Apple M2 Ultra (60 GPU cores), 64 GB unified memory, macOS 26.5.
• oMLX 0.7.1rc1, an OpenAI-compatible server for Apple Silicon.

Models

Method

• One model per server process. A fresh server for each phase, in the order OFF, ON, OFF, ON.
• Temperature 0, streaming, one request at a time unless stated.
• Prefix cache off (--no-cache), oMLX memory guard off.
• Two prompts: a Kotlin coding task and a 350-word prose explanation. Each with thinking on and thinking off.
• 10 runs per prompt per phase, 512 max tokens. Warm-up runs to compile kernels are not counted.
• Decode tok/s = completion tokens / (time of last streamed chunk − time of first chunk). This excludes time to first token.
• Prefill = the server's prompt_eval_duration for Python standard-library source code as the prompt: 2,037, 3,913, 7,875 and 16,125 tokens, 5 runs each.
• A sampler recorded GPU use, memory, swap and thermal state every 2 seconds.

Limits of this test

• One machine, one runtime version, community quantizations of each model.
• Temperature 0 only. Sampling at higher temperatures can lower draft acceptance.
• Background processes (a browser, Activity Monitor, a code-indexing tool) used CPU during parts of the runs.
• Qwen MTP-off phases agree within 1.4%. Ornith has more spread: a standard deviation of up to 3.9 tok/s with MTP off (one phase with four slow runs excluded, see below) and up to 10.5 tok/s with MTP on.
• Prefill, long-context and concurrency tests ran in one phase only, with 3 to 5 runs each.
• Swap stayed at or below 100 MB for the whole run, the CPU speed limit stayed at 100%, and the server logged zero "Prefill throttled" lines.

Results

Decode speed

Values are mean tok/s over 20 runs (2 phases × 10 runs). "Parked" counts runs in which oMLX paused MTP and used normal decoding for 128 tokens (log line: "parked for 128 standard tokens"). I did not check the exact rule that oMLX uses to decide this.

* In the second MTP-off phase, four runs dropped to 38 to 112 tok/s. During those runs, the sampler shows a background code-indexing process at high CPU use and GPU use falling well below normal. With medians, this cell is 132.7 off and 111.4 on, a ratio of 0.84x.

Three things stand out:

1. On the dense model, MTP helps on every task but one. The gain is largest on code without thinking.
2. On the MoE model, MTP helps a little on code and hurts on prose.
3. Parking did not protect Ornith. It parked MTP in all 20 Qwen prose-with-thinking runs, which kept that case at +1%. It never parked MTP in Ornith's 512-token runs, even when they were 19% slower.

The control model, Qwen3-4B, ran at 159.2 to 159.9 tok/s in all four scenarios. (It has no thinking mode, and its output was the same with thinking on and off.) Ornith, with 35B total parameters, decodes at 83% of the speed of this 4B dense model. With MTP off, Ornith decodes 3.95x faster than Qwen3.8-27B on a single request.

Prefill

Prefill is the time to read the prompt before the first token. Values are server-side, with MTP off.

With 16,125 prompt tokens, Qwen3.8-27B waits 75.2 seconds before the first token. Ornith waits 11.9 seconds. MTP does not help prefill. With MTP on, prefill was 1.7% to 6.2% slower for Qwen and 2.5% to 3.7% slower for Ornith.

Long context and concurrency

These tests used thinking off. Concurrency requests alternate between the code prompt and the prose prompt. Aggregate tok/s = all completion tokens / wall time, so it includes time to first token.

The 1-request row is a different metric (decode only, the mean of code and prose). I show it only for scale. On the dense model, MTP still helps with four requests in parallel. On Ornith with the long prompt, oMLX parked MTP in all 5 runs, and the result was still 11% slower than MTP off.

Why the dense and MoE models differ

This section is my explanation of the data. The numbers come from oMLX's own MTP log lines. The cause is an inference that I did not prove with a profiler.

Each MTP cycle drafts several tokens and verifies them in one pass of the main model. Two numbers decide the result:

• Tokens per cycle: how many tokens one cycle produces.
• Cycle cost: how long one cycle takes, in units of one normal decode step with MTP off.

The speedup is about tokens per cycle divided by cycle cost. This is accounting, not a model, but it shows which factor matters. Values below come from runs that were not parked.

Qwen prose with thinking on is not in the table, because MTP was parked in every run.

The acceptance rates are similar. Ornith on code accepted 85% of drafts, more than Qwen on code with thinking on (78%). The big difference is cycle cost. A Qwen cycle costs about 2.1 to 2.2 normal steps. An Ornith cycle costs about 2.6 to 2.7. So Ornith needs about 2.6 tokens per cycle only to break even, and on prose it produces about 2.1.

Why is verification more expensive for the MoE model? My explanation: in a dense model, all tokens in the verification pass use the same weights, so one read of the weights from memory serves several tokens. In an MoE model, each token selects 8 of 256 experts, and different tokens select different experts. Verifying four or five tokens therefore reads many more expert weights than one normal step does. Also, a normal Ornith step is short (7.5 ms against 29.8 ms for Qwen), so any fixed cost per cycle is a larger share. In the logs, the main model's pass takes 92% of Ornith's cycle time and 96% of Qwen's. I did not profile the kernels, so treat this as a hypothesis. It can also be a property of the oMLX implementation and not of MoE models in general.

MTP changed the output text

Speculative decoding with greedy verification is supposed to produce exactly the text the model produces alone. In my runs, it did not:

• With MTP off, every scenario produced one identical text across all 20 runs.
• With MTP on, the output matched the MTP-off text in only one scenario: Qwen code with thinking off (20 of 20 runs).
• In the other seven scenarios, 0 of 20 runs matched. The first difference appeared between character 26 and character 423, depending on the scenario, and the texts continued differently after it. The MTP-on texts also differed from each other between runs.

In the samples I read, the differences are changes of wording, not obvious errors. But I did not measure quality, so I cannot say if they matter. One possible cause is small numerical differences between the multi-token verification pass and the normal single-token pass. Such differences can flip a near-tie between two tokens. I did not prove this. For the speed numbers, it means that MTP on and MTP off produced texts of the same length (almost all runs reached the 512-token limit), but not the same texts.

Practical takeaways

These apply to these models on oMLX 0.7.1rc1 and an M2 Ultra:

• Turn MTP on for Qwen3.8-27B on code. It gave +92% with thinking off and +39% with thinking on.
• Leave MTP off for Ornith-1.5-35B-A3B by default. It gave +9% to +12% on code, but −7% to −19% on prose, −11% with a long prompt and −8% with four parallel requests.
• Measure each model. Dense or MoE mattered more for the result than any headline number for the technique.
• For agents, prefill matters as much as decode. The dense model needs 75 seconds to read a 16k-token prompt, and MTP makes prefill slightly slower. With MTP off, Ornith reads the same prompt in 12 seconds.
• Check output quality before you rely on MTP. It changed the text in 7 of 8 scenarios.

What I did not measure

• Quality of any output, with MTP on or off. Ornith's 4x decode speed says nothing about whether its code is as good as Qwen's.
• Temperatures above 0.
• Other runtimes (llama.cpp, mlx-lm directly, LM Studio) and other drafters (EAGLE-3, DFlash).
• Other Apple chips. An M1, M3 or M4 has different bandwidth and compute, and the balance can change.
• Prompts longer than 16k tokens, and real agent sessions with tool calls.
• Energy use.

What comes next

The next test is quality. I will use real commits from my open-source Kotlin Multiplatform app, MyEuroStatApp, as tasks. I will compare what these local models produce, with MTP on and off, against the output of my usual coding-agent workflow on the same commits. I do not have results yet, and I do not know what they will show.

Reproduce it

The scripts are here: github.com/RomanTsisyk/mtp-bench-m2ultra. The repository contains:

• suite.py: the benchmark. It starts one oMLX server per phase, runs the warm-up, decode, prefill, long-context and concurrency tests, and writes summary.json, outputs.jsonl and server.log for each phase.
• sampler.py: records GPU use, memory, swap, CPU speed limit and the top processes every 2 seconds.
• run_all.sh: runs the full plan in the order I used.
• verify_numbers.py: recomputes every number in this post from the raw files.

To repeat the test:

1. Install oMLX 0.7.1rc1 and download the three models from Hugging Face.
2. Close other heavy applications. Background CPU load affected some of my runs.
3. Run ./run_all.sh. The full plan took about 1 hour 43 minutes on my machine.
4. Run python3 -I verify_numbers.py <results folder> and compare its output with the tables above.

If you run it on a different Mac, I want to see your numbers, especially for MoE models on M3 and M4 chips.