如何排序提示詞以加快混合專家模型批次工作

Engineer's view. This is a batch job that touches records in arrival order. Each record needs a different index page, so the cache keeps reloading pages. Sort by key and each page loads once per batch. A mixture-of-experts model loads expert weights the same way, so a mixed batch pays for reads nobody shares.

TL;DR. A mixture-of-experts model reads only the experts the batch asked for, and similar prompts ask for the same ones. Sort an offline job by topic, and each step reads less, so the job finishes sooner.

Explain Like I'm 5

A library has a hundred shelves. A helper runs to fetch books for everyone at one table. If the table holds a cook, a pilot and a poet, the helper runs to many shelves each round. If everyone at the table is a cook, they mostly want the same cooking shelf. The helper makes fewer trips, and the table gets its books sooner. Nobody reads faster. The seating just changed.

The Problem

You have shipped this bug before, in a system with no AI in it.

A nightly job processes a million records in the order they arrived. Each record looks up a row in a big table. Records from the same customer sit on the same index pages, but arrival order scatters them. So the database keeps loading a page, evicting it, and loading it again an hour later. The job is slow, and the code is fine. Someone sorts the input by customer ID, and the job runs twice as fast. Nothing about the work changed. Only the order did.

A mixture-of-experts model has the same bug inside every decode step. Each layer holds many small blocks of weights, called experts. Each token uses only a few of them. Qwen3-30B-A3B has 128 experts per layer, and each token picks 8. But a server runs many requests together as one batch. On each step it must read every expert that any request in the batch picked.

That read is the expensive part. Producing one token per request is limited by how fast the GPU reads weights from memory, not by arithmetic. A random batch of 32 Qwen requests touches about 94 of the 128 experts in a layer. Code prompts favor one set of experts, French prompts another. Mix them, and the step reads both sets.

The fix: before an offline batch job, sort the prompts so similar ones run together. Then each step reads fewer experts. Hugging Face measured this on unchanged vLLM and got 5 to 17% more throughput.

The Fix: Group Similar Prompts Before the Batch Job

New to how a model produces text one token at a time? Start at AI basics → Tokens and sampling.

Why does the order of a list change what the GPU reads?

Because the server fills its batch from the front of the list. vLLM admits waiting requests first come, first served. So neighbors in your list end up decoding side by side. Sort the list, and each batch is mostly one topic.

Topics really do prefer their own experts. In one layer of Qwen3-30B-A3B, a topic's 32 favorite experts took 68 to 80% of its routing decisions. Even routing would give them 25%.

How much does one expert cost?

About 2.9 microseconds per layer on an AMD MI300X. That is the time to read about 9.4 MB of expert weights. The MoE layer's time rose in a straight line with the number of distinct experts the batch touched.

Why was the measured gain smaller than the reads saved?

Sorted Qwen batches read 19 to 25% fewer experts, which predicts a 20 to 27% speedup. The measured gain was about half that. Only the expert layers got cheaper. Attention, sampling and scheduling cost the same as before.

Do I need topic labels?

No. Embed each prompt with a small embedding model, cluster the vectors with k-means, and sort by cluster. On unlabeled chat data, this still gave Qwen about 5% more throughput.

What This Means for You

When this matters. You run offline jobs on a mixture-of-experts model that you host yourself. Think evals, synthetic data, labeling, summarizing a backlog. Qwen3 MoE, gpt-oss, DeepSeek and Mixtral are all this kind of model. The job generates many tokens per prompt, and nobody waits on any single answer.

How it affects you. The win is GPU hours. A ten-hour job on a rented node can finish in about nine. Your outputs will also shift a little. In the Hugging Face run, 92% of greedy outputs matched between sorted and shuffled order. Two shuffled runs matched each other 90% of the time. That drift is ordinary batch nondeterminism, the effect from How the Same Model Gives Two Different Answers. Sorting does not add new drift.

The gain also depends on where your batch size sits. The simulation in code_example.py shows the shape: zero at batch 1, largest at small and middle batches, fading once a one-topic batch reads nearly every expert.

What to do about it. Today, look at your next offline job's input file. If it already has a dataset, task or language column, sort by it before you submit. That costs one line. If it has no labels, add the clustering step below. Then run the A/B test in Implementing It before you trust any percentage, including the one in this article.

Implementing It

The change.

Three roles touch this. Whoever builds the input file groups the prompts. Whoever runs the job submits them in that order and restores the original order. Whoever owns the cluster checks the gain is real. The serving side changes nothing for an offline job.

Role 1, grouping. Use labels when you have them. Otherwise, cluster embeddings. The source used Qwen3-Embedding-0.6B and 16 clusters:

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans

emb = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B").encode(
    prompts, normalize_embeddings=True, batch_size=64)
labels = KMeans(n_clusters=16, n_init="auto", random_state=0).fit_predict(emb)
order = np.argsort(labels, kind="stable")  # stable keeps file order inside a cluster

Keep the embedding model small. It runs once over the file, before the expensive model loads.

You may not be able to sort the whole file at once, for example when you stream it in pieces. Then sort inside each piece. At batch 32, sorting within chunks of 128 prompts gave +11.7%, and chunks of 256 gave +14.8%, against +17.3% for a full sort. Make each piece several batches long, so every batch can fill from one cluster.

Role 2, the batch job. vLLM returns outputs in the order you submitted. So submit in sorted order, then put each output back where its prompt came from:

# before
outs = llm.generate(prompts, params)

# after
outs = llm.generate([prompts[i] for i in order], params)
results = [None] * len(prompts)
for pos, i in enumerate(order):
    results[i] = outs[pos]

Skip the restore step and every downstream join is wrong. You will see it as answers attached to the wrong question, not as an error.

Your batch size is max_num_seqs on the LLM constructor. It decides where you sit on the curve above, so record it with every result.

Role 3, measuring. Run the same prompts in both orders and compare tokens per second. Force equal output lengths so only the order differs:

import time
params = SamplingParams(temperature=0, max_tokens=256, ignore_eos=True)

def tok_per_s(batch):
    t = time.perf_counter()
    outs = llm.generate(batch, params)
    return sum(len(o.outputs[0].token_ids) for o in outs) / (time.perf_counter() - t)

Alternate the runs: shuffled, sorted, shuffled, sorted. A single pair can be skewed by a warm cache. The author's full harness is vllm_bench.py in the sort-your-prompts repository on GitHub. It had no license file when this was written, so read it for the method.

How you know it worked.

The sorted runs should beat the shuffled runs on every repeat, not just on average. On a mixture-of-experts model with labeled topics, expect something near 10%. Then run the same test on a dense model of similar size as a control. It should show no gain, because a dense model reads every weight on every step. If the dense model "improves" too, your measurement is noise. Last, compare outputs. The share that differs between sorted and shuffled should match the share that differs between two shuffled runs. If it is much higher, check the restore step.

When Sorting Prompts Is the Wrong Tool

When someone is waiting. Grouping means holding a request until similar ones arrive. In online serving, that queueing delay costs more than the step time it saves.

When the model is dense. Every weight is read on every step, whatever the order. A dense model like Llama 3 gains nothing.

When the work is mostly prefill. A long prompt with a short answer spends its time reading the prompt. Thousands of prompt tokens touch nearly every expert anyway. A classifier that emits one label per document is this case.

When batches are very large. Once even a one-topic batch touches almost every expert, there is nothing left to save. On unlabeled chat data, gpt-oss-120b at batch 128 showed no measurable gain.

When the model has few experts. A coarse model with 32 experts per layer saved 8 to 11% of reads, against 19 to 25% for 128 experts.

Before you adopt it, ask:

Is my model a mixture of experts, and is this job decode-heavy?
What batch size does my job really run at?
Did a dense control show zero gain on the same A/B test?

Glossary

experts — small blocks of weights in a layer; each token is routed to a few of them.
mixture-of-experts — a model whose layers hold many experts and use only a few per token.
decode step — one round in which every request in the batch produces its next token.
batch — the set of requests the server runs together on each step.
k-means — a clustering method that groups vectors around a chosen number of centers.
batch nondeterminism — small output changes caused only by which requests shared a batch.