BYOD:高效將自回歸語言模型轉換為擴散語言模型
BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion
BYOD研究展示將 Gemma 2 9B、Llama 3.1 8B、Qwen2.5 7B 和 Ministral 8B 轉為雙向遮罩擴散語言模型的方法。
BYOD: Build your own diffusion language model in Google Colab
Can an existing instruction model learn to generate text through diffusion in a few hours on one GPU?
In BYOD: Build Your Own DLM, we convert Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B into bidirectional masked-diffusion language models. Every pretrained weight stays frozen: training updates only LoRA adapters, representing 4.1–5.8% of the final model parameters. The four primary conversions each took 3.1–4.0 hours on one GPU in Google Colab.
We want other people to be able to train, inspect, and experiment with these models too. The training notebook and interactive demos are available now.
Train your own model
Open the training notebook in Google Colab · Code and instructions
The notebook has been designed to make it as easy as possible to help you train your own DLM. No coding is needed, you only need to run each code cell:
- Choose a model.
- Use the provided tokenized training data, or configure your own instruction–response dataset.
- Train while watching the loss curve to see if the model keeps improving.
- Open the interactive inference interface via the notebook step 6 and try out your model. You can also stop a run early and still use it in the interface.
Quick mode trains smaller Llama, Gemma, or Qwen models and rank-128 LoRA for initial experiments. Paper mode uses the original 7B–9B models and rank-1024 adapters and needs a paid GPU; our paper runs used a 96 GB NVIDIA RTX PRO 6000.
Try the fine-tuned diffusion models before training your own:
These demos run directly in your browser:
| Model | Interactive demo |
|---|---|
| Gemma 2 9B | Try BYOD-Gemma |
| Llama 3.1 8B | Try BYOD-Llama |
| Qwen2.5 7B | Try BYOD-Qwen |
| Ministral 8B | Try BYOD-Ministral |
Instead of appending one token at a time, the model starts with a masked answer and fills positions through successive denoising steps, with a flexible inference budget. Try the same prompt with different step budgets and see how it influences inference speed and quality. The hosted demos also include experimental decoding options beyond the paper’s evaluation settings.
For related demos, try our earlier LAD model, LLaDA-8B-Instruct, which also enabled the changing of unmasked tokens.
What did we find?
The same conversion recipe works across all four families. All four generate from fully masked answers and use information to the right of a masked position in our controlled attention test.
BYOD-Gemma comes within 1.3 percentage points of LLaDA on our nine-task average: 50.6% versus 51.9%, evaluated in the same harness. The unchanged AR parents still perform better on average, with particularly large conversion gaps in mathematics and code.
Figure 1: Task-level accuracy in the matched evaluation. Performance depends on both the task and the converted model family.
Generation at a low number of inference steps: At 64 denoising steps for 128 output tokens, externally scored perplexity is 8.3–11.1 for BYOD, versus 4.1 for LLaDA and 2.2–3.0 for the AR parents. Increasing the denoising budget improves the results, but our low-cost conversion still leaves a quality gap.
Figure 3: Generation quality across inference budgets. Lower perplexity and higher Distinct-1 are better; crosses mark the AR parents.
We hope BYOD makes diffusion-model experiments easier to start: change a backbone, try different training data, inspect a denoising trajectory, and see what improves.
Start with the Colab notebook · Read the paper · Get the code
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co