BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion

AK's Picks LLM LoRA/Adapter Other LLM GenAI Diffusion
与自回归(AR)模型不同,扩散语言模型(DLM)支持词元的并行更新、灵活的生成方式以及双向注意力机制;但当前主流系统往往依赖高昂的预训练开销或对整个模型进行全参数微调。本文提出“构建你自己的DLM”(Build Your Own DLM,简称BYOD)——一种仅基于LoRA(低秩适配器)的通用化方法,可高效地将AR模型转化为DLM,且仅需极少的可训练参数。无需针对特定架构进行调优,BYOD即可成功在Gemma 2 9B、Llama 3.1 8B、Qwen2.5 7B及Ministral 8B四种模型上诱导出扩散式生成能力;据我们所知,这是首次报道将Mistral系列模型(此处为Ministral 8B)从AR范式成功转换为DLM的工作。所有适配器参数总量仅占最终模型总参数量的4.1%–5.8%,而基础模型的所有原始参数均保持完全冻结;每次转换仅需最多25,000轮迭代,处理至多1.024亿个训练token,在单块Google Colab GPU上耗时仅为3.1–4.0小时。上述四个经BYOD转换后的模型均能从完全掩码的初始序列出发完成生成,在受控的注意力机制测试中展现出对双向上下文的有效利用,并在很大程度上保留了其原始AR母模型的知识能力。在涵盖九项任务的统一评估框架下,BYOD-Gemma展现出最强的整体性能,平均准确率达50.6%,略低于同评估环境下LLaDA 8B Instruct的51.9%。然而,当前方法的主要局限在于:当网络函数求值次数(NFE)预算较低时,开放生成质量仍显不足——例如,在64步内生成128个token的设定下,BYOD各模型的困惑度(perplexity)为8.3–11.1,而LLaDA为4.1,其对应的原始AR母模型则仅为2.2–3.0。为支持后续研究与深入分析,我们开源了一套独立完整的训练与评估代码实现,并为全部四种转换模型提供了交互式演示界面。综上,BYOD不仅为将扩散生成能力拓展至其他现有大模型家族提供了一条切实可行的技术路径,同时也清晰揭示了高效AR-to-DLM转换下一步亟需填补的关键能力缺口。
Unlike autoregressive (AR) models, diffusion language models (DLMs) permit parallel token updates, flexible generation, and bidirectional attention, but leading systems rely on costly pretraining or full-model fine-tuning. We present Build Your Own DLM (BYOD), a generalizable LoRA-only recipe for parameter-efficient AR-to-DLM conversion. Without architecture-specific tuning, BYOD induces diffusion generation in Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B; to our knowledge, this is the first reported AR-to-DLM conversion of a Mistral model. The adapters comprise 4.1-5.8% of final parameters while every base-model parameter remains frozen; each 25,000-iteration conversion needs at most 102.4M tokens, processed in 3.1-4.0 hours on one Google Colab GPU. All four models generate from fully masked sequences, use bidirectional context in a controlled attention test, and retain substantial parent-model knowledge. BYOD-Gemma is the strongest BYOD conversion on our nine-task evaluation, reaching 50.6% average accuracy compared with 51.9% for LLaDA 8B Instruct in the same harness. Open-generation quality at low network function evaluation (NFE) budgets remains the principal limitation: BYOD perplexity is 8.3-11.1 versus 4.1 for LLaDA and 2.2-3.0 for the AR parents when generating 128 tokens in 64 steps. To enable further experimentation and inspection, we provide a standalone training and evaluation implementation and interactive demos for all four conversions. BYOD thus provides a practical route for extending diffusion generation to other existing model families while identifying the gap that efficient conversion must next close.
许愿