GEAR: Guided End-to-End AutoRegression for Image Synthesis

Bin Lin ,

Zheyuan Liu ,

Chenguo Lin ,

Sixiang Chen ,

Yunyang Ge ,

Yunlong Lin ,

Jianwei Zhang ,

Miles Yang ,

Zhao Zhong ,

Liefeng Bo ,

Li Yuan

AK's Picks GenAI Diffusion TDIG SCCE

2026年06月30日

视觉生成模型通常采用两阶段方式进行训练：首先训练一个用于重建任务的分词器（tokenizer），随后将其冻结；接着，在该分词器输出的离散索引或连续隐变量上训练生成器。这种解耦式训练方式导致分词器无法感知生成器在建模过程中所面临的难易程度。为此，我们提出了GEAR（引导式端到端自回归模型，Guided End-to-end AutoRegression），它通过表征对齐（representation alignment）的引导，实现向量量化（VQ）分词器与自回归（AR）生成器的联合、端到端训练。其核心难点在于：输入至AR模型的VQ索引是非可微的，因而反向传播梯度无法回传至分词器，而常用的直通估计器（straight-through estimator）又容易导致训练崩溃。GEAR通过代码本（codebook）分配的双重读出机制克服了这一难题：一方面，采用硬性的一热（one-hot）分支，以预测下一个token的方式训练AR模型；另一方面，采用可微的软性分支，承载表征对齐损失，并将该损失反向传播，仅用于指导分词器的优化。由此，AR模型能够主动引导其分词器朝向一种更易于自身预测的索引分布进行调整。这一机制将表征对齐的负担从分词器转移至AR模型：分词器自身提取的特征逐渐减弱其与DINOv2特征的相似性，而AR模型输出的特征则相应增强这种相似性——这与扩散模型中强调让潜在表示本身具备语义性的主流做法恰好相反。相较于强基线模型LlamaGen-REPA，GEAR将ImageNet数据集上的生成FID（gFID）收敛速度最高提升达10倍；同时，它能学习到显著更优的图像块级（patch-level）特征与空间一致性更强的特征；此外，该方法还展现出良好的泛化能力，可适配多种量化器（如VQVAE、LFQ、IBQ），并成功迁移应用于文生图（text-to-image）生成任务。

Visual generative models are typically trained in two stages. A tokenizer is first trained for reconstruction and then frozen, after which a generator is trained on its discrete indices or continuous latents. This decoupling leaves the tokenizer unaware of what the generator finds easy to model. We present GEAR (Guided End-to-end AutoRegression), which trains a vector-quantized (VQ) tokenizer and an autoregressive (AR) generator jointly and end-to-end, guided by representation alignment. The key obstacle is that the VQ index fed to the AR model is non-differentiable, so gradients cannot reach the tokenizer, and a straight-through estimator collapses. GEAR resolves this with a dual read-out of the codebook assignment. A hard, one-hot branch trains the AR with next-token prediction, while a differentiable soft branch carries a representation-alignment loss that flows back to guide only the tokenizer. The AR model thereby steers its tokenizer toward an index distribution it can predict more easily. This shifts the alignment burden from the tokenizer to the AR: the tokenizer's own features become less DINOv2-like while the AR's become more so, the opposite of diffusion-side recipes that make the latent itself semantic. GEAR speeds up ImageNet gFID convergence by up to 10x relative to the strong LlamaGen-REPA baseline, learns markedly better patch-level and spatially-coherent features, and generalizes across quantizers (VQVAE, LFQ, IBQ) and to text-to-image generation.

许愿