From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models

AK's Picks Multimodal Intelligence VLM PEFT Other MMI GenAI Diffusion CV SST Monocular or Binocular Depth Estimation and Occlusion Handling
2026年07月07日
大规模文本到图像模型之所以成为密集预测任务的理想骨干网络，是因为其RGB图像生成预训练过程能够学习到丰富且多层次的语义、结构与几何先验知识。当前主流的生成式与编辑式方法均通过将密集预测任务重构为目标生成任务来复用这些先验：诸如深度图、法线图、Alpha抠图、掩膜（mask）及热力图等标注信息，首先被编码至面向RGB图像训练的变分自编码器（VAE）潜在空间中，再解码为类图像形式的目标输出。然而我们认为，这种范式沿用了远超密集预测实际需求的生成式输出接口：与RGB图像合成不同，密集预测任务要求的是在原始图像平面上逐像素精确、且与具体任务天然适配的特征场（field），而非需重新渲染的新RGB内容。我们的核心发现是：一个预训练完成的DiT（Diffusion Transformer）模型，已天然地在图像平面上构建起“图像块→标记（token）→图像块”的映射格网（lattice）结构；因此，每个标记固定对应图像平面上一个特定输出图像块，其通道维度可直接承载任务原生的物理量（如深度值、法向量分量等），而无需拘泥于RGB外观表示。我们据此提出ReChannel方法：保留VAE编码器以维持DiT输入分布的一致性，但完全移除目标侧的解码器；对冻结的DiT主干网络仅采用任务专用的LoRA（Low-Rank Adaptation）进行轻量适配；并借助一个共享的、仅作用于标记局部的线性投影头（token-local linear head），将每个标记直接映射至其对应的p×p×Kₜ像素空间图像块——该投影头参数量仅约33K，且不引入任何空间混合（spatial mixing）操作。我们在FLUX-Klein模型上开展实验，覆盖六类密集预测任务及十余项基准评测。这一极简接口在无三元图（trimap-free）图像抠图、KITTI深度估计以及指代分割（referring segmentation）三项任务上刷新了当前最优性能；同时在法线估计、显著性检测与姿态估计等任务上亦保持强劲竞争力。在参数量严格对齐的40亿（4B）规模设定下，ReChannel不仅精度更高，推理速度更是同类“编辑+潜在空间解码”方案的2.48倍——这表明，密集感知任务完全可以受益于生成式预训练所蕴含的丰富先验知识，却无需承袭其原本面向RGB合成的输出接口设计。
Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets. We argue this inherits more of the generative output interface than dense prediction requires: unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered. Our key observation is that a pretrained DiT already organizes RGB inputs through a patch-to-token-to-patch lattice on the image plane, so each token indexes a fixed output patch whose channels can carry task-native quantities instead of RGB appearance. We instantiate this as ReChannel: we keep the VAE encoder for the DiT's input distribution but drop the target-side decoder, adapt the frozen DiT with task LoRA, and map each token to its p x p x K_t pixel-space patch through a shared token-local linear head--about 33K parameters, no spatial mixing. Using FLUX-Klein, we evaluate on six dense prediction tasks and over a dozen benchmarks. This minimal interface sets new state-of-the-art on trimap-free matting, KITTI depth, and referring segmentation, and stays competitive on normals, saliency, and pose. In a matched 4B setting it is more accurate and 2.48x faster than an edit-plus-latent-decode counterpart--dense perception can benefit from generative pretraining without inheriting its output interface.
许愿