Vision as Unified Multimodal Generation

AK's Picks Multimodal Intelligence Vision-Language Pre-training Other MMI GenAI TDIG Other GenAI CV ODBO SST
2026年07月07日
我们将计算机视觉建模为统一的多模态生成任务:各类异构的视觉任务均在统一多模态模型原生的文本与图像生成空间内进行表达,无需针对特定任务设计专用网络架构。在此范式下,SenseNova-Vision 通过自然语言指令(并可选配视觉提示)来明确指定任务类型、目标区域或视角,以及解码规范;其输出形式则根据任务需求灵活适配——符号化结果以纯文本形式生成,稠密空间预测以图像形式生成,而复合型任务则输出图文混合结果。为支撑大规模训练,我们将来源多样的计算机视觉标注数据统一转化为适配上述生成空间的“指令–响应”样本,由此构建出 SenseNova-Vision 语料库——一个覆盖文本、图像及图文混合目标的计算机视觉指令–响应语料库。SenseNova-Vision 以现成的预训练统一多模态模型为起点,主要基于该语料库开展训练;同时引入辅助性多模态数据作为能力保持型混合数据源,全程无需增设任何任务专属的预测头,亦不需对模型架构进行任何修改。最终所得模型可覆盖广泛的视觉任务,包括目标检测、光学字符识别(OCR)、关键点估计、语义/实例分割、深度估计、表面法向量预测、点云图(point maps)以及相机位姿估计,并支持由语言定义的多样化变体任务,灵活融合类别、颜色、空间区域及其他视觉线索。实验表明,这一单一统一模型在结构化视觉理解、稠密几何预测、分割任务以及多视角视觉几何等关键领域,性能均可媲美当前领先的专用任务模型。上述结果表明,统一多模态生成是一条可扩展的技术路径,有望将计算机视觉能力无缝集成至通用基础模型之中。本模型及其配套语料库均已开源发布。
We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest unified multimodal generation as a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.
许愿