Vision as Unified Multimodal Generation

• URL: http://arxiv.org/abs/2607.06560v1
• Date: Tue, 07 Jul 2026 17:58:33 GMT
• Title: Vision as Unified Multimodal Generation
• Abstract summary: SenseNova-Vision is a computer-vision instruction-response corpus spanning text, image, and mixed targets.<n>The model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation.<n> Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry.
• Score: 73.31482353559736
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest unified multimodal generation as a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.

Related papers

• More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe [52.210953881548996]
  We show that a vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks.<n>Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding.<n>Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks.
  arXiv  Detail & Related papers  (2026-07-17T13:25:44Z)
• FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching [86.31254356971506]
  FlowInOne is a framework that reformulates multimodal generation as a purely visual flow.<n>It unifies text-to-image generation, layout-guided editing, and visual instruction following under one coherent paradigm.<n>It achieves state-of-the-art performance across all unified generation tasks, surpassing both open-source models and competitive commercial systems.
  arXiv  Detail & Related papers  (2026-04-08T07:22:08Z)
• DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning [16.880791276029964]
  "Thinking with images" is a shift in Vision Language Models from text-dominant chain-of-thought to image-interactive reasoning.<n>We present DeepSketcher, a comprehensive suite comprising both an image-text interleaved dataset and a self-contained model.<n>We design a model that performs interleaved image-text reasoning and generates "visual thoughts" by operating directly in the visual embedding space.
  arXiv  Detail & Related papers  (2025-09-30T07:02:01Z)
• Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model [83.85856356798531]
  VistaLLM is a visual system that addresses coarse- and fine-grained vision-language tasks. It employs a gradient-aware adaptive sampling technique to represent binary segmentation masks as sequences. We also introduce a novel task, AttCoSeg, which boosts the model's reasoning and grounding capability over multiple input images.
  arXiv  Detail & Related papers  (2023-12-19T18:53:01Z)
• InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists [66.85125112199898]
  We develop a unified language interface for computer vision tasks that abstracts away task-specific design choices. Our model, dubbed InstructCV, performs competitively compared to other generalist and task-specific vision models.
  arXiv  Detail & Related papers  (2023-09-30T14:26:43Z)
• InstructDiffusion: A Generalist Modeling Interface for Vision Tasks [52.981128371910266]
  We present InstructDiffusion, a framework for aligning computer vision tasks with human instructions. InstructDiffusion could handle a variety of vision tasks, including understanding tasks and generative tasks. It even exhibits the ability to handle unseen tasks and outperforms prior methods on novel datasets.
  arXiv  Detail & Related papers  (2023-09-07T17:56:57Z)
• Unifying (Machine) Vision via Counterfactual World Modeling [5.001446411351483]
  We introduce Counterfactual World Modeling (CWM), a framework for constructing a visual foundation model. CWM has two key components, which resolve the core issues that have hindered application of the foundation model concept to vision. We show that CWM generates high-quality readouts on real-world images and videos for a diversity of tasks.
  arXiv  Detail & Related papers  (2023-06-02T17:45:44Z)
• Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks [39.12025963907317]
  Unified-IO is a model that performs a large variety of AI tasks spanning classical computer vision tasks. We achieve this unification by homogenizing every supported input and output into a sequence of discrete vocabulary tokens. Unified-IO is the first model capable of performing all 7 tasks on the GRIT benchmark.
  arXiv  Detail & Related papers  (2022-06-17T17:53:47Z)
• Contextual Encoder-Decoder Network for Visual Saliency Prediction [42.047816176307066]
  We propose an approach based on a convolutional neural network pre-trained on a large-scale image classification task. We combine the resulting representations with global scene information for accurately predicting visual saliency. Compared to state of the art approaches, the network is based on a lightweight image classification backbone.
  arXiv  Detail & Related papers  (2019-02-18T16:15:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.