Let RGB Be the Language of Vision

• URL: http://arxiv.org/abs/2607.12450v1
• Date: Tue, 14 Jul 2026 07:25:58 GMT
• Title: Let RGB Be the Language of Vision
• Abstract summary: This work introduces a unified formulation for vision models, where diverse forms of visual information beyond natural images are represented as RGB images.<n>General visual tasks can be converted into a common RGB-to-RGB image editing problem.<n>Built upon a generic image editing backbone without task-specific fine-tuning, RINO demonstrates robust and competitive zero-shot performance.
• Score: 62.25975815664782
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work introduces a unified formulation for vision models, where diverse forms of visual information beyond natural images, such as masks, depth maps, and other structured visual signals, are all represented as RGB images, while general visual tasks can be converted into a common RGB-to-RGB image editing problem. In this paradigm, different types of visual information internally share the same encoding and decoding architecture and parameters as natural images, enabling a single model to transfer across tasks through a unified visual interface, in a way analogous to how language models operate over text. We refer to this formulation as RGB In and RGB Out (RINO). Built upon a generic image editing backbone without task-specific fine-tuning, RINO demonstrates robust and competitive zero-shot performance on both dense understanding tasks such as segmentation and depth estimation (where we unify outputs as RGB), and dense-conditioned generation tasks such as pose-to-image generation (where we unify inputs as RGB). We hope this study provides useful insights toward general unified vision-language systems, where diverse visual tasks can be expressed, interpreted, and solved through a shared visual language. Code is available at https://github.com/yangtiming/RINO.

Related papers

• Vision as Unified Multimodal Generation [73.31482353559736]
  SenseNova-Vision is a computer-vision instruction-response corpus spanning text, image, and mixed targets.<n>The model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation.<n> Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry.
  arXiv  Detail & Related papers  (2026-07-07T17:58:33Z)
• UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation [51.31795451147935]
  We present a unified generative model that supports visual understanding and visual generation within a single pixel-to-pixel diffusion framework.<n>Our goal is to achieve unification along three axes: the model, the tasks, and the representations.<n> Experiments on text-to-image synthesis and image-to-text understanding demonstrate strong cross-modal alignment.
  arXiv  Detail & Related papers  (2025-11-21T03:02:10Z)
• HDBFormer: Efficient RGB-D Semantic Segmentation with A Heterogeneous Dual-Branch Framework [0.0]
  In RGB-D semantic segmentation for indoor scenes, a key challenge is effectively integrating the rich color information from RGB images with the spatial distance information from depth images.<n>We propose a novel heterogeneous dual-branch framework called HDBFormer, specifically designed to handle these modality differences.<n>For RGB images, which contain rich detail, we employ both a basic and detail encoder to extract local and global features.<n>For the simpler depth images, we propose LDFormer, a lightweight hierarchical encoder that efficiently extracts depth features with fewer parameters.
  arXiv  Detail & Related papers  (2025-04-18T09:29:46Z)
• Diffusion-based RGB-D Semantic Segmentation with Deformable Attention Transformer [11.648973329789973]
  We introduce a diffusion-based framework to address the RGB-D semantic segmentation problem.<n>We demonstrate that utilizing a Deformable Attention Transformer as the encoder to extract features from depth images effectively captures the characteristics of invalid regions in depth measurements.
  arXiv  Detail & Related papers  (2024-09-23T15:23:01Z)
• SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval [82.51117533271517]
  Previous works typically only encode RGB videos to obtain high-level semantic features. Existing RGB-based sign retrieval works suffer from the huge memory cost of dense visual data embedding in end-to-end training. We propose a novel sign language representation framework called Semantically Enhanced Dual-Stream.
  arXiv  Detail & Related papers  (2024-07-23T11:31:11Z)
• In-Context Translation: Towards Unifying Image Recognition, Processing, and Generation [44.26537443476901]
  We propose In-Context Translation (ICT) to unify visual recognition, low-level image processing, and conditional image generation. ICT standardizes the training of different tasks into a general in-context learning, where "in-context" means the input comprises an example input-output pair of the target task and a query image. In experiments, ICT unifies ten vision tasks and showcases impressive performance on their respective benchmarks.
  arXiv  Detail & Related papers  (2024-04-15T10:05:36Z)
• DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation [76.81628995237058]
  DFormer is a novel framework to learn transferable representations for RGB-D segmentation tasks. It pretrains the backbone using image-depth pairs from ImageNet-1K. DFormer achieves new state-of-the-art performance on two popular RGB-D tasks.
  arXiv  Detail & Related papers  (2023-09-18T11:09:11Z)
• Semantic RGB-D Image Synthesis [22.137419841504908]
  We introduce semantic RGB-D image synthesis to address this problem. Current approaches, however, are uni-modal and cannot cope with multi-modal data. We propose a generator for multi-modal data that separates modal-independent information of the semantic layout from the modal-dependent information.
  arXiv  Detail & Related papers  (2023-08-22T11:16:24Z)
• GeoVLN: Learning Geometry-Enhanced Visual Representation with Slot Attention for Vision-and-Language Navigation [52.65506307440127]
  We propose GeoVLN, which learns Geometry-enhanced visual representation based on slot attention for robust Visual-and-Language Navigation. We employ V&L BERT to learn a cross-modal representation that incorporate both language and vision informations.
  arXiv  Detail & Related papers  (2023-05-26T17:15:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.