Omni-Referring Image Segmentation

• URL: http://arxiv.org/abs/2512.06862v1
• Date: Sun, 07 Dec 2025 14:32:34 GMT
• Title: Omni-Referring Image Segmentation
• Abstract summary: We propose a novel task Omni-Referring Image termed ( OmniRIS) towards highly generalized image segmentation.<n>Compared with existing unimodally conditioned segmentation tasks, such as RIS and visual RIS, OmniRIS supports the input of text instructions and reference images with masks, boxes or scribbles as omni-prompts.<n>We rigorously design and construct a large dataset termed OmniRef, which consists of 186,939 omni-prompts for 30,956 images.
• Score: 106.67482846604877
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose a novel task termed Omni-Referring Image Segmentation (OmniRIS) towards highly generalized image segmentation. Compared with existing unimodally conditioned segmentation tasks, such as RIS and visual RIS, OmniRIS supports the input of text instructions and reference images with masks, boxes or scribbles as omni-prompts. This property makes it can well exploit the intrinsic merits of both text and visual modalities, i.e., granular attribute referring and uncommon object grounding, respectively. Besides, OmniRIS can also handle various segmentation settings, such as one v.s. many and many v.s. many, further facilitating its practical use. To promote the research of OmniRIS, we also rigorously design and construct a large dataset termed OmniRef, which consists of 186,939 omni-prompts for 30,956 images, and establish a comprehensive evaluation system. Moreover, a strong and general baseline termed OmniSegNet is also proposed to tackle the key challenges of OmniRIS, such as omni-prompt encoding. The extensive experiments not only validate the capability of OmniSegNet in following omni-modal instructions, but also show the superiority of OmniRIS for highly generalized image segmentation.

Related papers

• Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video [5.732421858297378]
  We present Omni-Embed-Nemotron, a unified multimodal retrieval embedding model developed to handle the increasing complexity of real-world information needs.<n>We describe the architecture, training setup, and evaluation results of Omni-Embed-Nemotron, and demonstrate its effectiveness in text, image, and video retrieval.
  arXiv  Detail & Related papers  (2025-10-03T19:29:50Z)
• OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation [74.55725909072903]
  We propose a novel multi-modal learning framework, termed OmniSegmentor.<n>Based on ImageNet, we assemble a large-scale dataset for multi-modal pretraining, called ImageNeXt.<n>We introduce a universal multi-modal pretraining framework that consistently amplifies the model's perceptual capabilities across various scenarios.
  arXiv  Detail & Related papers  (2025-09-18T15:52:44Z)
• Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation [68.4547223233201]
  We propose Omnimodal Referring Audio-Visual expressions ( OmniAVS), a new dataset containing 2,104 videos and 61,095 multimodal referring expressions.<n> OmniAVS stands out with three key innovations: (1) 8 types of multimodal expressions that flexibly combine text, speech, sound, and visual cues; (2) an emphasis on understanding audio content beyond just detecting their presence; and (3) the inclusion of complex reasoning in expressions.
  arXiv  Detail & Related papers  (2025-07-30T17:59:31Z)
• OmniGen2: Exploration to Advanced Multimodal Generation [102.55457132692969]
  We introduce OmniGen2, a versatile and open-source generative model designed to provide a unified solution for diverse generation tasks.<n>Unlike OmniGen v1, OmniGen2 features two distinct decoding pathways for text and image modalities, utilizing unshared parameters and a decoupled image tokenizer.
  arXiv  Detail & Related papers  (2025-06-23T17:38:54Z)
• OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities [124.05360767047539]
  We introduce OmnixR, an evaluation suite designed to benchmark SoTA Omni-modality Language Models. evaluating OLMs, which integrate multiple modalities such as text, vision, and audio, presents unique challenges. Our experiments find that all state-of-the-art OLMs struggle with OmnixR questions that require integrating information from multiple modalities to answer.
  arXiv  Detail & Related papers  (2024-10-16T04:29:46Z)
• OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal Instruction [32.08995899903304]
  We present OmniBooth, an image generation framework that enables spatial control with instance-level multi-modal customization. Our approach significantly expands the scope of text-to-image generation, and elevates it to a more versatile and practical dimension in controllability.
  arXiv  Detail & Related papers  (2024-10-07T11:26:13Z)
• OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering [3.8233569758620063]
  We introduce OmniQuery, a novel system that is able to answer complex personal memory-related questions.<n>Given a question, OmniQuery retrieves relevant augmented memories and uses a large language model (LLM) to generate answers with references.<n>In human evaluations, we show the effectiveness of OmniQuery with an accuracy of 71.5%, outperforming a conventional RAG system by winning or tying for 74.5% of the time.
  arXiv  Detail & Related papers  (2024-09-12T17:48:08Z)
• OmniGlue: Generalizable Feature Matching with Foundation Model Guidance [21.692165943308797]
  We introduce OmniGlue, the first learnable image matcher that is designed with generalization as a core principle. We propose a novel keypoint position-guided attention mechanism which disentangles spatial and appearance information, leading to enhanced matching descriptors.
  arXiv  Detail & Related papers  (2024-05-21T17:59:22Z)
• Segment Everything Everywhere All at Once [124.90835636901096]
  We present SEEM, a promptable and interactive model for segmenting everything everywhere all at once in an image. We propose a novel decoding mechanism that enables diverse prompting for all types of segmentation tasks. We conduct a comprehensive empirical study to validate the effectiveness of SEEM across diverse segmentation tasks.
  arXiv  Detail & Related papers  (2023-04-13T17:59:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.