Omni-Interactive Universal Embedder

• URL: http://arxiv.org/abs/2608.27044v1
• Date: Thu, 27 Aug 2026 12:31:55 GMT
• Title: Omni-Interactive Universal Embedder
• Abstract summary: We propose the first Omni-Interactive Universal Embedder ( OmniUE)<n>It learns a unified embedding space across text, video, and audio by leveraging intermediate-layer representations from dedicated learnable tokens.<n>It also supports omni-interactive querying, enabling users to provide inputs in the form of text, visual regions of interest, and audio spans.
• Score: 36.73317890827303
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal representation learning has been shifting from traditional two-tower architectures to large language model (LLM)-based embedders due to their strong instruction-following capabilities. Despite this progress, existing approaches primarily focus on language and image modalities, which also remain the dominant modalities for user-conditioned interactions in current embedders. In this paper, we propose the first Omni-Interactive Universal Embedder (OmniUE), which not only learns a unified embedding space across text, video, and audio by leveraging intermediate-layer representations from dedicated learnable tokens, but also supports omni-interactive querying, enabling users to provide inputs in the form of text, visual regions of interest, and audio spans. Within OmniUE, visual and audio segmenters process diverse user interactions and integrate them with an omni-LLM to produce user-conditioned any-to-any embeddings via context aggregation. To evaluate OmniUE's omni-interactive capabilities, we introduce OmniCHOIR, benchmarking models for omni-interactive compositional audio retrieval based on the given text, video, and audio as well as unimodal or multimodal interaction prompts. OmniUE consistently surpasses state-of-the-art baselines across diverse modalities, with average improvements of 10.5% on textual-interactive video benchmarks (MMEB-v2-video), 1.1% on audio tasks (MAEB), 83.7% on visual-interactive benchmarks (SCaR), and 24.1% on our omni-interactive OmniCHOIR benchmark. We believe that jointly advancing omni-modal representation learning and omni-interactive querying paves the way toward universal embedders.

Related papers

• Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation [93.44732526074876]
  We present an end-to-end framework targeted at the precise binding and seamless fusion of multiple identities.<n>Within this architecture, we propose Semantic-Anchored Multimodal Ro (SA-MRo) to anchor visual and audio reference tokens, along with TTS embeddings, to their corresponding semantic descriptions.<n>Experiments demonstrate that Omni-Context achieves state-of-the-art performance in dual-modal customized generation.
  arXiv  Detail & Related papers  (2026-05-17T14:56:52Z)
• Complementary and Contrastive Learning for Audio-Visual Segmentation [74.11434759171199]
  We present Complementary and Contrastive Transformer (CCFormer), a novel framework adept at processing both local and global information.<n>Our method sets new state-of-the-art benchmarks across the S4, MS3 and AVSS datasets.
  arXiv  Detail & Related papers  (2025-10-11T06:36:59Z)
• Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video [5.732421858297378]
  We present Omni-Embed-Nemotron, a unified multimodal retrieval embedding model developed to handle the increasing complexity of real-world information needs.<n>We describe the architecture, training setup, and evaluation results of Omni-Embed-Nemotron, and demonstrate its effectiveness in text, image, and video retrieval.
  arXiv  Detail & Related papers  (2025-10-03T19:29:50Z)
• Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation [68.4547223233201]
  We propose Omnimodal Referring Audio-Visual expressions ( OmniAVS), a new dataset containing 2,104 videos and 61,095 multimodal referring expressions.<n> OmniAVS stands out with three key innovations: (1) 8 types of multimodal expressions that flexibly combine text, speech, sound, and visual cues; (2) an emphasis on understanding audio content beyond just detecting their presence; and (3) the inclusion of complex reasoning in expressions.
  arXiv  Detail & Related papers  (2025-07-30T17:59:31Z)
• MAGMaR Shared Task System Description: Video Retrieval with OmniEmbed [55.526939500742]
  We use OmniEmbed, a powerful multimodal embedding model from the Tevatron 2.0 toolkit, to generate unified embeddings for text, images, audio, and video.<n>Our submission achieved the highest score on the MAGMaR shared task leaderboard among public submissions as of May 20th, 2025.
  arXiv  Detail & Related papers  (2025-06-11T05:40:26Z)
• OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking [22.337906095079198]
  We present OmniTalker, a unified framework that jointly generates synchronized talking audio-video content from input text.<n>Our framework adopts a dual-branch diffusion transformer (DiT) architecture, with one branch dedicated to audio generation and the other to video synthesis.
  arXiv  Detail & Related papers  (2025-04-03T09:48:13Z)
• OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs [6.365802395342737]
  We present OmniVox, the first systematic evaluation of four omni-LLMs on the zero-shot emotion recognition task.<n>We evaluate on two widely used multimodal emotion benchmarks: IEMOCAP and MELD, and find zero-shot omni-LLMs outperform or are competitive with fine-tuned audio models.<n>We present acoustic prompting, an audio-specific prompting strategy for omni-LLMs which focuses on acoustic feature analysis, conversation context analysis, and step-by-step reasoning.
  arXiv  Detail & Related papers  (2025-03-27T13:12:49Z)
• OmniBench: Towards The Future of Universal Omni-Language Models [63.16606414452612]
  We introduce OmniBench, a novel benchmark designed to evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously.<n>Our evaluation reveals that open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts.<n>We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance.
  arXiv  Detail & Related papers  (2024-09-23T17:59:05Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.