OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering

• URL: http://arxiv.org/abs/2604.08209v1
• Date: Thu, 09 Apr 2026 13:09:40 GMT
• Title: OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering
• Abstract summary: We propose OmniJigsaw, a generic self-supervised framework built upon a temporal reordering proxy task.<n>Joint Modality Integration, Sample-level Modality Selection, and Clip-level Modality Masking are used to compel cross-modal integration.<n>Our analysis reveals a bi-modal shortcut phenomenon'' in joint modality integration and demonstrates that fine-grained clip-level modality masking mitigates this issue.
• Score: 53.30273345137238
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: To extend the reinforcement learning post-training paradigm to omni-modal models for concurrently bolstering video-audio understanding and collaborative reasoning, we propose OmniJigsaw, a generic self-supervised framework built upon a temporal reordering proxy task. Centered on the chronological reconstruction of shuffled audio-visual clips, this paradigm strategically orchestrates visual and auditory signals to compel cross-modal integration through three distinct strategies: Joint Modality Integration, Sample-level Modality Selection, and Clip-level Modality Masking. Recognizing that the efficacy of such proxy tasks is fundamentally tied to puzzle quality, we design a two-stage coarse-to-fine data filtering pipeline, which facilitates the efficient adaptation of OmniJigsaw to massive unannotated omni-modal data. Our analysis reveals a ``bi-modal shortcut phenomenon'' in joint modality integration and demonstrates that fine-grained clip-level modality masking mitigates this issue while outperforming sample-level modality selection. Extensive evaluations on 15 benchmarks show substantial gains in video, audio, and collaborative reasoning, validating OmniJigsaw as a scalable paradigm for self-supervised omni-modal learning.

Related papers

• Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation [93.44732526074876]
  We present an end-to-end framework targeted at the precise binding and seamless fusion of multiple identities.<n>Within this architecture, we propose Semantic-Anchored Multimodal Ro (SA-MRo) to anchor visual and audio reference tokens, along with TTS embeddings, to their corresponding semantic descriptions.<n>Experiments demonstrate that Omni-Context achieves state-of-the-art performance in dual-modal customized generation.
  arXiv  Detail & Related papers  (2026-05-17T14:56:52Z)
• OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation [60.95158352332569]
  Recent advances in joint audio-video generation demand strong per-modality fidelity, cross-modal alignment, and fine-grained synchronization.<n>These shortcomings suggest that vanilla RL fine-tuning strategy with a single global advantage often leads to suboptimal results.<n>We propose OmniNFT, a novel modality-aware online diffusion RL framework with three key innovations.
  arXiv  Detail & Related papers  (2026-05-12T17:56:59Z)
• OmniRet: Efficient and High-Fidelity Omni Modality Retrieval [51.80205678389465]
  We present OmniRet, the first retrieval model capable of handling complex, composed queries spanning three key modalities: text, vision, and audio.<n>Our model demonstrates significant improvements on composed query, audio and video retrieval tasks, while achieving on-par performance with state-of-the-art models on others.
  arXiv  Detail & Related papers  (2026-03-02T17:19:55Z)
• Apollo: Unified Multi-Task Audio-Video Joint Generation [15.004783109205666]
  Non-commercial approaches still suffer audio-visual asynchrony, poor lip-speech alignment, and unimodal degradation.<n>We introduce Apollo and delve into three axes--model architecture, training strategy, and data curation.<n>For datasets, we present the first large-scale audio-video dataset with dense captions.
  arXiv  Detail & Related papers  (2026-01-07T18:03:45Z)
• AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering [33.971153351649754]
  We propose a novel framework named AV-Master to address the challenges of Audio-Visual Question Answering.<n>It enhances the model's ability to extract key information from complex audio-visual scenes.<n>Experiments on four large-scale benchmarks show that AV-Master significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2025-10-21T06:58:34Z)
• Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models [35.86252379746625]
  The goal of this work is to enhance balanced multimodal understanding in audio-visual large language models (AV-LLMs)<n>In current AV-LLMs, audio and video features are typically processed jointly in the decoder.<n>We propose Fork-Merge Decoding (FMD), a simple yet effective inference-time strategy that requires no additional training or architectural modifications.
  arXiv  Detail & Related papers  (2025-05-27T08:22:56Z)
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection [57.649223695021114]
  We present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection.<n>Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T13:59:16Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)
• Multimodal Variational Auto-encoder based Audio-Visual Segmentation [46.67599800471001]
  ECMVAE factorizes the representations of each modality with a modality-shared representation and a modality-specific representation. Our approach leads to a new state-of-the-art for audio-visual segmentation, with a 3.84 mIOU performance leap.
  arXiv  Detail & Related papers  (2023-10-12T13:09:40Z)
• An Efficient End-to-End Transformer with Progressive Tri-modal Attention for Multi-modal Emotion Recognition [27.96711773593048]
  We propose the multi-modal end-to-end transformer (ME2ET), which can effectively model the tri-modal features interaction. At the low-level, we propose the progressive tri-modal attention, which can model the tri-modal feature interactions by adopting a two-pass strategy. At the high-level, we introduce the tri-modal feature fusion layer to explicitly aggregate the semantic representations of three modalities.
  arXiv  Detail & Related papers  (2022-09-20T14:51:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.