Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations

• URL: http://arxiv.org/abs/2605.12145v2
• Date: Wed, 13 May 2026 09:22:06 GMT
• Title: Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations
• Abstract summary: Multimodal learning seeks to integrate information across diverse sensory sources.<n>Current approaches struggle to balance cross-modal generalizability with modality-specific structure.<n>We introduce CoDAAR, a novel framework that establishes semantic consensus across modality-specific codebooks.
• Score: 4.67724003380452
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multimodal learning seeks to integrate information across diverse sensory sources, yet current approaches struggle to balance cross-modal generalizability with modality-specific structure. Continuous (implicit) methods preserve fine-grained priors but render generalization challenging, while discrete (explicit) approaches enforce shared prototypes at the expense of modality specificity. We introduce CoDAAR (Cross-modal Discrete Alignment And Reconstruction), a novel framework that resolves this long-standing trade-off by establishing semantic consensus across modality-specific codebooks through index-level alignment. This design uniquely allows CoDAAR to preserve modality-unique structures while achieving generalizable cross-modal representations within a unified discrete space. CoDAAR combines two complementary mechanisms: Discrete Temporal Alignment (DTA), which enables fine-grained temporal quantization, and Cascading Semantic Alignment (CSA), which promotes progressive cross-modal semantic agreement. Together, they establish a competition-free unified representation space. Trained with self-supervised reconstruction objectives on paired multimodal sequences, CoDAAR demonstrates robust cross-modal and cross-domain generalization. Across Cross-Modal Generalization benchmarks, including event classification, localization, video segmentation, and cross-dataset transfer, CoDAAR achieves state-of-the-art performance, establishing a new paradigm for discrete and generalizable multimodal representation learning.

Related papers

• Discrete Diffusion Models: A Unified Framework from Tokenization to Generation [66.16696467962375]
  This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space.<n>Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space.
  arXiv  Detail & Related papers  (2026-07-15T04:23:22Z)
• Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis [41.310321156046165]
  We propose textbfSeRIn (textbfSegregate, textbfRefine, textbfIntegrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior.<n> Modality-specific evolve representations along isolated pathways, each refined against its respective encoder context.<n>A dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams.
  arXiv  Detail & Related papers  (2026-07-14T12:13:53Z)
• RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities [34.35941675436447]
  RePercENT is a self-supervised framework designed to unlock scalable pairwise disentanglement beyond two modalities.<n>Our approach operates directly on pre-extracted embeddings, eliminating the need for extensive joint pre-training.<n>Across diverse modalities and tasks, RePercENT successfully recovers disentangled components while maintaining competitive performance.
  arXiv  Detail & Related papers  (2026-06-03T17:10:39Z)
• Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs [84.3271821505699]
  Chain of Modality (CoM) is an agentic framework that transitions multimodal fusion from passive concatenation to dynamic orchestration.<n> operating in either a training-free or a data-efficient SFT setting, CoM achieves robust and consistent generalization across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-04-16T01:21:14Z)
• CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning [10.210493389825116]
  Multimodal learning aims to capture both shared and private information from multiple modalities.<n>Existing methods that project all modalities into a single latent space for fusion often overlook the asynchronous, multi-level semantic structure of multimodal data.<n>We propose Cross-Level Co-Representation (CLCR), which explicitly organizes each modality's features into a three-level semantic hierarchy.
  arXiv  Detail & Related papers  (2026-02-23T08:47:19Z)
• Towards Uniformity and Alignment for Multimodal Representation Learning [66.87764574237532]
  We identify two conflicts in the multimodal regime, both exacerbated as the number of modalities increases.<n>We propose a principled decoupling of alignment and uniformity for multimodal representations.<n>We then provide a theoretical guarantee that our method acts as an efficient proxy for a global Hlder divergence over multiple modality distributions.
  arXiv  Detail & Related papers  (2026-02-10T08:08:07Z)
• Efficient Generalization via Multimodal Co-Training under Data Scarcity and Distribution Shift [0.6331016589903705]
  multimodal co-training is designed to enhance model generalization in situations where labeled data is limited.<n>We examine the theoretical foundations of this framework, deriving conditions under which the use of unlabeled data leads to significant improvements in generalization.<n>We establish a novel generalization bound that, for the first time in a multimodal co-training context, decomposes and quantifies the advantages gained from leveraging unlabeled multimodal data.
  arXiv  Detail & Related papers  (2025-10-08T20:13:17Z)
• UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception [54.53657134205492]
  UniAlignment is a unified multimodal generation framework within a single diffusion transformer.<n>It incorporates both intrinsic-modal semantic alignment and cross-modal semantic alignment, thereby enhancing the model's cross-modal consistency and instruction-following robustness.<n>We present SemGen-Bench, a new benchmark specifically designed to evaluate multimodal semantic consistency under complex textual instructions.
  arXiv  Detail & Related papers  (2025-09-28T09:11:30Z)
• CHARM: Collaborative Harmonization across Arbitrary Modalities for Modality-agnostic Semantic Segmentation [44.48226146116737]
  Modality-agnostic Semantic (MaSS) aims to achieve robust scene understanding across arbitrary combinations of input modality.<n>We propose CHARM, a novel complementary learning framework designed to implicitly align content while preserving modality-specific advantages.
  arXiv  Detail & Related papers  (2025-08-05T04:10:51Z)
• Continual Cross-Modal Generalization [48.56694158680082]
  Cross-modal generalization aims to learn a shared representation space from multimodal pairs.<n>We propose a continual learning approach that incrementally maps new modalities into a shared codebook via a mediator modality.<n>Experiments on image-text, audio-text, video-text, and speech-text show that our method achieves strong performance on various cross-modal generalization tasks.
  arXiv  Detail & Related papers  (2025-04-01T09:16:20Z)
• Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence [83.15764564701706]
  We propose a novel framework that performs vision-language alignment by integrating Cauchy-Schwarz divergence with mutual information.<n>We find that the CS divergence seamlessly addresses the InfoNCE's alignment-uniformity conflict and serves complementary roles with InfoNCE.<n> Experiments on text-to-image generation and cross-modality retrieval tasks demonstrate the effectiveness of our method on vision-language alignment.
  arXiv  Detail & Related papers  (2025-02-24T10:29:15Z)
• Enhancing Multimodal Unified Representations for Cross Modal Generalization [52.16653133604068]
  We propose Training-free Optimization of Codebook (TOC) and Fine and Coarse cross-modal Information Disentangling (FCID)<n>These methods refine the unified discrete representations from pretraining and perform fine- and coarse-grained information disentanglement tailored to the specific characteristics of each modality.
  arXiv  Detail & Related papers  (2024-03-08T09:16:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.