CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook

• URL: http://arxiv.org/abs/2605.18257v1
• Date: Mon, 18 May 2026 11:56:19 GMT
• Title: CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook
• Abstract summary: We propose CodeBind, a framework that optimize multimodal representation spaces through a modality-shared-specific codebook design.<n>CodeBind decomposes features into shared components for semantic consistency and specific components for modality-unique details.<n>This design utilizes a compositional vector quantization scheme, where a shared codebook bridges modality gaps and modality-specific codebooks mitigate representation bias.
• Score: 24.224314755265556
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal representation alignment is pivotal for large language models and robotics. Traditional methods are often hindered by cross-modal information discrepancies and data scarcity, leading to suboptimal alignment spaces that overlook modality-unique features. We propose CodeBind, a framework that optimizes multimodal representation spaces through a modality-shared-specific codebook design. By incrementally aligning target and bridging modalities, CodeBind bypasses the need for fully paired data. Unlike traditional hard alignment, CodeBind decomposes features into shared components for semantic consistency and specific components for modality-unique details. This design utilizes a compositional vector quantization scheme, where a shared codebook bridges modality gaps and modality-specific codebooks mitigate representation bias by preventing dominant modalities from overshadowing others. Validated across nine modalities (text, image, video, audio, depth, thermal, tactile, 3D point cloud, EEG), CodeBind achieves state-of-the-art performance in multimodal classification and retrieval tasks.

Related papers

• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• Learning Relative Representations for Fine-Grained Multimodal Alignment with Limited Data [0.5156484100374059]
  We propose a post-hoc alignment method that learns token-level cross-modal structure using relative representations.<n>Our approach consistently outperforms existing methods in zero-shot classification, cross-modal retrieval, and zero-shot segmentation.
  arXiv  Detail & Related papers  (2026-05-16T06:33:38Z)
• Linking Modality Isolation in Heterogeneous Collaborative Perception [41.68601421239159]
  We propose CodeAlign, a framework that smoothly aligns modalities via cross-modal featurecode-feature(FCF) translation.<n>CodeAlign learns FCF translations that map features to the corresponding codes of other modalities, which are then decoded back into features in the target code space.<n> Experiments show that, when integrating three modalities, CodeAlign requires only 8% of the training parameters of prior alignment methods, reduces communication load by 1024x, and state-of-the-art perception performance on both OPV2V and DAIRV2X dataset achieves.
  arXiv  Detail & Related papers  (2026-02-28T12:09:08Z)
• Semantics Meet Signals: Dual Codebook Representationl Learning for Generative Recommendation [39.88791326869038]
  We introduce FlexCode, a popularity-aware framework that allocates a fixed token budget between a collaborative filtering codebook and a semantic codebook.<n>A lightweight MoE dynamically balances CF-specific precision and semantic generalization, while an alignment and objective smoothness maintains coherence across the popularity spectrum.<n>We perform experiments on both public and industrial-scale datasets, showing that FlexCode consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2025-11-15T05:51:38Z)
• MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval [34.21875369884307]
  Multimodal large language models (MLLMs) enable a unified encoder that directly processes composed inputs.<n>While flexible and advanced, we identify that unified encoders trained with conventional contrastive learning are prone to learn modality shortcut.<n>We propose a modality composition awareness framework to mitigate this issue.
  arXiv  Detail & Related papers  (2025-10-17T11:20:35Z)
• UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation [5.286124283204446]
  We propose UniCode$2$, a cascaded codebook framework enabling large-scale, semantically aligned, and stable visual tokenization.<n>By clustering millions of SigLIP sequence embeddings, we build a 500K-entry codebook that preserves vision-language alignment while expanding capacity.<n>UniCode2 delivers strong performance across diverse benchmarks, demonstrating the viability of scaling visual token spaces without sacrificing stability, semantics, or modularity.
  arXiv  Detail & Related papers  (2025-06-25T07:57:09Z)
• Universal Item Tokenization for Transferable Generative Recommendation [89.42584009980676]
  We propose UTGRec, a universal item tokenization approach for transferable Generative Recommendation.<n>By devising tree-structured codebooks, we discretize content representations into corresponding codes for item tokenization.<n>For raw content reconstruction, we employ dual lightweight decoders to reconstruct item text and images from discrete representations.<n>For collaborative knowledge integration, we assume that co-occurring items are similar and integrate collaborative signals through co-occurrence alignment and reconstruction.
  arXiv  Detail & Related papers  (2025-04-06T08:07:49Z)
• Bridging Textual-Collaborative Gap through Semantic Codes for Sequential Recommendation [91.13055384151897]
  CCFRec is a novel Code-based textual and Collaborative semantic Fusion method for sequential Recommendation.<n>We generate fine-grained semantic codes from multi-view text embeddings through vector quantization techniques.<n>In order to further enhance the fusion of textual and collaborative semantics, we introduce an optimization strategy.
  arXiv  Detail & Related papers  (2025-03-15T15:54:44Z)
• Learning Multi-Aspect Item Palette: A Semantic Tokenization Framework for Generative Recommendation [55.99632509895994]
  We introduce LAMIA, a novel approach for multi-aspect semantic tokenization.<n>Unlike RQ-VAE, which uses a single embedding, LAMIA learns an item palette''--a collection of independent and semantically parallel embeddings.<n>Our results demonstrate significant improvements in recommendation accuracy over existing methods.
  arXiv  Detail & Related papers  (2024-09-11T13:49:48Z)
• Preserving Modality Structure Improves Multi-Modal Learning [64.10085674834252]
  Self-supervised learning on large-scale multi-modal datasets allows learning semantically meaningful embeddings without relying on human annotations. These methods often struggle to generalize well on out-of-domain data as they ignore the semantic structure present in modality-specific embeddings. We propose a novel Semantic-Structure-Preserving Consistency approach to improve generalizability by preserving the modality-specific relationships in the joint embedding space.
  arXiv  Detail & Related papers  (2023-08-24T20:46:48Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.