Linking Modality Isolation in Heterogeneous Collaborative Perception

• URL: http://arxiv.org/abs/2603.00609v1
• Date: Sat, 28 Feb 2026 12:09:08 GMT
• Title: Linking Modality Isolation in Heterogeneous Collaborative Perception
• Abstract summary: We propose CodeAlign, a framework that smoothly aligns modalities via cross-modal featurecode-feature(FCF) translation.<n>CodeAlign learns FCF translations that map features to the corresponding codes of other modalities, which are then decoded back into features in the target code space.<n> Experiments show that, when integrating three modalities, CodeAlign requires only 8% of the training parameters of prior alignment methods, reduces communication load by 1024x, and state-of-the-art perception performance on both OPV2V and DAIRV2X dataset achieves.
• Score: 41.68601421239159
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Collaborative perception leverages data exchange among multiple agents to enhance overall perception capabilities. However, heterogeneity across agents introduces domain gaps that hinder collaboration, and this is further exacerbated by an underexplored issue: modality isolation. It arises when multiple agents with different modalities never co-occur in any training data frame, enlarging cross-modal domain gaps. Existing alignment methods rely on supervision from spatially overlapping observations, thus fail to handle modality isolation. To address this challenge, we propose CodeAlign, the first efficient, co-occurrence-free alignment framework that smoothly aligns modalities via cross-modal feature-code-feature(FCF) translation. The key idea is to explicitly identify the representation consistency through codebook, and directly learn mappings between modality-specific feature spaces, thereby eliminating the need for spatial correspondence. Codebooks regularize feature spaces into code spaces, providing compact yet expressive representations. With a prepared code space for each modality, CodeAlign learns FCF translations that map features to the corresponding codes of other modalities, which are then decoded back into features in the target code space, enabling effective alignment. Experiments show that, when integrating three modalities, CodeAlign requires only 8% of the training parameters of prior alignment methods, reduces communication load by 1024x, and achieves state-of-the-art perception performance on both OPV2V and DAIR-V2X dataset. Code will be released on https://github.com/cxliu0314/CodeAlign.

Related papers

• CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook [24.224314755265556]
  We propose CodeBind, a framework that optimize multimodal representation spaces through a modality-shared-specific codebook design.<n>CodeBind decomposes features into shared components for semantic consistency and specific components for modality-unique details.<n>This design utilizes a compositional vector quantization scheme, where a shared codebook bridges modality gaps and modality-specific codebooks mitigate representation bias.
  arXiv  Detail & Related papers  (2026-05-18T11:56:19Z)
• GT-Space: Enhancing Heterogeneous Collaborative Perception with Ground Truth Feature Space [30.468574984070894]
  In autonomous driving, multi-agent collaborative perception enhances sensing capabilities by enabling agents to share perceptual data.<n>Existing approaches often require retraining encoders or designing interpreter modules for pairwise feature alignment.<n>We propose em GT-Space, a flexible and scalable collaborative perception framework for heterogeneous agents.
  arXiv  Detail & Related papers  (2026-03-13T14:39:40Z)
• CoDS: Enhancing Collaborative Perception in Heterogeneous Scenarios via Domain Separation [25.442016145030383]
  Collaborative perception has been proven to improve individual perception in autonomous driving.<n>Most methods assume identical encoders for all agents, which does not hold true when these models are deployed in real-world applications.<n>Existing methods usually align neighbor features to those of the ego vehicle, which is vulnerable to noise from domain gaps.
  arXiv  Detail & Related papers  (2025-10-15T11:29:14Z)
• Cross-modal Full-mode Fine-grained Alignment for Text-to-Image Person Retrieval [54.90229711181207]
  Text-to-Image Person Retrieval (TIPR) aims to retrieve the most relevant person images based on a given text query.<n>The key challenge in TIPR lies in achieving effective alignment between textual and visual modalities.<n>We propose FMFA, a cross-modal Full-Mode Fine-grained Alignment framework.
  arXiv  Detail & Related papers  (2025-09-17T07:12:05Z)
• Style Quantization for Data-Efficient GAN Training [18.40243591024141]
  Under limited data setting, GANs often struggle to navigate and effectively exploit the input latent space.<n>We propose textitSQ-GAN, a novel approach that enhances consistency regularization.<n>Experiments demonstrate significant improvements in both discriminator robustness and generation quality.
  arXiv  Detail & Related papers  (2025-03-31T16:28:44Z)
• From Limited Labels to Open Domains:An Efficient Learning Method for Drone-view Geo-Localization [12.785100004522059]
  Traditional supervised drone-view geo-localization (DVGL) methods heavily depend on paired training data.<n>DVGL methods require obtaining the new paired data and subsequent retraining for model adaptation.<n>We propose a novel cross-domain invariant knowledge transfer network (CDIKTNet) with limited supervision.
  arXiv  Detail & Related papers  (2025-03-10T16:46:43Z)
• Exploring Homogeneous and Heterogeneous Consistent Label Associations for Unsupervised Visible-Infrared Person ReID [57.500045584556794]
  We introduce a Modality-Unified Label Transfer (MULT) module that simultaneously accounts for both homogeneous and heterogeneous fine-grained instance-level structures.<n>The proposed MULT ensures that the generated pseudo-labels maintain alignment across modalities while upholding structural consistency within intra-modality.<n> Experiments demonstrate that our proposed method outperforms existing state-of-the-art USL-VI-ReID methods.
  arXiv  Detail & Related papers  (2024-02-01T15:33:17Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)
• HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning [74.76431541169342]
  Zero-shot learning (ZSL) tackles the unseen class recognition problem, transferring semantic knowledge from seen classes to unseen ones. We propose a novel hierarchical semantic-visual adaptation (HSVA) framework to align semantic and visual domains. Experiments on four benchmark datasets demonstrate HSVA achieves superior performance on both conventional and generalized ZSL.
  arXiv  Detail & Related papers  (2021-09-30T14:27:50Z)
• AlignSeg: Feature-Aligned Segmentation Networks [109.94809725745499]
  We propose Feature-Aligned Networks (AlignSeg) to address misalignment issues during the feature aggregation process. Our network achieves new state-of-the-art mIoU scores of 82.6% and 45.95%, respectively.
  arXiv  Detail & Related papers  (2020-02-24T10:00:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.