Simultaneous Long-tailed Recognition and Multi-modal Fusion for Highly Imbalanced Multi-modal Data

• URL: http://arxiv.org/abs/2605.10498v1
• Date: Mon, 11 May 2026 12:56:53 GMT
• Title: Simultaneous Long-tailed Recognition and Multi-modal Fusion for Highly Imbalanced Multi-modal Data
• Abstract summary: Long-tailed distributions in class-imbalanced data present a fundamental challenge for deep learning models.<n>We introduce a new framework for long-tailed recognition that explicitly handles multi-modal inputs.<n>Our approach extends multi-expert architectures to the multi-modal setting by fusing heterogeneous data into a unified representation.
• Score: 9.797319790710711
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-tailed distributions in class-imbalanced data present a fundamental challenge for deep learning models, which tend to be biased toward majority classes. While recent methods for long-tailed recognition have mitigated this issue, they are largely restricted to single-modal inputs and cannot fully exploit complementary information from diverse data sources. In this work, we introduce a new framework for long-tailed recognition that explicitly handles multi-modal inputs. Our approach extends multi-expert architectures to the multi-modal setting by fusing heterogeneous data into a unified representation while leveraging modality-specific networks to estimate the informativeness of each modality. These confidence-guided weights dynamically modulate the fusion process, ensuring that more informative modalities contribute more strongly to the final decision. To further enhance performance, we design specialized training and test procedures that accommodate diverse modality combinations, including images and tabular data. Extensive experiments on benchmark and real-world datasets demonstrate that the proposed approach not only effectively integrates multi-modal information but also outperforms existing methods in handling long-tailed, class-imbalanced scenarios, highlighting its robustness and generalization capability.

Related papers

• MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning [13.327494159383356]
  MultiLoReFT is an efficient and scalable low-rank representation framework for multimodal learning.<n>It learns interpretable projection subspaces that decouple shared and modality-specific information.<n>It produces representations that support multimodal prediction across simulated and real-world benchmarks.
  arXiv  Detail & Related papers  (2026-07-18T11:53:22Z)
• CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning [35.562458985015944]
  We present a novel Cyclic INformative Learning framework (CyIN) to bridge the gap between complete and incomplete multimodal learning.<n>To supplement the missing information caused by incomplete multimodal input, we propose cross-modal cyclic translation.<n>CyIN succeeds in jointly optimizing complete and incomplete multimodal learning in one unified model.
  arXiv  Detail & Related papers  (2026-02-04T07:05:15Z)
• Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process [55.91649771370862]
  Dirichlet process (DP) mixture model is a powerful non-parametric method that can amplify the most prominent features.<n>We propose a new DP-driven multimodal learning framework that automatically achieves an optimal balance between prominent intra-modal representation learning and cross-modal alignment.
  arXiv  Detail & Related papers  (2025-10-23T16:53:24Z)
• Deep Correlated Prompting for Visual Recognition with Missing Modalities [22.40271366031256]
  Large-scale multimodal models have shown excellent performance over a series of tasks powered by the large corpus of paired multimodal training data. However, this simple assumption may not always hold in the real world due to privacy constraints or collection difficulty. We refer to prompt learning to adapt large pretrained multimodal models to handle missing-modality scenarios by regarding different missing cases as different types of input.
  arXiv  Detail & Related papers  (2024-10-09T05:28:43Z)
• Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models [6.610033827647869]
  In real-world scenarios, consistently acquiring complete multimodal data presents significant challenges. This often leads to the issue of missing modalities, where data for certain modalities are absent. We propose a novel framework integrating parameter-efficient fine-tuning of unimodal pretrained models with a self-supervised joint-embedding learning method.
  arXiv  Detail & Related papers  (2024-07-17T14:44:25Z)
• U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation [63.31007867379312]
  We introduce U3M: An Unbiased Multiscale Modal Fusion Model for Multimodal Semantics. We employ feature fusion at multiple scales to ensure the effective extraction and integration of both global and local features. Experimental results demonstrate that our approach achieves superior performance across multiple datasets.
  arXiv  Detail & Related papers  (2024-05-24T08:58:48Z)
• NativE: Multi-modal Knowledge Graph Completion in the Wild [51.80447197290866]
  We propose a comprehensive framework NativE to achieve MMKGC in the wild. NativE proposes a relation-guided dual adaptive fusion module that enables adaptive fusion for any modalities. We construct a new benchmark called WildKGC with five datasets to evaluate our method.
  arXiv  Detail & Related papers  (2024-03-28T03:04:00Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning [112.51498431119616]
  This paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. A single model, HighMMT, scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas.
  arXiv  Detail & Related papers  (2022-03-02T18:56:20Z)
• Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models [86.9292779620645]
  We develop a contrastive framework for generative model learning, allowing us to train the model not just by the commonality between modalities, but by the distinction between "related" and "unrelated" multimodal data. Under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.
  arXiv  Detail & Related papers  (2020-07-02T15:08:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.