Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification

• URL: http://arxiv.org/abs/2606.05455v1
• Date: Wed, 03 Jun 2026 21:24:05 GMT
• Title: Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification
• Abstract summary: DFPL is a novel framework for fine-grained prototype learning.<n>SSPM extracts compact and diverse shared and modality-specific prototypes.<n>PFA module jointly enforces prototype-level distribution matching and prototype-to-class semantic alignment.
• Score: 28.25868366778138
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The missing-modality problem poses a significant challenge in image-tabular multimodal learning across a wide range of multimedia applications, including product understanding, recommendation systems, and medical diagnosis. This challenge is particularly pronounced when the two modalities are highly heterogeneous, as images and tabular attributes differ substantially in their semantic granularity and data distributions. Existing methods learn modality-invariant representations through disentanglement and alignment over global token-averaged features, capturing only coarse cross-modal consistency and overlooking fine-grained semantic and distributional misalignment, which hampers the exploitation of complementary cues under missing modalities. To address this, we propose DFPL, a novel framework for fine-grained prototype learning. Specifically, Shared-Specific Prototype Modeling (SSPM) extracts compact and diverse shared and modality-specific prototypes, and further performs prototype-level disentanglement to suppress redundant intra-modality correlations. Additionally, we propose a Prototype-guided Fine-grained Alignment (PFA) module that jointly enforces prototype-level distribution matching and prototype-to-class semantic alignment within a unified prototype space, thereby preserving both fine-grained distributional and semantic consistency across modalities. We further introduce a Class-aware Multi-scale Aggregation (CMA) module to adaptively aggregate shared semantics and modality-specific characteristics from global and prototype levels for robust predictions. Extensive experiments on three diverse image-tabular benchmarks demonstrate the superiority of our method compared to the previous approaches under various missing-modality settings. Code will be made publicly available.

Related papers

• PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging [80.17966517536102]
  We argue that the core of Multimodal Large Language Models pre-training lies in establishing effective cross-modal alignment.<n>Motivated by this insight, we introduce the post-alignment merging task, which aims to integrate cross-modal alignment capabilities learned from multimodal pre-training.<n>We propose PivotMerge, a post-alignment merging framework for cross-modal projectors.
  arXiv  Detail & Related papers  (2026-04-18T09:38:03Z)
• Divide, Conquer and Unite: Hierarchical Style-Recalibrated Prototype Alignment for Federated Medical Image Segmentation [66.82598255715696]
  Federated learning enables multiple medical institutions to train a global model without sharing data.<n>Current approaches primarily focus on final-layer features, overlooking critical multi-level cues.<n>We propose FedBCS to bridge feature representation gaps via domain-invariant contextual prototypes alignment.
  arXiv  Detail & Related papers  (2025-11-14T04:15:34Z)
• Libra-MIL: Multimodal Prototypes Stereoscopic Infused with Task-specific Language Priors for Few-shot Whole Slide Image Classification [18.928408687991368]
  Large Language Models (LLMs) are emerging as a promising direction in computational pathology.<n>Existing vision-language Multi-Instance Learning (MIL) methods often employ unidirectional guidance.<n>We introduce a novel approach, Multimodal Prototype-based Multi-Instance Learning, that promotes bidirectional interaction.
  arXiv  Detail & Related papers  (2025-11-11T07:46:38Z)
• DPL: Spatial-Conditioned Diffusion Prototype Enhancement for One-Shot Medical Segmentation [0.0]
  One-shot medical image segmentation faces fundamental challenges in prototype representation due to limited annotated data and anatomical variability across patients.<n>Traditional prototype-based methods rely on deterministic averaging of support features, creating brittle representations that fail to capture intra-class diversity essential for robust generalization.<n>This work introduces Diffusion Prototype Learning, a novel framework that reformulates prototype construction through diffusion-based feature space exploration.
  arXiv  Detail & Related papers  (2025-10-14T05:28:58Z)
• Leveraging Shared Prototypes for a Multimodal Pulse Motion Foundation Model [4.895784700544358]
  ProtoMM is a novel framework that introduces a shared prototype dictionary to anchor heterogeneous modalities in a common embedding space.<n>By clustering representations around shared prototypes rather than explicit negative sampling, our method captures complementary information across modalities and provides a coherent "common language" for physiological signals.
  arXiv  Detail & Related papers  (2025-10-10T18:13:38Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• Probabilistic Prototype Calibration of Vision-Language Models for Generalized Few-shot Semantic Segmentation [75.18058114915327]
  Generalized Few-Shot Semanticnative (GFSS) aims to extend a segmentation model to novel classes with only a few annotated examples.<n>We propose FewCLIP, a probabilistic prototype calibration framework over multi-modal prototypes from the pretrained CLIP.<n>We show FewCLIP significantly outperforms state-of-the-art approaches across both GFSS and class-incremental setting.
  arXiv  Detail & Related papers  (2025-06-28T18:36:22Z)
• Partially Supervised Unpaired Multi-Modal Learning for Label-Efficient Medical Image Segmentation [53.723234136550055]
  We term the new learning paradigm as Partially Supervised Unpaired Multi-Modal Learning (PSUMML)<n>We propose a novel Decomposed partial class adaptation with snapshot Ensembled Self-Training (DEST) framework for it.<n>Our framework consists of a compact segmentation network with modality specific normalization layers for learning with partially labeled unpaired multi-modal data.
  arXiv  Detail & Related papers  (2025-03-07T07:22:42Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Semi-supervised Semantic Segmentation with Prototype-based Consistency Regularization [20.4183741427867]
  Semi-supervised semantic segmentation requires the model to propagate the label information from limited annotated images to unlabeled ones. A challenge for such a per-pixel prediction task is the large intra-class variation. We propose a novel approach to regularize the distribution of within-class features to ease label propagation difficulty.
  arXiv  Detail & Related papers  (2022-10-10T01:38:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.