LARGO: Low-Rank Hypernetwork for Handling Missing Modalities

• URL: http://arxiv.org/abs/2605.06086v1
• Date: Thu, 07 May 2026 12:07:55 GMT
• Title: LARGO: Low-Rank Hypernetwork for Handling Missing Modalities
• Abstract summary: We propose LARGO, a hypernetwork that compresses the $2N-1$ dedicated missing-modality models into a single network.<n>Our method ranks first in 47 out of 52 configurations, achieving average Dice improvements of +0.68$%$ and +2.53$%$ over state-of-the-art baselines.<n>A proof-of-concept experiment on avMNIST suggests that LARGO may extend beyond medical imaging to heterogeneous non-medical modalities.
• Score: 41.99844472131922
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Addressing missing modalities is an important challenge in multimodal image analysis and often relies on complex architectures that do not transfer easily to different datasets without architectural modifications or hyperparameter tuning. While most existing methods tackle this problem in feature space by engineering representations that are robust to missing inputs, we instead operate in weight space. We propose LARGO, a hypernetwork that compresses the $2^N-1$ dedicated missing-modality models into a single network by modelling the convolutional weights using the Canonical Polyadic (CP) tensor decomposition. Extensive experimental validation on BraTS 2018 (4 modalities, 15 scenarios) and ISLES 2022 (3 modalities, 7 scenarios) shows that our method ranks first in 47 out of 52 configurations, achieving average Dice improvements of +0.68$\%$ and +2.53$\%$ over state-of-the-art baselines (mmFormer, M$^{3}$AE, ShaSpec, SimMLM). A proof-of-concept experiment on avMNIST suggests that LARGO may extend beyond medical imaging to heterogeneous non-medical modalities.

Related papers

• Linearizing Vision Transformer with Test-Time Training [54.52616866374482]
  Inheriting weights from pretrained Transformers provides an appealing shortcut, yet the fundamental representational gap between Softmax and linear attention prevents effective weight transfer.<n>We identify Test-Time Training (TTT) as a linear-complexity architecture whose two-layer dynamic formulation is structurally aligned with Softmax attention.<n>We validate our approach by linearizing Stable Diffusion 3.5 and introduce SD3.5-T$5$ (Transformer To Test Time Training)
  arXiv  Detail & Related papers  (2026-05-04T16:16:26Z)
• $M^3-Verse$: A "Spot the Difference" Challenge for Large Multimodal Models [26.923709172476958]
  We introduce $M3-Verse$, a Multi-Modal, Multi-State, Multi-Dimensional benchmark.<n>It is built upon paired videos that provide multi-perspective observations of an indoor scene before and after a state change.<n>We evaluate 16 state-of-the-art LMMs and observe their limitations in tracking state transitions.
  arXiv  Detail & Related papers  (2025-12-21T13:50:26Z)
• Multimodal Federated Learning With Missing Modalities through Feature Imputation Network [9.384737026881504]
  Multimodal federated learning holds immense potential for collaboratively training models from multiple sources without sharing raw data.<n>Previous methods typically rely on publicly available real datasets or synthetic data to compensate for missing modalities.<n>We propose a novel, lightweight, low-dimensional feature translator to reconstruct bottleneck features of the missing modalities.
  arXiv  Detail & Related papers  (2025-05-26T17:11:03Z)
• UniF$^2$ace: A Unified Fine-grained Face Understanding and Generation Model [62.66515621965686]
  We introduce a novel theoretical framework with a Dual Discrete Diffusion (D3Diff) loss, unifying masked generative models with discrete score matching diffusion.<n>This D3Diff significantly enhances the model's ability to synthesize high-fidelity facial details aligned with text input.<n>We construct UniF$2$aceD-1M, a large-scale dataset comprising 130K fine-grained image-caption pairs and 1M visual question-answering pairs.
  arXiv  Detail & Related papers  (2025-03-11T07:34:59Z)
• MINIMA: Modality Invariant Image Matching [52.505282811925454]
  We present MINIMA, a unified image matching framework for multiple cross-modal cases.<n>We scale up the modalities from cheap but rich RGB-only matching data, by means of generative models.<n>With MD-syn, we can directly train any advanced matching pipeline on randomly selected modality pairs to obtain cross-modal ability.
  arXiv  Detail & Related papers  (2024-12-27T02:39:50Z)
• $M^3EL$: A Multi-task Multi-topic Dataset for Multi-modal Entity Linking [11.334577756093923]
  We propose a dataset construction pipeline and publish $M3EL$, a large-scale dataset for MEL. $M3EL$ includes 79,625 instances, covering 9 diverse multi-modal tasks, and 5 different topics. Our dataset effectively addresses these issues, and the $textitCLIP_textitND$ model fine-tuned with $M3EL$ shows a significant improvement in accuracy.
  arXiv  Detail & Related papers  (2024-10-08T10:52:23Z)
• Heterogenous Ensemble of Models for Molecular Property Prediction [55.91865861896012]
  We propose a method for considering different modalities on molecules. We ensemble these models with a HuberRegressor. This yields a winning solution to the 2textsuperscriptnd edition of the OGB Large-Scale Challenge (2022)
  arXiv  Detail & Related papers  (2022-11-20T17:25:26Z)
• Exploring Sparse Expert Models and Beyond [51.90860155810848]
  Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost. We propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing. This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models.
  arXiv  Detail & Related papers  (2021-05-31T16:12:44Z)
• Brain Image Synthesis with Unsupervised Multivariate Canonical CSC$\ell_4$Net [122.8907826672382]
  We propose to learn dedicated features that cross both intre- and intra-modal variations using a novel CSC$ell_4$Net.
  arXiv  Detail & Related papers  (2021-03-22T05:19:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.