Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso

• URL: http://arxiv.org/abs/2604.03953v1
• Date: Sun, 05 Apr 2026 04:07:59 GMT
• Title: Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso
• Abstract summary: We propose Cross-Modal Graphical Lasso (CM-GLasso) for learning interpretable multimodal representations.<n>By coupling a novel text-visualization strategy with a unified vision-language encoder, we strictly align multimodal features into a shared latent space.<n>We show that CM-GLasso establishes a new state-of-the-art in generative classification and dense semantic segmentation tasks.
• Score: 12.238577096609761
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning interpretable multimodal representations inherently relies on uncovering the conditional dependencies between heterogeneous features. However, sparse graph estimation techniques, such as Graphical Lasso (GLasso), to visual-linguistic domains is severely bottlenecked by high-dimensional noise, modality misalignment, and the confounding of shared versus category-specific topologies. In this paper, we propose Cross-Modal Graphical Lasso (CM-GLasso) that overcomes these fundamental limitations. By coupling a novel text-visualization strategy with a unified vision-language encoder, we strictly align multimodal features into a shared latent space. We introduce a cross-attention distillation mechanism that condenses high-dimensional patches into explicit semantic nodes, naturally extracting spatial-aware cross-modal priors. Furthermore, we unify tailored GLasso estimation and Common-Specific Structure Learning (CSSL) into a joint objective optimized via the Alternating Direction Method of Multiplier (ADMM). This formulation guarantees the simultaneous disentanglement of invariant and class-specific precision matrices without multi-step error accumulation. Extensive experiments across eight benchmarks covering both natural and medical domains demonstrate that CM-GLasso establishes a new state-of-the-art in generative classification and dense semantic segmentation tasks.

Related papers

• Towards Effective Federated Multimodal Graph Learning via Navigating Multifaceted Heterogeneity [12.526349151730921]
  We propose Federated multimodal graph learning with Topology-aware Cross-modal Routing (FedTCR)<n>FedTCR employs a two-stage paradigm that comprises federated task-agnostic pre-training followed by isolated task-oriented fine-tuning.<n> Experiments across 7 domains demonstrate that FedTCR outperforms state-of-the-art baselines on both graph-centric and modality-centric tasks.
  arXiv  Detail & Related papers  (2026-08-01T12:29:38Z)
• Anisotropic Modality Align [91.23979617826926]
  Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data.<n>Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data.<n>The core obstacle lies in the persistent Modality Gap in the shared space.
  arXiv  Detail & Related papers  (2026-05-08T14:53:24Z)
• Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning [12.074231630572067]
  We propose Topology-Aware Multimodal Representation Alignment (ToMA) for vision-language learning.<n>ToMA identifies salient edges and aligns them across modalities through available cross-modal correspondences.<n> Experiments show that ToMA yields stable gains, with clear improvements on remote sensing and modest but consistent benefits on fashion retrieval.
  arXiv  Detail & Related papers  (2026-04-29T07:30:33Z)
• PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation [58.1914505657064]
  This paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the challenge of knowledge interference between class-level semantics and spatial context.<n>Experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.
  arXiv  Detail & Related papers  (2026-03-18T09:26:43Z)
• AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models [21.682989096955467]
  AG-VAS (Anchor-Guided Visual Anomaly) is a new framework that expands the LMM vocabulary with three learnable semantic anchor tokens.<n>AG-VAS achieves consistent state-of-the-art performance in the zero-shot setting.
  arXiv  Detail & Related papers  (2026-03-01T22:25:23Z)
• Delving into Spectral Clustering with Vision-Language Representations [27.433418706301477]
  We propose Neural Tangent Kernel Spectral Clustering that leverages cross-modal alignment in pre-trained vision-language models.<n>We show that this formulation amplifies within-cluster connections while suppressing spurious ones across clusters.<n>Our method consistently outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2026-02-10T09:36:24Z)
• Toward Effective Multimodal Graph Foundation Model: A Divide-and-Conquer Based Approach [42.970648490410504]
  Multimodal Graph Foundation Models (MGFMs) allow for leveraging the rich multimodal information in Multimodal-Attributed Graphs (MAGs)<n>We propose PLANET, a novel framework employing a Divide-and-Conquer strategy to decouple modality interaction and alignment across distinct granularities.<n>We show that PLANET significantly outperforms state-of-the-art baselines across diverse graph-centric and multimodal generative tasks.
  arXiv  Detail & Related papers  (2026-02-04T01:05:12Z)
• Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models [84.78794648147608]
  A persistent geometric anomaly, the Modality Gap, remains.<n>Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions.<n>We propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap into stable biases and anisotropic residuals.<n>We then introduce ReAlign, a training-free modality alignment strategy.
  arXiv  Detail & Related papers  (2026-02-02T13:59:39Z)
• Multi-label Classification with Panoptic Context Aggregation Networks [61.82285737410154]
  This paper introduces the Deep Panoptic Context Aggregation Network (PanCAN), a novel approach that hierarchically integrates multi-order geometric contexts.<n>PanCAN learns multi-order neighborhood relationships at each scale by combining random walks with an attention mechanism.<n>Experiments on NUS-WIDE, PASCAL VOC,2007, and MS-COCO benchmarks demonstrate that PanCAN consistently achieves competitive results.
  arXiv  Detail & Related papers  (2025-12-29T14:16:21Z)
• Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds [49.95082206008502]
  Alignment across Trees is a method that constructs and aligns tree-like hierarchical features for both image and text modalities.<n>We introduce a semantic-aware visual feature extraction framework that applies a cross-attention mechanism to visual class tokens from intermediate Transformer layers.
  arXiv  Detail & Related papers  (2025-10-31T11:32:15Z)
• Revisiting Self-Supervised Heterogeneous Graph Learning from Spectral Clustering Perspective [52.662463893268225]
  Self-supervised heterogeneous graph learning (SHGL) has shown promising potential in diverse scenarios.<n>Existing SHGL methods encounter two significant limitations.<n>We introduce a novel framework enhanced by rank and dual consistency constraints.
  arXiv  Detail & Related papers  (2024-12-01T09:33:20Z)
• Improving Anomaly Segmentation with Multi-Granularity Cross-Domain Alignment [17.086123737443714]
  Anomaly segmentation plays a pivotal role in identifying atypical objects in images, crucial for hazard detection in autonomous driving systems. While existing methods demonstrate noteworthy results on synthetic data, they often fail to consider the disparity between synthetic and real-world data domains. We introduce the Multi-Granularity Cross-Domain Alignment framework, tailored to harmonize features across domains at both the scene and individual sample levels.
  arXiv  Detail & Related papers  (2023-08-16T22:54:49Z)
• Consistency and Diversity induced Human Motion Segmentation [231.36289425663702]
  We propose a novel Consistency and Diversity induced human Motion (CDMS) algorithm. Our model factorizes the source and target data into distinct multi-layer feature spaces. A multi-mutual learning strategy is carried out to reduce the domain gap between the source and target data.
  arXiv  Detail & Related papers  (2022-02-10T06:23:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.