Text-Guided Visual Dependency Graph Learning with Cross-Modal Attention Priors

• URL: http://arxiv.org/abs/2608.21443v1
• Date: Wed, 19 Aug 2026 09:57:34 GMT
• Title: Text-Guided Visual Dependency Graph Learning with Cross-Modal Attention Priors
• Abstract summary: We propose CM-GLasso, a framework that bridges vision-language representation learning and sparse Gaussian Models.<n> CM-GLasso introduces three key components: (i) a text visualization strategy that renders class-attribute descriptions as images and processes them through the same SigLIP-2 vision encoder as natural images, (ii) a cross-attention distillation mechanism that condenses high-dimensional patches into a small set of semantic graph nodes, and (iii) a joint ADMM formulation that estimates shared and class-specific precision components within a single convex objective.
• Score: 11.338098263634537
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating interpretable conditional-dependence structures from multimodal visual-linguistic features remains largely unexplored. We propose CM-GLasso (Cross-Modal Graphical Lasso), a framework that bridges vision-language representation learning and sparse Gaussian Graphical Models. CM-GLasso introduces three key components: (i) a text visualization strategy that renders class-attribute descriptions as images and processes them through the same SigLIP-2 vision encoder as natural images, yielding prototype-indexed patch-level attention footprints in a shared feature coordinate system; (ii) a cross-attention distillation mechanism that condenses high-dimensional patches into a small set of semantic graph nodes, whose attention-footprint similarities yield cross-modal structural priors for non-uniform L1 penalization; (iii) a joint ADMM formulation that estimates shared and class-specific precision components within a single convex objective, avoiding the need to first estimate and then decompose separate class-wise graphs. The learned sparse graph topologies directly support a parameter-free, precision-based classification rule and a lightweight topology-aware segmentation head. Extensive experiments on eight benchmarks demonstrate that CM-GLasso achieves competitive or superior performance compared with strong feature-based and task-specific baselines. Under the matched controlled protocol, it attains the highest average classification accuracy (91.97%) and the highest segmentation mIoU among the controlled baselines on VOC (74.75%) and ADE20K (64.01%), while also yielding explicit sparse conditional-dependence graphs with common-specific decomposition.

Related papers

• Integrating Implicit and Explicit Relational Biases through Graph-Based Multiple Instance Learning: A Case Study in Skin Lesion Diagnosis [0.0]
  This study investigates a dual-level framework for image classification.<n>It bridges the gap between implicit representation learning and explicit structural modelling.<n> Experimental results show that combining implicit interpatch modelling with explicit graph-based messages yields the best results.
  arXiv  Detail & Related papers  (2026-08-06T13:47:18Z)
• Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso [12.238577096609761]
  We propose Cross-Modal Graphical Lasso (CM-GLasso) for learning interpretable multimodal representations.<n>By coupling a novel text-visualization strategy with a unified vision-language encoder, we strictly align multimodal features into a shared latent space.<n>We show that CM-GLasso establishes a new state-of-the-art in generative classification and dense semantic segmentation tasks.
  arXiv  Detail & Related papers  (2026-04-05T04:07:59Z)
• CrossHGL: A Text-Free Foundation Model for Cross-Domain Heterogeneous Graph Learning [18.84875776859183]
  CrossHGL is a semantic-preserving transformation strategy that homogenizes heterogeneous graphs and encodes interaction semantics into edge features.<n>A prompt-aware multi-domain pre-training framework with a Tri-Prompt mechanism captures transferable knowledge across feature, edge, and structure perspectives.<n>Experiments on node-level and graph-level tasks show that CrossHGL consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-29T13:17:15Z)
• GLASS: Graph and Vision-Language Assisted Semantic Shape Correspondence [16.503835991826367]
  We present GLASS, a framework that bridges the gap by integrating geometric spectral analysis with rich semantic priors from vision-language foundation models.<n> GLASS introduces three key innovations: (i) a view-consistent strategy that enables robust multi-view visual feature extraction from powerful vision foundation models.<n>It achieves average geodesic errors of 0.21, 4.5, and 5.6 on the inter-class benchmark SMAL and non-isometric benchmarks SMAL and TOPKIDS, reducing errors from URSSM baselines of 0.49, 6.0, and 8.9 by 57%, 25%, and 37%, respectively.
  arXiv  Detail & Related papers  (2026-03-08T14:33:50Z)
• Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Generation [11.18352269863283]
  Vision-Language Models (VLMs) have shown promise in generating plotting code from chart images.<n>Existing approaches largely rely on supervised fine-tuning, encouraging surface-level token imitation.<n>We propose Chart Specification, a structured intermediate representation that shifts training from text imitation to semantically grounded supervision.
  arXiv  Detail & Related papers  (2026-02-11T14:08:06Z)
• Graph Structure Refinement with Energy-based Contrastive Learning [56.957793274727514]
  We introduce an unsupervised method based on a joint of generative training and discriminative training to learn graph structure and representation.<n>We propose an Energy-based Contrastive Learning (ECL) guided Graph Structure Refinement (GSR) framework, denoted as ECL-GSR.<n>ECL-GSR achieves faster training with fewer samples and memories against the leading baseline, highlighting its simplicity and efficiency in downstream tasks.
  arXiv  Detail & Related papers  (2024-12-20T04:05:09Z)
• Affinity-Graph-Guided Contractive Learning for Pretext-Free Medical Image Segmentation with Minimal Annotation [55.325956390997]
  This paper proposes an affinity-graph-guided semi-supervised contrastive learning framework (Semi-AGCL) for medical image segmentation. The framework first designs an average-patch-entropy-driven inter-patch sampling method, which can provide a robust initial feature space. With merely 10% of the complete annotation set, our model approaches the accuracy of the fully annotated baseline, manifesting a marginal deviation of only 2.52%.
  arXiv  Detail & Related papers  (2024-10-14T10:44:47Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• Deep Contrastive Graph Learning with Clustering-Oriented Guidance [61.103996105756394]
  Graph Convolutional Network (GCN) has exhibited remarkable potential in improving graph-based clustering. Models estimate an initial graph beforehand to apply GCN. Deep Contrastive Graph Learning (DCGL) model is proposed for general data clustering.
  arXiv  Detail & Related papers  (2024-02-25T07:03:37Z)
• Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation [62.96628432641806]
  Scene Graph Generation aims to first encode the visual contents within the given image and then parse them into a compact summary graph. We first present a novel Stacked Hybrid-Attention network, which facilitates the intra-modal refinement as well as the inter-modal interaction. We then devise an innovative Group Collaborative Learning strategy to optimize the decoder.
  arXiv  Detail & Related papers  (2022-03-18T09:14:13Z)
• Effective and Efficient Graph Learning for Multi-view Clustering [173.8313827799077]
  We propose an effective and efficient graph learning model for multi-view clustering. Our method exploits the view-similar between graphs of different views by the minimization of tensor Schatten p-norm. Our proposed algorithm is time-economical and obtains the stable results and scales well with the data size.
  arXiv  Detail & Related papers  (2021-08-15T13:14:28Z)
• High-Order Information Matters: Learning Relation and Topology for Occluded Person Re-Identification [84.43394420267794]
  We propose a novel framework by learning high-order relation and topology information for discriminative features and robust alignment. Our framework significantly outperforms state-of-the-art by6.5%mAP scores on Occluded-Duke dataset.
  arXiv  Detail & Related papers  (2020-03-18T12:18:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.