GOMA: Toward Structure-Driven Multimodal Alignment from a Graph Signal Smoothing Perspective

• URL: http://arxiv.org/abs/2605.15723v1
• Date: Fri, 15 May 2026 08:21:22 GMT
• Title: GOMA: Toward Structure-Driven Multimodal Alignment from a Graph Signal Smoothing Perspective
• Abstract summary: We propose a structure-driven post-alignment framework that views frozen multimodal embeddings as graph signals.<n>GOMA decouples three key design choices: where messages should flow, how multimodal evidence should propagate, and which smoothing depth should be retained.<n>All experiments follow a transductive MAG retrieval protocol where the graph serves only as unlabeled context and diagonal self-pair edges are removed.
• Score: 39.44851011793204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal alignment is commonly learned from isolated image-text pairs via CLIP-style dual encoders, leaving the relational context among entities largely unused. Multimodal attributed graphs (MAGs), where nodes carry multimodal attributes and edges encode corpus structure, provide a natural setting for refining frozen vision-language embeddings. This refinement is challenging: visual, textual, and cross-modal relations often induce different neighborhood geometries, while unrestricted graph propagation can quickly over-smooth retrieval representations. Effectively leveraging graph context therefore requires simultaneously breaking modality-specific topological barriers, controlling the smoothing regime, and preserving informative smoothing before semantic boundaries collapse. We propose Graph-Optimized Multimodal Alignment (GOMA), a structure-driven post-alignment framework that views frozen multimodal embeddings as graph signals and addresses these requirements through a unified retrieval-oriented design. GOMA decouples three key design choices: where messages should flow, how multimodal evidence should propagate, and which smoothing depth should be retained. Concretely, it learns modality-aware propagation operators, performs finite-step coupled smoothing without diagonal cross-modal shortcuts, and adaptively reads out node-specific smoothing trajectories to preserve useful smoothing before collapse. All experiments follow a transductive MAG retrieval protocol where the graph serves only as unlabeled context and diagonal self-pair edges are removed. On seven MAG benchmarks, GOMA achieves state-of-the-art or tied state-of-the-art retrieval and remains substantially more stable than the strongest graph competitor, demonstrating that MAG structure can serve as an effective post-encoder for frozen multimodal embeddings.

Related papers

• Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework [35.35423006939282]
  FedGAMMA casts federated multimodal graph foundation learning as a two-stage semantic-structural alignment problem.<n>During pre-training, a shared-private semantic enhancer disentangles cross-modal commonality from modality-specific information.<n>During fine-tuning, FedGAMMA adapts the pretrained encoder through lightweight graph-aware prompts.
  arXiv  Detail & Related papers  (2026-07-17T06:57:41Z)
• Context-aware Modality-Topology Co-Alignment for Multimodal Attributed Graphs [8.922335341563363]
  MultimodalAttributed Graphs (MAGs) model real-world entities by coupling graph topology with heterogeneous attributes such as text and images.<n>Existing MAG methods often rely on fixed graph contexts or uniformly fused representations, causing task-agnostic propagation and over-compressed fusion.<n>We propose CoMAG, a unified MAG that learns task-adaptive reliable contexts and modality-preserving alignment within them.
  arXiv  Detail & Related papers  (2026-06-12T06:53:46Z)
• SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphs [38.400117494307615]
  Multimodal-attributed graphs (MAGs) couple graph topology with node semantics from text, images, and other modalities.<n>We propose SMGFM, a spectral multimodal graph pretraining framework that decomposes each modality-specific node signal into graph-frequency bands.<n>SMGFM achieves state-of-the-art performance across graph-level and modality-level tasks.
  arXiv  Detail & Related papers  (2026-06-11T03:54:35Z)
• STAGE: Tackling Semantic Drift in Multimodal Federated Graph Learning [48.96526409527749]
  Federated graph learning (FGL) enables collaborative training on graph data across multiple clients.<n>The key challenge is that clients from different modality domains may not share a common semantic space.<n>We propose textbfSTAGE, a protocol-first framework for MM-FGL.
  arXiv  Detail & Related papers  (2026-05-12T10:35:43Z)
• ColGraphRAG: Late-Interaction Evidence Retrieval for Multimodal GraphRAG [0.0]
  We evaluate replacing the visual candidate-ranking operator over graph-linked image nodes with late-interaction MaxSim-style multi-vector scoring in the ColBERT/ColPali lineage.<n>On MultimodalQA, this change is associated with improved retrieval-stage point estimates for graph-linked image candidates and downstream QA gains.
  arXiv  Detail & Related papers  (2026-05-09T08:30:18Z)
• Robust Multimodal Recommendation via Graph Retrieval-Enhanced Modality Completion [71.20734649881258]
  Real-world multimodal datasets often suffer from modality incompleteness due to sensor failures, annotation scarcity, or privacy constraints.<n>One effective solution is modality completion, which reconstructs missing features to provide modality-complete graphs for downstream tasks.<n>We propose GRE-MC, a Graph Retrieval-Enhanced Modality Completion framework, to overcome these limitations.
  arXiv  Detail & Related papers  (2026-05-01T13:50:52Z)
• Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso [12.238577096609761]
  We propose Cross-Modal Graphical Lasso (CM-GLasso) for learning interpretable multimodal representations.<n>By coupling a novel text-visualization strategy with a unified vision-language encoder, we strictly align multimodal features into a shared latent space.<n>We show that CM-GLasso establishes a new state-of-the-art in generative classification and dense semantic segmentation tasks.
  arXiv  Detail & Related papers  (2026-04-05T04:07:59Z)
• TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution [12.027339810975079]
  Task-aware Modality and Topology co-Evolution (TMTE) is a novel MGL framework that jointly and iteratively optimize graph topology and multimodal representations toward the target task.<n>TMTE consistently achieves state-of-the-art performance across 6 graph-centric and modality-centric tasks.
  arXiv  Detail & Related papers  (2026-03-29T14:54:59Z)
• Mario: Multimodal Graph Reasoning with Large Language Models [10.232888977666418]
  Mario is a graph-conditioned VLM that refines textual and visual features through fine-grained cross-modal contrastive learning.<n>Mario consistently outperforms state-of-the-art graph models in both supervised and zero-shot scenarios for node classification and link prediction.
  arXiv  Detail & Related papers  (2026-03-05T13:49:41Z)
• Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models [84.78794648147608]
  A persistent geometric anomaly, the Modality Gap, remains.<n>Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions.<n>We propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap into stable biases and anisotropic residuals.<n>We then introduce ReAlign, a training-free modality alignment strategy.
  arXiv  Detail & Related papers  (2026-02-02T13:59:39Z)
• OptiMAG: Structure-Semantic Alignment via Unbalanced Optimal Transport [37.640303159988015]
  Multimodal Attributed Graphs (MAGs) have been widely adopted for modeling complex systems by integrating multi-modal information, such as text and images, on nodes.<n>We identify a discrepancy between the implicit semantic structure induced by different modality embeddings and the explicit graph structure.<n>Since existing methods typically perform message passing over the fixed explicit graph structure, they inadvertently aggregate dissimilar features.<n>We propose OptiMAG, an Unbalanced Optimal Transport-based regularization framework.
  arXiv  Detail & Related papers  (2026-01-30T11:29:03Z)
• Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds [49.95082206008502]
  Alignment across Trees is a method that constructs and aligns tree-like hierarchical features for both image and text modalities.<n>We introduce a semantic-aware visual feature extraction framework that applies a cross-attention mechanism to visual class tokens from intermediate Transformer layers.
  arXiv  Detail & Related papers  (2025-10-31T11:32:15Z)
• Learning Noise-Resilient and Transferable Graph-Text Alignment via Dynamic Quality Assessment [19.204800655283744]
  Pre-training Graph Foundation Models (GFMs) on text-attributed graphs (TAGs) is central to web-scale applications such as search, recommendation, and knowledge discovery.<n> existing CLIP-style graph-text amplifies face two key limitations: they assume strict one-to-one correspondences between nodes and texts, and they rely on static alignment objectives that cannot adapt to varying data quality, making them brittle under noisy supervision.<n>We propose ADAligner, a quality-aware graphtext alignment framework that dynamically adjusts between expressive many-to-many and conservative one-to-one objectives according to supervision quality
  arXiv  Detail & Related papers  (2025-10-22T09:01:17Z)
• Graph Optimal Transport for Cross-Domain Alignment [121.80313648519203]
  Cross-domain alignment is fundamental to computer vision and natural language processing. We propose Graph Optimal Transport (GOT), a principled framework that germinates from recent advances in Optimal Transport (OT) Experiments show consistent outperformance of GOT over baselines across a wide range of tasks.
  arXiv  Detail & Related papers  (2020-06-26T01:14:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.