Two Facets of the Same Optimization Coin: Model Degradation and Representation Collapse in Graph Foundation Models

• URL: http://arxiv.org/abs/2509.08401v4
• Date: Fri, 19 Sep 2025 06:07:23 GMT
• Title: Two Facets of the Same Optimization Coin: Model Degradation and Representation Collapse in Graph Foundation Models
• Authors: Xunkai Li, Daohan Su, Sicheng Liu, Ru Zhang, Zhenjun Li, Bing Zhou, Rong-Hua Li, Guoren Wang,
• Abstract summary: GFMs are designed to learn the optimal embedding functions from multi-domain text-attributed graphs.<n> graph VQ-MAE stands out among the increasingly diverse landscape of GFM.<n>Despite its potential, domain generalization conflicts cause imperceptible pitfalls.
• Score: 41.39154912912367
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Inspired by the success of LLMs, GFMs are designed to learn the optimal embedding functions from multi-domain text-attributed graphs for the downstream cross-task generalization capability. Among the diverse architectures, graph VQ-MAE stands out among the increasingly diverse landscape of GFM. This is attributed to its ability to jointly encode topology and textual attributes from multiple domains into discrete embedding spaces with clear semantic boundaries. Despite its potential, domain generalization conflicts cause imperceptible pitfalls. In this paper, we instantiate two of them, and they are just like two sides of the same GFM optimization coin - Side 1 Model Degradation: The encoder and codebook fail to capture the diversity of inputs; Side 2 Representation Collapse: The hidden embedding and codebook vector fail to preserve semantic separability due to constraints from narrow representation subspaces. These two pitfalls (sides) collectively impair the decoder and generate the low-quality reconstructed supervision, causing the GFM optimization dilemma during pre-training (coin). Through empirical investigation, we attribute the above challenges to Information Bottleneck and Regularization Deficit. To address them, we propose MoT - (1) Information Tinker for Two Pitfalls, which utilizes an edge-wise semantic fusion strategy and a mixture-of-codebooks with domain-aware routing to improve information capacity. (2) Regularization Tinker for Optimization Coin, which utilizes two additional regularizations to further improve gradient supervision in our proposed Information Tinker. Notably, as a flexible architecture, MoT adheres to the scaling laws of GFM, offering a controllable model scale. Compared to SOTA baselines, experiments on 22 datasets across 6 domains demonstrate that MoT achieves significant improvements in supervised, few-shot, and zero-shot scenarios.

Related papers

• Toward Effective Multimodal Graph Foundation Model: A Divide-and-Conquer Based Approach [42.970648490410504]
  Multimodal Graph Foundation Models (MGFMs) allow for leveraging the rich multimodal information in Multimodal-Attributed Graphs (MAGs)<n>We propose PLANET, a novel framework employing a Divide-and-Conquer strategy to decouple modality interaction and alignment across distinct granularities.<n>We show that PLANET significantly outperforms state-of-the-art baselines across diverse graph-centric and multimodal generative tasks.
  arXiv  Detail & Related papers  (2026-02-04T01:05:12Z)
• RAG-GFM: Overcoming In-Memory Bottlenecks in Graph Foundation Models via Retrieval-Augmented Generation [27.59455285600957]
  Graph Foundation Models (GFMs) have emerged as a frontier in graph learning, which are expected to deliver transferable representations across diverse tasks.<n>We propose RAG-GFM, a Retrieval-Augmented Generation aided Graph Foundation Model that offloads knowledge from parameters.<n>We show that RAG-GFM consistently outperforms 13 state-of-the-art baselines in both cross-domain node and graph classification.
  arXiv  Detail & Related papers  (2026-01-21T16:02:43Z)
• GRASP: Geospatial pixel Reasoning viA Structured Policy learning [16.023628299873494]
  GRASP is a structured policy-learning framework that integrates a multimodal large language model with a pretrained segmentation model in a cascaded manner.<n> PRIME is a training paradigm that replaces supervised fine-tuning with reinforcement learning to better align reasoning and grounding behaviors with task objectives.<n>We release GRASP-1k, a fully out-of-domain benchmark with reasoning-intensive queries, reasoning traces, and fine-grained masks.
  arXiv  Detail & Related papers  (2025-08-23T18:05:06Z)
• Fast State-Augmented Learning for Wireless Resource Allocation with Dual Variable Regression [83.27791109672927]
  We show how a state-augmented graph neural network (GNN) parametrization for the resource allocation policy circumvents the drawbacks of the ubiquitous dual subgradient methods.<n>Lagrangian maximizing state-augmented policies are learned during the offline training phase.<n>We prove a convergence result and an exponential probability bound on the excursions of the dual function (iterate) optimality gaps.
  arXiv  Detail & Related papers  (2025-06-23T15:20:58Z)
• Towards Effective Federated Graph Foundation Model via Mitigating Knowledge Entanglement [9.87623531653534]
  Graph foundation models (GFM) offer strong domain generalization but are usually trained on single machines.<n>We propose FedGFM, a novel decentralized GFM training paradigm.<n>Key challenge is knowledge entanglement, where multi-domain knowledge merges into indistinguishable representations.<n>We present FedGFM+, an enhanced framework with two core modules to reduce knowledge entanglement.
  arXiv  Detail & Related papers  (2025-05-19T04:06:32Z)
• Dual-stream Feature Augmentation for Domain Generalization [16.495752769624872]
  We propose a Dual-stream Feature Augmentation(DFA) method by constructing some hard features from two perspectives. Our approach could achieve state-of-the-art performance for domain generalization.
  arXiv  Detail & Related papers  (2024-09-07T03:41:05Z)
• DGMamba: Domain Generalization via Generalized State Space Model [80.82253601531164]
  Domain generalization(DG) aims at solving distribution shift problems in various scenes. Mamba, as an emerging state space model (SSM), possesses superior linear complexity and global receptive fields. We propose a novel framework for DG, named DGMamba, that excels in strong generalizability toward unseen domains.
  arXiv  Detail & Related papers  (2024-04-11T14:35:59Z)
• UGMAE: A Unified Framework for Graph Masked Autoencoders [67.75493040186859]
  We propose UGMAE, a unified framework for graph masked autoencoders. We first develop an adaptive feature mask generator to account for the unique significance of nodes. We then design a ranking-based structure reconstruction objective joint with feature reconstruction to capture holistic graph information.
  arXiv  Detail & Related papers  (2024-02-12T19:39:26Z)
• Semi-Supervised Domain Generalization with Stochastic StyleMatch [90.98288822165482]
  In real-world applications, we might have only a few labels available from each source domain due to high annotation cost. In this work, we investigate semi-supervised domain generalization, a more realistic and practical setting. Our proposed approach, StyleMatch, is inspired by FixMatch, a state-of-the-art semi-supervised learning method based on pseudo-labeling.
  arXiv  Detail & Related papers  (2021-06-01T16:00:08Z)
• Supervised Domain Adaptation using Graph Embedding [86.3361797111839]
  Domain adaptation methods assume that distributions between the two domains are shifted and attempt to realign them. We propose a generic framework based on graph embedding. We show that the proposed approach leads to a powerful Domain Adaptation framework.
  arXiv  Detail & Related papers  (2020-03-09T12:25:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.