Transformers from Diffusion: A Unified Framework for Neural Message Passing

• URL: http://arxiv.org/abs/2409.09111v4
• Date: Tue, 01 Jul 2025 01:04:17 GMT
• Title: Transformers from Diffusion: A Unified Framework for Neural Message Passing
• Authors: Qitian Wu, David Wipf, Junchi Yan,
• Abstract summary: Message passing neural networks (MPNNs) have become a de facto class of model solutions.<n>We propose an energy-constrained diffusion model, which integrates the inductive bias of diffusion with layer-wise constraints of energy.<n>Building on these insights, we devise a new class of message passing models, dubbed Transformers (DIFFormer), whose global attention layers are derived from the principled energy-constrained diffusion framework.
• Score: 79.9193447649011
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning representations for structured data with certain geometries (e.g., observed or unobserved) is a fundamental challenge, wherein message passing neural networks (MPNNs) have become a de facto class of model solutions. In this paper, inspired by physical systems, we propose an energy-constrained diffusion model, which integrates the inductive bias of diffusion on manifolds with layer-wise constraints of energy minimization. We identify that the diffusion operators have a one-to-one correspondence with the energy functions implicitly descended by the diffusion process, and the finite-difference iteration for solving the energy-constrained diffusion system induces the propagation layers of various types of MPNNs operating on observed or latent structures. This leads to a unified mathematical framework for common neural architectures whose computational flows can be cast as message passing (or its special case), including MLPs, GNNs, and Transformers. Building on these insights, we devise a new class of neural message passing models, dubbed diffusion-inspired Transformers (DIFFormer), whose global attention layers are derived from the principled energy-constrained diffusion framework. Across diverse datasets ranging from real-world networks to images, texts, and physical particles, we demonstrate that the new model achieves promising performance in scenarios where the data structures are observed (as a graph), partially observed, or entirely unobserved.

Related papers

• Inferring Diffusion Structures of Heterogeneous Network Cascade [10.129643823066159]
  We propose a novel model for inferring multi-layer diffusion networks from cascade data.<n>The proposed model represents cascade pathways as a mixture of diffusion networks across different layers.<n>We conduct extensive simulation studies to demonstrate the model's performance in recovering diverse diffusion structures.
  arXiv  Detail & Related papers  (2025-06-23T21:26:19Z)
• KITINet: Kinetics Theory Inspired Network Architectures with PDE Simulation Approaches [43.872190335490515]
  This paper introduces KITINet, a novel architecture that reinterprets feature propagation through the lens of non-equilibrium particle dynamics.<n>At its core, we propose a residual module that models update as the evolution of a particle system.<n>This formulation mimics particle collisions and energy exchange, enabling adaptive feature refinement via physics-informed interactions.
  arXiv  Detail & Related papers  (2025-05-23T13:58:29Z)
• Exploring the flavor structure of leptons via diffusion models [0.0]
  We consider a simple extension of the Standard Model with the type I seesaw mechanism and train a neural network to generate the neutrino mass matrix. By utilizing transfer learning, the diffusion model generates 104 solutions consistent with the neutrino mass squared differences and the leptonic mixing angles.
  arXiv  Detail & Related papers  (2025-03-27T12:17:00Z)
• Predicting Cascading Failures with a Hyperparametric Diffusion Model [66.89499978864741]
  We study cascading failures in power grids through the lens of diffusion models. Our model integrates viral diffusion principles with physics-based concepts. We show that this diffusion model can be learned from traces of cascading failures.
  arXiv  Detail & Related papers  (2024-06-12T02:34:24Z)
• Diffusion models as probabilistic neural operators for recovering unobserved states of dynamical systems [49.2319247825857]
  We show that diffusion-based generative models exhibit many properties favourable for neural operators.<n>We propose to train a single model adaptable to multiple tasks, by alternating between the tasks during training.
  arXiv  Detail & Related papers  (2024-05-11T21:23:55Z)
• An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization [59.63880337156392]
  Diffusion models have achieved tremendous success in computer vision, audio, reinforcement learning, and computational biology. Despite the significant empirical success, theory of diffusion models is very limited. This paper provides a well-rounded theoretical exposure for stimulating forward-looking theories and methods of diffusion models.
  arXiv  Detail & Related papers  (2024-04-11T14:07:25Z)
• Graph Neural Aggregation-diffusion with Metastability [4.040326569845733]
  Continuous graph neural models based on differential equations have expanded the architecture of graph neural networks (GNNs) We propose GRADE inspired by graph aggregation-diffusion equations, which includes the delicate balance between nonlinear diffusion and aggregation induced by interaction potentials. We prove that GRADE achieves competitive performance across various benchmarks and alleviates the over-smoothing issue in GNNs.
  arXiv  Detail & Related papers  (2024-03-29T15:05:57Z)
• Accelerating Inference in Molecular Diffusion Models with Latent Representations of Protein Structure [0.0]
  Diffusion generative models operate directly on 3D molecular structures. We present a novel GNN-based architecture for learning latent representations of molecular structure. Our model achieves comparable performance to one with an all-atom protein representation while exhibiting a 3-fold reduction in inference time.
  arXiv  Detail & Related papers  (2023-11-22T15:32:31Z)
• Supercharging Graph Transformers with Advective Diffusion [28.40109111316014]
  This paper proposes Advective Diffusion Transformer (AdvDIFFormer), a physics-inspired graph Transformer model designed to address this challenge.<n>We show that AdvDIFFormer has provable capability for controlling generalization error with topological shifts.<n> Empirically, the model demonstrates superiority in various predictive tasks across information networks, molecular screening and protein interactions.
  arXiv  Detail & Related papers  (2023-10-10T08:40:47Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance [95.12230117950232]
  We show that a common latent space emerges from two diffusion models trained independently on related domains. Applying CycleDiffusion to text-to-image diffusion models, we show that large-scale text-to-image diffusion models can be used as zero-shot image-to-image editors.
  arXiv  Detail & Related papers  (2022-10-11T15:53:52Z)
• Diffusion Models in Vision: A Survey [80.82832715884597]
  A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage. Diffusion models are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens.
  arXiv  Detail & Related papers  (2022-09-10T22:00:30Z)
• WaveNet-Based Deep Neural Networks for the Characterization of Anomalous Diffusion (WADNet) [0.0]
  Anomalous diffusion is involved in the evolution of physical, chemical, biological, and economic systems. This challenge aims at objectively assessing and comparing new approaches for single trajectory characterization. We develop a WaveNet-based deep neural network (WADNet) by combining a modified WaveNet encoder with long short-term memory networks.
  arXiv  Detail & Related papers  (2021-06-14T19:41:15Z)
• Diffusion Mechanism in Residual Neural Network: Theory and Applications [12.573746641284849]
  In many learning tasks with limited training samples, the diffusion connects the labeled and unlabeled data points. We propose a novel diffusion residual network (Diff-ResNet) internally introduces diffusion into the architectures of neural networks. Under the structured data assumption, it is proved that the proposed diffusion block can increase the distance-diameter ratio that improves the separability of inter-class points.
  arXiv  Detail & Related papers  (2021-05-07T10:42:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.