Transformers Meet Directed Graphs

• URL: http://arxiv.org/abs/2302.00049v3
• Date: Thu, 31 Aug 2023 14:38:57 GMT
• Title: Transformers Meet Directed Graphs
• Authors: Simon Geisler, Yujia Li, Daniel Mankowitz, Ali Taylan Cemgil, Stephan G\"unnemann, Cosmin Paduraru
• Abstract summary: Transformers for directed graphs are a surprisingly underexplored topic, despite their applicability to ubiquitous domains. In this work, we propose two direction- and structure-aware positional encodings for directed graphs. We show that the extra directionality information is useful in various downstream tasks, including correctness testing of sorting networks and source code understanding.
• Score: 18.490890946129284
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers were originally proposed as a sequence-to-sequence model for text but have become vital for a wide range of modalities, including images, audio, video, and undirected graphs. However, transformers for directed graphs are a surprisingly underexplored topic, despite their applicability to ubiquitous domains, including source code and logic circuits. In this work, we propose two direction- and structure-aware positional encodings for directed graphs: (1) the eigenvectors of the Magnetic Laplacian - a direction-aware generalization of the combinatorial Laplacian; (2) directional random walk encodings. Empirically, we show that the extra directionality information is useful in various downstream tasks, including correctness testing of sorting networks and source code understanding. Together with a data-flow-centric graph construction, our model outperforms the prior state of the art on the Open Graph Benchmark Code2 relatively by 14.7%.

Related papers

• Graph Transformers Dream of Electric Flow [72.06286909236827]
  We show that the linear Transformer, when applied to graph data, can implement algorithms that solve canonical problems. We present explicit weight configurations for implementing each such graph algorithm, and we bound the errors of the constructed Transformers by the errors of the underlying algorithms.
  arXiv  Detail & Related papers  (2024-10-22T05:11:45Z)
• Graph as Point Set [31.448841287258116]
  This paper introduces a novel graph-to-set conversion method that transforms interconnected nodes into a set of independent points. It enables using set encoders to learn from graphs, thereby significantly expanding the design space of Graph Neural Networks. To demonstrate the effectiveness of our approach, we introduce Point Set Transformer (PST), a transformer architecture that accepts a point set converted from a graph as input.
  arXiv  Detail & Related papers  (2024-05-05T02:29:41Z)
• Graph Transformers without Positional Encodings [0.7252027234425334]
  We introduce Eigenformer, a Graph Transformer employing a novel spectrum-aware attention mechanism cognizant of the Laplacian spectrum of the graph. We empirically show that it achieves performance competetive with SOTA Graph Transformers on a number of standard GNN benchmarks.
  arXiv  Detail & Related papers  (2024-01-31T12:33:31Z)
• Gramformer: Learning Crowd Counting via Graph-Modulated Transformer [68.26599222077466]
  Gramformer is a graph-modulated transformer to enhance the network by adjusting the attention and input node features respectively. A feature-based encoding is proposed to discover the centrality positions or importance of nodes. Experiments on four challenging crowd counting datasets have validated the competitiveness of the proposed method.
  arXiv  Detail & Related papers  (2024-01-08T13:01:54Z)
• Discrete Graph Auto-Encoder [52.50288418639075]
  We introduce a new framework named Discrete Graph Auto-Encoder (DGAE) We first use a permutation-equivariant auto-encoder to convert graphs into sets of discrete latent node representations. In the second step, we sort the sets of discrete latent representations and learn their distribution with a specifically designed auto-regressive model.
  arXiv  Detail & Related papers  (2023-06-13T12:40:39Z)
• Graph Inductive Biases in Transformers without Message Passing [47.238185813842996]
  New Graph Inductive bias Transformer (GRIT) incorporates graph inductive biases without using message passing. GRIT achieves state-of-the-art empirical performance across a variety of graph datasets.
  arXiv  Detail & Related papers  (2023-05-27T22:26:27Z)
• Transformers over Directed Acyclic Graphs [6.263470141349622]
  We study transformers over directed acyclic graphs (DAGs) and propose architecture adaptations tailored to DAGs. We show that it is effective in making graph transformers generally outperform graph neural networks tailored to DAGs and in improving SOTA graph transformer performance in terms of both quality and efficiency.
  arXiv  Detail & Related papers  (2022-10-24T12:04:52Z)
• Transformer for Graphs: An Overview from Architecture Perspective [86.3545861392215]
  It's imperative to sort out the existing Transformer models for graphs and systematically investigate their effectiveness on various graph tasks. We first disassemble the existing models and conclude three typical ways to incorporate the graph information into the vanilla Transformer. Our experiments confirm the benefits of current graph-specific modules on Transformer and reveal their advantages on different kinds of graph tasks.
  arXiv  Detail & Related papers  (2022-02-17T06:02:06Z)
• GraphiT: Encoding Graph Structure in Transformers [37.33808493548781]
  We show that viewing graphs as sets of node features and structural and positional information is able to outperform representations learned with classical graph neural networks (GNNs) Our model, GraphiT, encodes such information by (i) leveraging relative positional encoding strategies in self-attention scores based on positive definite kernels on graphs, and (ii) enumerating and encoding local sub-structures such as paths of short length.
  arXiv  Detail & Related papers  (2021-06-10T11:36:22Z)
• Do Transformers Really Perform Bad for Graph Representation? [62.68420868623308]
  We present Graphormer, which is built upon the standard Transformer architecture. Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
  arXiv  Detail & Related papers  (2021-06-09T17:18:52Z)
• A Generalization of Transformer Networks to Graphs [5.736353542430439]
  We introduce a graph transformer with four new properties compared to the standard model. The architecture is extended to edge feature representation, which can be critical to tasks s.a. chemistry (bond type) or link prediction (entity relationship in knowledge graphs)
  arXiv  Detail & Related papers  (2020-12-17T16:11:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.