To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graph

• URL: http://arxiv.org/abs/2101.06397v1
• Date: Sat, 16 Jan 2021 08:12:03 GMT
• Title: To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graph
• Authors: Sufeng Duan, Hai Zhao, Rui Wang
• Abstract summary: We propose a unified explanation of representation for layer-aware neural sequence encoders. Our proposed MoG explanation allows to precisely observe every step of the generation of representation. We also propose a graph-based self-attention network empowered Graph-Transformer.
• Score: 45.51774590045651
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose a unified explanation of representation for layer-aware neural sequence encoders, which regards the representation as a revisited multigraph called multi-order-graph (MoG), so that model encoding can be viewed as a processing to capture all subgraphs in MoG. The relationship reflected by Multi-order-graph, called $n$-order dependency, can present what existing simple directed graph explanation cannot present. Our proposed MoG explanation allows to precisely observe every step of the generation of representation, put diverse relationship such as syntax into a unifiedly depicted framework. Based on the proposed MoG explanation, we further propose a graph-based self-attention network empowered Graph-Transformer by enhancing the ability of capturing subgraph information over the current models. Graph-Transformer accommodates different subgraphs into different groups, which allows model to focus on salient subgraphs. Result of experiments on neural machine translation tasks show that the MoG-inspired model can yield effective performance improvement.

Related papers

• Isomorphic-Consistent Variational Graph Auto-Encoders for Multi-Level Graph Representation Learning [9.039193854524763]
  We propose the Isomorphic-Consistent VGAE (IsoC-VGAE) for task-agnostic graph representation learning. We first devise a decoding scheme to provide a theoretical guarantee of keeping the isomorphic consistency. We then propose the Inverse Graph Neural Network (Inv-GNN) decoder as its intuitive realization.
  arXiv  Detail & Related papers  (2023-12-09T10:16:53Z)
• Discrete Graph Auto-Encoder [52.50288418639075]
  We introduce a new framework named Discrete Graph Auto-Encoder (DGAE) We first use a permutation-equivariant auto-encoder to convert graphs into sets of discrete latent node representations. In the second step, we sort the sets of discrete latent representations and learn their distribution with a specifically designed auto-regressive model.
  arXiv  Detail & Related papers  (2023-06-13T12:40:39Z)
• Deep Graph Neural Networks via Flexible Subgraph Aggregation [50.034313206471694]
  Graph neural networks (GNNs) can learn from graph-structured data and learn the representation of nodes through aggregating neighborhood information. In this paper, we evaluate the expressive power of GNNs from the perspective of subgraph aggregation. We propose a sampling-based node-level residual module (SNR) that can achieve a more flexible utilization of different hops of subgraph aggregation.
  arXiv  Detail & Related papers  (2023-05-09T12:03:42Z)
• Combining Stochastic Explainers and Subgraph Neural Networks can Increase Expressivity and Interpretability [12.526174412246107]
  Subgraph-enhanced graph neural networks (SGNN) can increase the power of the standard message-passing framework. We introduce a novel framework that jointly predicts the class of the graph and a set of explanatory sparse subgraphs.
  arXiv  Detail & Related papers  (2023-04-14T14:21:20Z)
• Gradient Gating for Deep Multi-Rate Learning on Graphs [62.25886489571097]
  We present Gradient Gating (G$2$), a novel framework for improving the performance of Graph Neural Networks (GNNs) Our framework is based on gating the output of GNN layers with a mechanism for multi-rate flow of message passing information across nodes of the underlying graph.
  arXiv  Detail & Related papers  (2022-10-02T13:19:48Z)
• Convolutional Learning on Multigraphs [153.20329791008095]
  We develop convolutional information processing on multigraphs and introduce convolutional multigraph neural networks (MGNNs) To capture the complex dynamics of information diffusion within and across each of the multigraph's classes of edges, we formalize a convolutional signal processing model. We develop a multigraph learning architecture, including a sampling procedure to reduce computational complexity. The introduced architecture is applied towards optimal wireless resource allocation and a hate speech localization task, offering improved performance over traditional graph neural networks.
  arXiv  Detail & Related papers  (2022-09-23T00:33:04Z)
• Soft-mask: Adaptive Substructure Extractions for Graph Neural Networks [40.64326531965043]
  A graph neural network should be able to efficiently extract task-relevant structures and be invariant to irrelevant parts. In this work, we propose to learn graph representations from a sequence of subgraphs of the original graph to better capture task-relevant substructures or hierarchical structures and skip $noisy$ parts. The soft-mask GNN layer is not limited by the fixed sample or drop ratio, and therefore is more flexible to extract subgraphs with arbitrary sizes.
  arXiv  Detail & Related papers  (2022-06-11T11:04:23Z)
• Equivariant Subgraph Aggregation Networks [23.26140936226352]
  This paper proposes a novel framework called Equivariant Subgraph Aggregation Networks (ESAN) to address this issue. While two graphs may not be distinguishable by an MPNN, they often contain distinguishable subgraphs. We develop novel variants of the 1-dimensional Weisfeiler-Leman (1-WL) test for graph isomorphism, and prove lower bounds on the expressiveness of ESAN. We provide theoretical results that describe how design choices such as the subgraph selection policy and equivariant neural architecture affect our architecture's expressive power.
  arXiv  Detail & Related papers  (2021-10-06T16:45:07Z)
• Multiresolution Graph Variational Autoencoder [11.256959274636724]
  We propose Multiresolution Graph Networks (MGN) and Multiresolution Graph Variational Autoencoders (MGVAE) At each resolution level, MGN employs higher order message passing to encode the graph while learning to partition it into mutually exclusive clusters and coarsening into a lower resolution. MGVAE constructs a hierarchical generative model based on MGN to variationally autoencode the hierarchy of coarsened graphs.
  arXiv  Detail & Related papers  (2021-06-02T06:28:47Z)
• Graph-to-Sequence Neural Machine Translation [79.0617920270817]
  We propose a graph-based SAN-based NMT model called Graph-Transformer. Subgraphs are put into different groups according to their orders, and every group of subgraphs respectively reflect different levels of dependency between words. Our method can effectively boost the Transformer with an improvement of 1.1 BLEU points on WMT14 English-German dataset and 1.0 BLEU points on IWSLT14 German-English dataset.
  arXiv  Detail & Related papers  (2020-09-16T06:28:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.