To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graph

• URL: http://arxiv.org/abs/2101.06397v1
• Date: Sat, 16 Jan 2021 08:12:03 GMT
• Title: To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graph
• Authors: Sufeng Duan, Hai Zhao, Rui Wang
• Abstract summary: We propose a unified explanation of representation for layer-aware neural sequence encoders. Our proposed MoG explanation allows to precisely observe every step of the generation of representation. We also propose a graph-based self-attention network empowered Graph-Transformer.
• Score: 45.51774590045651
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose a unified explanation of representation for layer-aware neural sequence encoders, which regards the representation as a revisited multigraph called multi-order-graph (MoG), so that model encoding can be viewed as a processing to capture all subgraphs in MoG. The relationship reflected by Multi-order-graph, called $n$-order dependency, can present what existing simple directed graph explanation cannot present. Our proposed MoG explanation allows to precisely observe every step of the generation of representation, put diverse relationship such as syntax into a unifiedly depicted framework. Based on the proposed MoG explanation, we further propose a graph-based self-attention network empowered Graph-Transformer by enhancing the ability of capturing subgraph information over the current models. Graph-Transformer accommodates different subgraphs into different groups, which allows model to focus on salient subgraphs. Result of experiments on neural machine translation tasks show that the MoG-inspired model can yield effective performance improvement.

Related papers

• Graph-Dictionary Signal Model for Sparse Representations of Multivariate Data [49.77103348208835]
  We define a novel Graph-Dictionary signal model, where a finite set of graphs characterizes relationships in data distribution through a weighted sum of their Laplacians. We propose a framework to infer the graph dictionary representation from observed data, along with a bilinear generalization of the primal-dual splitting algorithm to solve the learning problem. We exploit graph-dictionary representations in a motor imagery decoding task on brain activity data, where we classify imagined motion better than standard methods.
  arXiv  Detail & Related papers  (2024-11-08T17:40:43Z)
• Scalable Weibull Graph Attention Autoencoder for Modeling Document Networks [50.42343781348247]
  We develop a graph Poisson factor analysis (GPFA) which provides analytic conditional posteriors to improve the inference accuracy. We also extend GPFA to a multi-stochastic-layer version named graph Poisson gamma belief network (GPGBN) to capture the hierarchical document relationships at multiple semantic levels. Our models can extract high-quality hierarchical latent document representations and achieve promising performance on various graph analytic tasks.
  arXiv  Detail & Related papers  (2024-10-13T02:22:14Z)
• Isomorphic-Consistent Variational Graph Auto-Encoders for Multi-Level Graph Representation Learning [9.039193854524763]
  We propose the Isomorphic-Consistent VGAE (IsoC-VGAE) for task-agnostic graph representation learning. We first devise a decoding scheme to provide a theoretical guarantee of keeping the isomorphic consistency. We then propose the Inverse Graph Neural Network (Inv-GNN) decoder as its intuitive realization.
  arXiv  Detail & Related papers  (2023-12-09T10:16:53Z)
• Combining Stochastic Explainers and Subgraph Neural Networks can Increase Expressivity and Interpretability [12.526174412246107]
  Subgraph-enhanced graph neural networks (SGNN) can increase the power of the standard message-passing framework. We introduce a novel framework that jointly predicts the class of the graph and a set of explanatory sparse subgraphs.
  arXiv  Detail & Related papers  (2023-04-14T14:21:20Z)
• Gradient Gating for Deep Multi-Rate Learning on Graphs [62.25886489571097]
  We present Gradient Gating (G$2$), a novel framework for improving the performance of Graph Neural Networks (GNNs) Our framework is based on gating the output of GNN layers with a mechanism for multi-rate flow of message passing information across nodes of the underlying graph.
  arXiv  Detail & Related papers  (2022-10-02T13:19:48Z)
• Convolutional Learning on Multigraphs [153.20329791008095]
  We develop convolutional information processing on multigraphs and introduce convolutional multigraph neural networks (MGNNs) To capture the complex dynamics of information diffusion within and across each of the multigraph's classes of edges, we formalize a convolutional signal processing model. We develop a multigraph learning architecture, including a sampling procedure to reduce computational complexity. The introduced architecture is applied towards optimal wireless resource allocation and a hate speech localization task, offering improved performance over traditional graph neural networks.
  arXiv  Detail & Related papers  (2022-09-23T00:33:04Z)
• Soft-mask: Adaptive Substructure Extractions for Graph Neural Networks [40.64326531965043]
  A graph neural network should be able to efficiently extract task-relevant structures and be invariant to irrelevant parts. In this work, we propose to learn graph representations from a sequence of subgraphs of the original graph to better capture task-relevant substructures or hierarchical structures and skip $noisy$ parts. The soft-mask GNN layer is not limited by the fixed sample or drop ratio, and therefore is more flexible to extract subgraphs with arbitrary sizes.
  arXiv  Detail & Related papers  (2022-06-11T11:04:23Z)
• Multiresolution Graph Variational Autoencoder [11.256959274636724]
  We propose Multiresolution Graph Networks (MGN) and Multiresolution Graph Variational Autoencoders (MGVAE) At each resolution level, MGN employs higher order message passing to encode the graph while learning to partition it into mutually exclusive clusters and coarsening into a lower resolution. MGVAE constructs a hierarchical generative model based on MGN to variationally autoencode the hierarchy of coarsened graphs.
  arXiv  Detail & Related papers  (2021-06-02T06:28:47Z)
• Graph-to-Sequence Neural Machine Translation [79.0617920270817]
  We propose a graph-based SAN-based NMT model called Graph-Transformer. Subgraphs are put into different groups according to their orders, and every group of subgraphs respectively reflect different levels of dependency between words. Our method can effectively boost the Transformer with an improvement of 1.1 BLEU points on WMT14 English-German dataset and 1.0 BLEU points on IWSLT14 German-English dataset.
  arXiv  Detail & Related papers  (2020-09-16T06:28:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.