Structural Adapters in Pretrained Language Models for AMR-to-text Generation

• URL: http://arxiv.org/abs/2103.09120v1
• Date: Tue, 16 Mar 2021 15:06:50 GMT
• Title: Structural Adapters in Pretrained Language Models for AMR-to-text Generation
• Authors: Leonardo F. R. Ribeiro, Yue Zhang, Iryna Gurevych
• Abstract summary: Previous work on text generation from graph-structured data relies on pretrained language models (PLMs) We propose StructAdapt, an adapter method to encode graph structure into PLMs.
• Score: 59.50420985074769
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Previous work on text generation from graph-structured data relies on pretrained language models (PLMs) and utilizes graph linearization heuristics rather than explicitly considering the graph structure. Efficiently encoding the graph structure in PLMs is challenging because they were pretrained on natural language, and modeling structured data may lead to catastrophic forgetting of distributional knowledge. In this paper, we propose StructAdapt, an adapter method to encode graph structure into PLMs. Contrary to prior work, StructAdapt effectively models interactions among the nodes based on the graph connectivity, only training graph structure-aware adapter parameters. In this way, we avoid catastrophic forgetting while maintaining the topological structure of the graph. We empirically show the benefits of explicitly encoding graph structure into PLMs using adapters and achieve state-of-the-art results on two AMR-to-text datasets, training only 5.1% of the PLM parameters.

Related papers

• Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrasting [50.181824673039436]
  We propose a Graph Structure Self-Contrasting (GSSC) framework that learns graph structural information without message passing. The proposed framework is based purely on Multi-Layer Perceptrons (MLPs), where the structural information is only implicitly incorporated as prior knowledge. It first applies structural sparsification to remove potentially uninformative or noisy edges in the neighborhood, and then performs structural self-contrasting in the sparsified neighborhood to learn robust node representations.
  arXiv  Detail & Related papers  (2024-09-09T12:56:02Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• GraphEdit: Large Language Models for Graph Structure Learning [62.618818029177355]
  Graph Structure Learning (GSL) focuses on capturing intrinsic dependencies and interactions among nodes in graph-structured data. Existing GSL methods heavily depend on explicit graph structural information as supervision signals. We propose GraphEdit, an approach that leverages large language models (LLMs) to learn complex node relationships in graph-structured data.
  arXiv  Detail & Related papers  (2024-02-23T08:29:42Z)
• Let Your Graph Do the Talking: Encoding Structured Data for LLMs [22.358472780103057]
  We introduce a parameter-efficient method to explicitly represent structured data for large language models (LLMs) Our method, GraphToken, learns an encoding function to extend prompts with explicit structured information. We show that explicitly representing the graph structure allows significant improvements to graph reasoning tasks.
  arXiv  Detail & Related papers  (2024-02-08T17:51:44Z)
• Unifying Structured Data as Graph for Data-to-Text Pre-Training [69.96195162337793]
  Data-to-text (D2T) generation aims to transform structured data into natural language text. Data-to-text pre-training has proved to be powerful in enhancing D2T generation. We propose a structure-enhanced pre-training method for D2T generation by designing a structure-enhanced Transformer.
  arXiv  Detail & Related papers  (2024-01-02T12:23:49Z)
• GraphGLOW: Universal and Generalizable Structure Learning for Graph Neural Networks [72.01829954658889]
  This paper introduces the mathematical definition of this novel problem setting. We devise a general framework that coordinates a single graph-shared structure learner and multiple graph-specific GNNs. The well-trained structure learner can directly produce adaptive structures for unseen target graphs without any fine-tuning.
  arXiv  Detail & Related papers  (2023-06-20T03:33:22Z)
• Investigating the Effect of Relative Positional Embeddings on AMR-to-Text Generation with Structural Adapters [5.468547489755107]
  We investigate the influence of Relative Position Embeddings (RPE) on AMR-to-Text generation. Through ablation studies, graph attack and link prediction, we reveal that RPE might be partially encoding input graphs. We suggest further research regarding the role of RPE will provide valuable insights for Graph-to-Text generation.
  arXiv  Detail & Related papers  (2023-02-12T12:43:36Z)
• Graph Pre-training for AMR Parsing and Generation [14.228434699363495]
  We investigate graph self-supervised training to improve structure awareness of PLMs over AMR graphs. We introduce two graph auto-encoding strategies for graph-to-graph pre-training and four tasks to integrate text and graph information during pre-training.
  arXiv  Detail & Related papers  (2022-03-15T12:47:00Z)
• Stage-wise Fine-tuning for Graph-to-Text Generation [25.379346921398326]
  Graph-to-text generation has benefited from pre-trained language models (PLMs) in achieving better performance than structured graph encoders. We propose a structured graph-to-text model with a two-step fine-tuning mechanism which first fine-tunes model on Wikipedia before adapting to the graph-to-text generation.
  arXiv  Detail & Related papers  (2021-05-17T17:15:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.