Evaluating Generative Models for Graph-to-Text Generation

• URL: http://arxiv.org/abs/2307.14712v1
• Date: Thu, 27 Jul 2023 09:03:05 GMT
• Title: Evaluating Generative Models for Graph-to-Text Generation
• Authors: Shuzhou Yuan and Michael F\"arber
• Abstract summary: We explore the capability of generative models to generate descriptive text from graph data in a zero-shot setting. Our results demonstrate that generative models are capable of generating fluent and coherent text. However, our error analysis reveals that generative models still struggle with understanding the semantic relations between entities.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have been widely employed for graph-to-text generation tasks. However, the process of finetuning LLMs requires significant training resources and annotation work. In this paper, we explore the capability of generative models to generate descriptive text from graph data in a zero-shot setting. Specifically, we evaluate GPT-3 and ChatGPT on two graph-to-text datasets and compare their performance with that of finetuned LLM models such as T5 and BART. Our results demonstrate that generative models are capable of generating fluent and coherent text, achieving BLEU scores of 10.57 and 11.08 for the AGENDA and WebNLG datasets, respectively. However, our error analysis reveals that generative models still struggle with understanding the semantic relations between entities, and they also tend to generate text with hallucinations or irrelevant information. As a part of error analysis, we utilize BERT to detect machine-generated text and achieve high macro-F1 scores. We have made the text generated by generative models publicly available.

Related papers

• Detection and Measurement of Syntactic Templates in Generated Text [58.111650675717414]
  We offer an analysis of syntactic features to characterize general repetition in models. We find that models tend to produce templated text in downstream tasks at a higher rate than what is found in human-reference texts.
  arXiv  Detail & Related papers  (2024-06-28T19:34:23Z)
• Prompt-based vs. Fine-tuned LLMs Toward Causal Graph Verification [0.0]
  This work aims toward an application of natural language processing (NLP) technology for automatic verification of causal graphs using text sources. We compare the performance of two types of NLP models: (1) pre-trained language models fine-tuned for causal relation classification task and, (2) prompt-based LLMs.
  arXiv  Detail & Related papers  (2024-05-29T09:06:18Z)
• Parameter-Efficient Tuning Large Language Models for Graph Representation Learning [62.26278815157628]
  We introduce Graph-aware. Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning. We use a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations.
  arXiv  Detail & Related papers  (2024-04-28T18:36:59Z)
• Triples-to-isiXhosa (T2X): Addressing the Challenges of Low-Resource Agglutinative Data-to-Text Generation [9.80836683456026]
  We tackle data-to-text for isiXhosa, which is low-resource and agglutinative. We introduce Triples-to-isiXhosa (T2X), a new dataset based on a subset of WebNLG. We develop an evaluation framework for T2X that measures how accurately generated text describes the data.
  arXiv  Detail & Related papers  (2024-03-12T11:53:27Z)
• Distilling Large Language Models for Text-Attributed Graph Learning [16.447635770220334]
  Text-Attributed Graphs (TAGs) are graphs of connected textual documents. Graph models can efficiently learn TAGs, but their training heavily relies on human-annotated labels. Large language models (LLMs) have recently demonstrated remarkable capabilities in few-shot and zero-shot TAG learning.
  arXiv  Detail & Related papers  (2024-02-19T10:31:53Z)
• Exploiting Contextual Target Attributes for Target Sentiment Classification [53.30511968323911]
  Existing PTLM-based models for TSC can be categorized into two groups: 1) fine-tuning-based models that adopt PTLM as the context encoder; 2) prompting-based models that transfer the classification task to the text/word generation task. We present a new perspective of leveraging PTLM for TSC: simultaneously leveraging the merits of both language modeling and explicit target-context interactions via contextual target attributes.
  arXiv  Detail & Related papers  (2023-12-21T11:45:28Z)
• Empower Text-Attributed Graphs Learning with Large Language Models (LLMs) [5.920353954082262]
  We propose a plug-and-play approach to empower text-attributed graphs through node generation using Large Language Models (LLMs) We employ an edge predictor to capture the structural information inherent in the raw dataset and integrate the newly generated samples into the original graph. Experiments demonstrate the outstanding performance of our proposed paradigm, particularly in low-shot scenarios.
  arXiv  Detail & Related papers  (2023-10-15T16:04:28Z)
• Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphs [4.56877715768796]
  We show that ChatGPT achieves near state-of-the-art performance on some measures of the WebNLG 2020 challenge. We also show that there is a significant connection between what the LLM already knows about the data it is parsing and the quality of the output text.
  arXiv  Detail & Related papers  (2023-07-14T12:45:03Z)
• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [51.90524745663737]
  A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks. Our method achieves state-of-the-art results on well-established TAG datasets. Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
  arXiv  Detail & Related papers  (2023-05-31T03:18:03Z)
• Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training [86.91380874390778]
  We present Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterances and table schemas by leveraging generation models to generate pre-train data. Based on experimental results, neural semantics that leverage GAP MODEL obtain new state-of-the-art results on both SPIDER and CRITERIA-TO-generative benchmarks.
  arXiv  Detail & Related papers  (2020-12-18T15:53:50Z)
• KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation [100.79870384880333]
  We propose a knowledge-grounded pre-training (KGPT) to generate knowledge-enriched text. We adopt three settings, namely fully-supervised, zero-shot, few-shot to evaluate its effectiveness. Under zero-shot setting, our model achieves over 30 ROUGE-L on WebNLG while all other baselines fail.
  arXiv  Detail & Related papers  (2020-10-05T19:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.