Procedural Text Mining with Large Language Models

• URL: http://arxiv.org/abs/2310.03376v1
• Date: Thu, 5 Oct 2023 08:27:33 GMT
• Title: Procedural Text Mining with Large Language Models
• Authors: Anisa Rula and Jennifer D'Souza
• Abstract summary: We tackle the problem of extracting procedures from unstructured PDF text in an incremental question-answering fashion. We leverage the current state-of-the-art GPT-4 (Generative Pre-trained Transformer 4) model, accompanied by two variations of in-context learning. The findings highlight both the promise of this approach and the value of the in-context learning customisations.
• Score: 0.21756081703275998
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Recent advancements in the field of Natural Language Processing, particularly the development of large-scale language models that are pretrained on vast amounts of knowledge, are creating novel opportunities within the realm of Knowledge Engineering. In this paper, we investigate the usage of large language models (LLMs) in both zero-shot and in-context learning settings to tackle the problem of extracting procedures from unstructured PDF text in an incremental question-answering fashion. In particular, we leverage the current state-of-the-art GPT-4 (Generative Pre-trained Transformer 4) model, accompanied by two variations of in-context learning that involve an ontology with definitions of procedures and steps and a limited number of samples of few-shot learning. The findings highlight both the promise of this approach and the value of the in-context learning customisations. These modifications have the potential to significantly address the challenge of obtaining sufficient training data, a hurdle often encountered in deep learning-based Natural Language Processing techniques for procedure extraction.

Related papers

• Scalable Language Model with Generalized Continual Learning [58.700439919096155]
  The Joint Adaptive Re-ization (JARe) is integrated with Dynamic Task-related Knowledge Retrieval (DTKR) to enable adaptive adjustment of language models based on specific downstream tasks. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting.
  arXiv  Detail & Related papers  (2024-04-11T04:22:15Z)
• SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
  We propose a framework to enable in-context learning in large language models. A meta-model can learn on self-supervised prompts consisting of tailored demonstrations. Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
  arXiv  Detail & Related papers  (2023-07-15T08:33:08Z)
• On the cross-lingual transferability of multilingual prototypical models across NLU tasks [2.44288434255221]
  Supervised deep learning-based approaches have been applied to task-oriented dialog and have proven to be effective for limited domain and language applications. In practice, these approaches suffer from the drawbacks of domain-driven design and under-resourced languages. This article proposes to investigate the cross-lingual transferability of using synergistically few-shot learning with prototypical neural networks and multilingual Transformers-based models.
  arXiv  Detail & Related papers  (2022-07-19T09:55:04Z)
• Leveraging pre-trained language models for conversational information seeking from text [2.8425118603312]
  In this paper we investigate the usage of in-context learning and pre-trained language representation models to address the problem of information extraction from process description documents. The results highlight the potential of the approach and the usefulness of the in-context learning customizations.
  arXiv  Detail & Related papers  (2022-03-31T09:00:46Z)
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey [67.82942975834924]
  Large, pre-trained language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches.
  arXiv  Detail & Related papers  (2021-11-01T20:08:05Z)
• A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios [30.391291221959545]
  Deep neural networks and huge language models are becoming omnipresent in natural language applications. As they are known for requiring large amounts of training data, there is a growing body of work to improve the performance in low-resource settings. Motivated by the recent fundamental changes towards neural models and the popular pre-train and fine-tune paradigm, we survey promising approaches for low-resource natural language processing.
  arXiv  Detail & Related papers  (2020-10-23T11:22:01Z)
• Knowledge-Aware Procedural Text Understanding with Multi-Stage Training [110.93934567725826]
  We focus on the task of procedural text understanding, which aims to comprehend such documents and track entities' states and locations during a process. Two challenges, the difficulty of commonsense reasoning and data insufficiency, still remain unsolved. We propose a novel KnOwledge-Aware proceduraL text understAnding (KOALA) model, which effectively leverages multiple forms of external knowledge.
  arXiv  Detail & Related papers  (2020-09-28T10:28:40Z)
• Neural Language Generation: Formulation, Methods, and Evaluation [13.62873478165553]
  Recent advances in neural network-based generative modeling have reignited the hopes in having computer systems capable of seamlessly conversing with humans. High capacity deep learning models trained on large scale datasets demonstrate unparalleled abilities to learn patterns in the data even in the lack of explicit supervision signals. There is no standard way to assess the quality of text produced by these generative models, which constitutes a serious bottleneck towards the progress of the field.
  arXiv  Detail & Related papers  (2020-07-31T00:08:28Z)
• Data Augmentation for Spoken Language Understanding via Pretrained Language Models [113.56329266325902]
  Training of spoken language understanding (SLU) models often faces the problem of data scarcity. We put forward a data augmentation method using pretrained language models to boost the variability and accuracy of generated utterances.
  arXiv  Detail & Related papers  (2020-04-29T04:07:12Z)
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer [64.22926988297685]
  Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP) In this paper, we explore the landscape of introducing transfer learning techniques for NLP by a unified framework that converts all text-based language problems into a text-to-text format.
  arXiv  Detail & Related papers  (2019-10-23T17:37:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.