Extracting Definienda in Mathematical Scholarly Articles with Transformers

• URL: http://arxiv.org/abs/2311.12448v1
• Date: Tue, 21 Nov 2023 08:58:57 GMT
• Title: Extracting Definienda in Mathematical Scholarly Articles with Transformers
• Authors: Shufan Jiang (VALDA), Pierre Senellart (DI-ENS, VALDA)
• Abstract summary: We consider automatically identifying the defined term within a mathematical definition from the text of an academic article. It is possible to reach high levels of precision and recall using either recent (and expensive) GPT 4 or simpler pre-trained models fine-tuned on our task.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider automatically identifying the defined term within a mathematical definition from the text of an academic article. Inspired by the development of transformer-based natural language processing applications, we pose the problem as (a) a token-level classification task using fine-tuned pre-trained transformers; and (b) a question-answering task using a generalist large language model (GPT). We also propose a rule-based approach to build a labeled dataset from the LATEX source of papers. Experimental results show that it is possible to reach high levels of precision and recall using either recent (and expensive) GPT 4 or simpler pre-trained models fine-tuned on our task.

Related papers

• In-Context Learning with Representations: Contextual Generalization of Trained Transformers [66.78052387054593]
  In-context learning (ICL) refers to a capability of pretrained large language models, which can learn a new task given a few examples during inference. This paper investigates the training dynamics of transformers by gradient descent through the lens of non-linear regression tasks.
  arXiv  Detail & Related papers  (2024-08-19T16:47:46Z)
• Limits of Transformer Language Models on Learning to Compose Algorithms [77.2443883991608]
  We evaluate training LLaMA models and prompting GPT-4 and Gemini on four tasks demanding to learn a composition of several discrete sub-tasks. Our results indicate that compositional learning in state-of-the-art Transformer language models is highly sample inefficient.
  arXiv  Detail & Related papers  (2024-02-08T16:23:29Z)
• Transformer Based Implementation for Automatic Book Summarization [0.0]
  Document Summarization is the procedure of generating a meaningful and concise summary of a given document. This work is an attempt to use Transformer based techniques for Abstract generation.
  arXiv  Detail & Related papers  (2023-01-17T18:18:51Z)
• Paragraph-based Transformer Pre-training for Multi-Sentence Inference [99.59693674455582]
  We show that popular pre-trained transformers perform poorly when used for fine-tuning on multi-candidate inference tasks. We then propose a new pre-training objective that models the paragraph-level semantics across multiple input sentences.
  arXiv  Detail & Related papers  (2022-05-02T21:41:14Z)
• BERT got a Date: Introducing Transformers to Temporal Tagging [4.651578365545765]
  We present a transformer encoder-decoder model using the RoBERTa language model as our best performing system. Our model surpasses previous works in temporal tagging and type classification, especially on rare classes.
  arXiv  Detail & Related papers  (2021-09-30T08:54:21Z)
• Hidden Markov Based Mathematical Model dedicated to Extract Ingredients from Recipe Text [0.0]
  Partof-speech tagging (POS tagging) is a pre-processing task that requires an annotated corpus. I performed a mathematical model based on Hidden Markov structures and obtained a high-level accuracy of ingredients extracted from text recipe.
  arXiv  Detail & Related papers  (2021-09-28T14:38:11Z)
• Matching with Transformers in MELT [1.2891210250935146]
  We provide an easy to use implementation in the MELT framework which is suited for ontology and knowledge graph matching. We show that a transformer-based filter helps to choose the correct correspondences given a high-recall alignment.
  arXiv  Detail & Related papers  (2021-09-15T16:07:43Z)
• Pretrained Transformers as Universal Computation Engines [105.00539596788127]
  We investigate the capability of a transformer pretrained on natural language to generalize to other modalities with minimal finetuning. We study finetuning it on a variety of sequence classification tasks spanning numerical computation, vision, and protein fold prediction. We find that such pretraining enables FPT to generalize in zero-shot to these modalities, matching the performance of a transformer fully trained on these tasks.
  arXiv  Detail & Related papers  (2021-03-09T06:39:56Z)
• Teach me how to Label: Labeling Functions from Natural Language with Text-to-text Transformers [0.5330240017302619]
  This paper focuses on the task of turning natural language descriptions into Python labeling functions. We follow a novel approach to semantic parsing with pre-trained text-to-text Transformers. Our approach can be regarded as a stepping stone towards models that are taught how to label in natural language.
  arXiv  Detail & Related papers  (2021-01-18T16:04:15Z)
• Exploring Software Naturalness through Neural Language Models [56.1315223210742]
  The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks.
  arXiv  Detail & Related papers  (2020-06-22T21:56:14Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.