Laying Anchors: Semantically Priming Numerals in Language Modeling

• URL: http://arxiv.org/abs/2404.01536v2
• Date: Wed, 7 Aug 2024 22:46:04 GMT
• Title: Laying Anchors: Semantically Priming Numerals in Language Modeling
• Authors: Mandar Sharma, Rutuja Murlidhar Taware, Pravesh Koirala, Nikhil Muralidhar, Naren Ramakrishnan,
• Abstract summary: We introduce strategies to semantically prime numerals in any corpus by generating anchors governed by the distribution of numerals in said corpus. We demonstrate significant improvements in the mathematical grounding of our learned embeddings.
• Score: 11.831883526217942
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Off-the-shelf pre-trained language models have become the de facto standard in NLP pipelines for a multitude of downstream tasks. However, the inability of these models to properly encode numerals limits their performance on tasks requiring numeric comprehension. We introduce strategies to semantically prime numerals in any corpus by generating anchors governed by the distribution of numerals in said corpus, thereby enabling mathematically grounded representations of these numeral tokens. We establish the superiority of our proposed techniques through evaluation on a range of numeracy tasks for both in-domain (seen) and out-domain (unseen) numerals. Further, we expand our empirical evaluations to numerals ranging from 1 to 10 billion, a significantly broader range compared to previous studies of the same nature, and we demonstrate significant improvements in the mathematical grounding of our learned embeddings.

Related papers

• Interleaving Text and Number Embeddings to Solve Mathemathics Problems [0.0]
  We build upon a recent approach by introducing more expressive numerical embeddings. Our method addresses key shortcomings, including the elimination of numerical artefacts and the ability to handle a wide range of magnitudes without clipping.
  arXiv  Detail & Related papers  (2024-10-25T07:21:57Z)
• MathPile: A Billion-Token-Scale Pretraining Corpus for Math [45.163340937419214]
  We introduce MathPile, a diverse and high-quality math-centric corpus comprising about 9.5 billion tokens. Our meticulous data collection and processing efforts included a complex suite of preprocessing. We aim for our MathPile to boost language models' mathematical reasoning abilities and open-source its different versions and processing scripts to advance the field.
  arXiv  Detail & Related papers  (2023-12-28T16:55:40Z)
• Estimating Numbers without Regression [30.79061214333164]
  Despite recent successes in language models, their ability to represent numbers is insufficient. Subword tokenization fails to explicitly capture magnitude by splitting numbers into arbitrary chunks. We show that changing the model's vocabulary instead (eg introduce a new token for numbers in range 10-100) is a far better trade-off.
  arXiv  Detail & Related papers  (2023-10-09T23:07:05Z)
• FERMAT: An Alternative to Accuracy for Numerical Reasoning [11.893004722079557]
  numerical reasoning is measured using a single score on existing datasets. We introduce a multi-view evaluation set for numerical reasoning in English, called FERMAT. FerMAT evaluates models on various key numerical reasoning aspects such as number understanding, mathematical operations, and training dependency.
  arXiv  Detail & Related papers  (2023-05-27T15:00:45Z)
• Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systems [42.782260686177395]
  We propose a novel method to elicit and exploit the numerical reasoning knowledge hidden in pre-trained language models. We first leverage simple numbers as anchors to probe the implicitly inferred arithmetic expressions from language models. We transform and formulate the task as an analytically solvable linear system.
  arXiv  Detail & Related papers  (2022-10-11T00:57:19Z)
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization [94.4409074435894]
  We propose a novel and effective fine-tuning framework, named Layerwise Noise Stability Regularization (LNSR) Specifically, we propose to inject the standard Gaussian noise and regularize hidden representations of the fine-tuned model. We demonstrate the advantages of the proposed method over other state-of-the-art algorithms including L2-SP, Mixout and SMART.
  arXiv  Detail & Related papers  (2022-06-12T04:42:49Z)
• Arithmetic-Based Pretraining -- Improving Numeracy of Pretrained Language Models [67.48894919842576]
  State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require numeracy. We propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly addresses both in one extended pretraining step. Our experiments show the effectiveness of Arithmetic-Based Pretraining in three different tasks that require improved numeracy.
  arXiv  Detail & Related papers  (2022-05-13T16:10:13Z)
• Number Entity Recognition [65.80137628972312]
  Numbers are essential components of text, like any other word tokens, from which natural language processing (NLP) models are built and deployed. In this work, we attempt to tap this potential of state-of-the-art NLP models and transfer their ability to boost performance in related tasks. Our proposed classification of numbers into entities helps NLP models perform well on several tasks, including a handcrafted Fill-In-The-Blank (FITB) task and on question answering using joint embeddings.
  arXiv  Detail & Related papers  (2022-05-07T05:22:43Z)
• Benchmarking Generalization via In-Context Instructions on 1,600+ Language Tasks [95.06087720086133]
  Natural-Instructions v2 is a collection of 1,600+ diverse language tasks and their expert written instructions. The benchmark covers 70+ distinct task types, such as tagging, in-filling, and rewriting. This benchmark enables large-scale evaluation of cross-task generalization of the models.
  arXiv  Detail & Related papers  (2022-04-16T03:12:30Z)
• NumGPT: Improving Numeracy Ability of Generative Pre-trained Models [59.931394234642816]
  We propose NumGPT, a generative pre-trained model that explicitly models the numerical properties of numbers in texts. Specifically, it leverages a prototype-based numeral embedding to encode the mantissa of the number and an individual embedding to encode the exponent of the number. A numeral-aware loss function is designed to integrate numerals into the pre-training objective of NumGPT.
  arXiv  Detail & Related papers  (2021-09-07T15:06:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.