NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning

• URL: http://arxiv.org/abs/2404.00459v2
• Date: Thu, 26 Sep 2024 09:54:57 GMT
• Title: NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning
• Authors: Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, Assaf Arbelle,
• Abstract summary: Language models struggle with handling numerical data and performing arithmetic operations. We propose a simple adjustment to how numbers are represented by including the count of digits before each number. By requiring the model to consider the number of digits first, it enhances the reasoning process before generating the actual number.
• Score: 27.584258258635945
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models struggle with handling numerical data and performing arithmetic operations. We hypothesize that this limitation can be partially attributed to non-intuitive textual numbers representation. When a digit is read or generated by a causal language model it does not know its place value (e.g. thousands vs. hundreds) until the entire number is processed. To address this issue, we propose a simple adjustment to how numbers are represented by including the count of digits before each number. For instance, instead of "42", we suggest using "{2:42}" as the new format. This approach, which we term NumeroLogic, offers an added advantage in number generation by serving as a Chain of Thought (CoT). By requiring the model to consider the number of digits first, it enhances the reasoning process before generating the actual number. We use arithmetic tasks to demonstrate the effectiveness of the NumeroLogic formatting. We further demonstrate NumeroLogic applicability to general natural language modeling, improving language understanding performance in the MMLU benchmark.

Related papers

• FoNE: Precise Single-Token Number Embeddings via Fourier Features [51.17846016593835]
  We propose a novel method that maps numbers into the embedding space with their Fourier features. FoNE encodes each number as a single token with only two embedding dimensions per digit, effectively capturing numerical values without fragmentation. On 6-digit decimal addition, FoNE requires 64$times$ less data to achieve 99% accuracy than subword and digit-wise embeddings. FoNE is the only method that yields 100% accuracy on over 100,000 test examples for addition, subtraction, and multiplication.
  arXiv  Detail & Related papers  (2025-02-13T19:54:59Z)
• Number Cookbook: Number Understanding of Language Models and How to Improve It [63.9542740221096]
  Large language models (LLMs) can solve an increasing number of complex reasoning tasks while making surprising mistakes in basic numerical understanding and processing. This paper comprehensively investigates the numerical understanding and processing ability (NUPA) of LLMs.
  arXiv  Detail & Related papers  (2024-11-06T08:59:44Z)
• Language Models Encode Numbers Using Digit Representations in Base 10 [12.913172023910203]
  We show that large language models (LLMs) make errors when handling simple numerical problems. LLMs internally represent numbers with individual circular representations per-digit in base 10. This digit-wise representation sheds light on the error patterns of models on tasks involving numerical reasoning.
  arXiv  Detail & Related papers  (2024-10-15T17:00:15Z)
• Positional Description Matters for Transformers Arithmetic [58.4739272381373]
  Transformers often falter on arithmetic tasks despite their vast capabilities. We propose several ways to fix the issue, either by modifying the positional encoding directly, or by modifying the representation of the arithmetic task to leverage standard positional encoding differently.
  arXiv  Detail & Related papers  (2023-11-22T00:31:01Z)
• Estimating Numbers without Regression [30.79061214333164]
  Despite recent successes in language models, their ability to represent numbers is insufficient. Subword tokenization fails to explicitly capture magnitude by splitting numbers into arbitrary chunks. We show that changing the model's vocabulary instead (eg introduce a new token for numbers in range 10-100) is a far better trade-off.
  arXiv  Detail & Related papers  (2023-10-09T23:07:05Z)
• FERMAT: An Alternative to Accuracy for Numerical Reasoning [11.893004722079557]
  numerical reasoning is measured using a single score on existing datasets. We introduce a multi-view evaluation set for numerical reasoning in English, called FERMAT. FerMAT evaluates models on various key numerical reasoning aspects such as number understanding, mathematical operations, and training dependency.
  arXiv  Detail & Related papers  (2023-05-27T15:00:45Z)
• PAL: Program-aided Language Models [112.94785609781503]
  We present Program-Aided Language models (PaL) to understand natural language problems. PaL offloads the solution step to a programmatic runtime such as a Python interpreter. We set new state-of-the-art results in all 12 benchmarks.
  arXiv  Detail & Related papers  (2022-11-18T18:56:13Z)
• Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systems [42.782260686177395]
  We propose a novel method to elicit and exploit the numerical reasoning knowledge hidden in pre-trained language models. We first leverage simple numbers as anchors to probe the implicitly inferred arithmetic expressions from language models. We transform and formulate the task as an analytically solvable linear system.
  arXiv  Detail & Related papers  (2022-10-11T00:57:19Z)
• NumGPT: Improving Numeracy Ability of Generative Pre-trained Models [59.931394234642816]
  We propose NumGPT, a generative pre-trained model that explicitly models the numerical properties of numbers in texts. Specifically, it leverages a prototype-based numeral embedding to encode the mantissa of the number and an individual embedding to encode the exponent of the number. A numeral-aware loss function is designed to integrate numerals into the pre-training objective of NumGPT.
  arXiv  Detail & Related papers  (2021-09-07T15:06:12Z)
• Investigating the Limitations of the Transformers with Simple Arithmetic Tasks [10.23804850480924]
  We find that how a number is represented in its surface form has a strong influence on the model's accuracy. We conclude that modern pretrained language models can easily learn arithmetic from very few examples.
  arXiv  Detail & Related papers  (2021-02-25T17:22:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.