DEL: Digit Entropy Loss for Numerical Learning of Large Language Models

• URL: http://arxiv.org/abs/2605.20369v1
• Date: Tue, 19 May 2026 18:18:59 GMT
• Title: DEL: Digit Entropy Loss for Numerical Learning of Large Language Models
• Abstract summary: Number prediction is a fundamental capability of large language models (LLMs) in mathematical problem-solving and code generation.<n>We present Digit Entropy Loss (DEL) for auto-regressive numerical learning.<n>Our formulation can incorporate integers, decimals, and decimal points, expanding the learning objective from a single digit to the floating-point number domain.
• Score: 63.37357414497361
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Number prediction stands as a fundamental capability of large language models (LLMs) in mathematical problem-solving and code generation. The widely adopted maximum likelihood estimation (MLE) for LLM training is not tailored to number prediction. Recently, penalty-driven approaches, e.g., Number Token Loss and Discretized Distance Loss, introduce an inductive bias of numerical distance but induce over-sharpened and over-flattened digit distributions, respectively. In this paper, we make an in-depth analysis on LLM numerical learning, and show that existing numerical learning methods conceptually follow a criterion-distance formulation, where the criterion term represents optimization pattern and the distance term instills geometric prior. Consequently, we present Digit Entropy Loss (DEL) for auto-regressive numerical learning, which reformulates the conventional unsupervised entropy optimization in three key designs: leveraging digit conditional probability and binary cross-entropy to guide the entropy optimization into a supervised manner; deprecating the distance term to bypass the issue of numerical distance; and generalizing the integer-based numerical learning to floating-point number optimization, enabling more accurate number prediction. Our DEL formulation can incorporate integers, decimals, and decimal points, expanding the learning objective from a single digit to the floating-point number domain. Experiments conducted on seven mathematical reasoning benchmarks with four representative LLMs, including CodeLlama, Mistral, DeepSeek, and Qwen-2.5, demonstrate that DEL consistently outperforms its counterparts in both overall prediction accuracy and numerical distance. Source codes are at https://github.com/PolyU-VCLab/DEL

Related papers

• When can isotropy help adapt LLMs' next word prediction to numerical domains? [53.98633183204453]
  It is shown that the isotropic property of LLM embeddings in contextual embedding space preserves the underlying structure of representations.<n> Experiments show that different characteristics of numerical data and model architectures have different impacts on isotropy.
  arXiv  Detail & Related papers  (2025-05-22T05:10:34Z)
• Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges [0.0]
  We introduce GSM-Ranges, a dataset generator that systematically perturbs numerical values in math problems to assess model robustness across varying numerical scales.<n>We also propose a novel grading methodology that distinguishes between logical and non-logical errors, offering a more precise evaluation of reasoning processes beyond computational accuracy.
  arXiv  Detail & Related papers  (2025-02-12T09:53:10Z)
• Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models [2.5464748274973026]
  We present a regression-like loss that operates purely on token level.<n>Our proposed Number Token Loss (NTL) comes in two flavors and minimizes either the $L_p$ norm or the Wasserstein distance.<n>We evaluate the proposed scheme on various mathematical datasets and find that it consistently improves performance in math-related tasks.
  arXiv  Detail & Related papers  (2024-11-04T13:43:24Z)
• DNA-SE: Towards Deep Neural-Nets Assisted Semiparametric Estimation [39.48526221316346]
  Semiparametric statistics play a pivotal role in a wide range of domains, including but not limited to missing data, causal inference, and transfer learning. We develop a scalable algorithm called Deep Neural-Nets Assisted Semiparametric Estimation (DNA-SE) by leveraging the universal approximation property of Deep Neural-Nets (DNN) to streamline semiparametric procedures.
  arXiv  Detail & Related papers  (2024-08-04T14:45:26Z)
• Graph-Structured Speculative Decoding [52.94367724136063]
  Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models. We introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. We observe a remarkable speedup of 1.73$times$ to 1.96$times$, significantly surpassing standard speculative decoding.
  arXiv  Detail & Related papers  (2024-07-23T06:21:24Z)
• Minimally Supervised Learning using Topological Projections in Self-Organizing Maps [55.31182147885694]
  We introduce a semi-supervised learning approach based on topological projections in self-organizing maps (SOMs) Our proposed method first trains SOMs on unlabeled data and then a minimal number of available labeled data points are assigned to key best matching units (BMU) Our results indicate that the proposed minimally supervised model significantly outperforms traditional regression techniques.
  arXiv  Detail & Related papers  (2024-01-12T22:51:48Z)
• Uncertainty quantification for deep learning-based schemes for solving high-dimensional backward stochastic differential equations [5.883258964010963]
  We study uncertainty quantification (UQ) for a class of deep learning-based BSDE schemes. We develop a UQ model that efficiently estimates the STD of the approximate solution using only a single run of the algorithm. Our numerical experiments show that the UQ model produces reliable estimates of the mean and STD of the approximate solution.
  arXiv  Detail & Related papers  (2023-10-05T09:00:48Z)
• Towards Constituting Mathematical Structures for Learning to Optimize [101.80359461134087]
  A technique that utilizes machine learning to learn an optimization algorithm automatically from data has gained arising attention in recent years. A generic L2O approach parameterizes the iterative update rule and learns the update direction as a black-box network. While the generic approach is widely applicable, the learned model can overfit and may not generalize well to out-of-distribution test sets. We propose a novel L2O model with a mathematics-inspired structure that is broadly applicable and generalized well to out-of-distribution problems.
  arXiv  Detail & Related papers  (2023-05-29T19:37:28Z)
• On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats [3.408623772471354]
  The convergence of single-precision Gradient Descent (SGD) is normally aligned with the theoretical results of real numbers. However, the numerical error increases when the computations are performed in low-precision number formats. This provides compelling reasons to study the SGD convergence adapted for low-precision computations.
  arXiv  Detail & Related papers  (2023-01-04T14:54:15Z)
• NumGPT: Improving Numeracy Ability of Generative Pre-trained Models [59.931394234642816]
  We propose NumGPT, a generative pre-trained model that explicitly models the numerical properties of numbers in texts. Specifically, it leverages a prototype-based numeral embedding to encode the mantissa of the number and an individual embedding to encode the exponent of the number. A numeral-aware loss function is designed to integrate numerals into the pre-training objective of NumGPT.
  arXiv  Detail & Related papers  (2021-09-07T15:06:12Z)
• Large-scale Neural Solvers for Partial Differential Equations [48.7576911714538]
  Solving partial differential equations (PDE) is an indispensable part of many branches of science as many processes can be modelled in terms of PDEs. Recent numerical solvers require manual discretization of the underlying equation as well as sophisticated, tailored code for distributed computing. We examine the applicability of continuous, mesh-free neural solvers for partial differential equations, physics-informed neural networks (PINNs) We discuss the accuracy of GatedPINN with respect to analytical solutions -- as well as state-of-the-art numerical solvers, such as spectral solvers.
  arXiv  Detail & Related papers  (2020-09-08T13:26:51Z)
• The Seven-League Scheme: Deep learning for large time step Monte Carlo simulations of stochastic differential equations [0.0]
  We propose an accurate data-driven numerical scheme to solve Differential Equations (SDEs) The SDE discretization is built up by means of a chaos expansion method on the basis of accurately determined (SC) points. With a method called the compression-decompression and collocation technique, we can drastically reduce the number of neural network functions that have to be learned.
  arXiv  Detail & Related papers  (2020-09-07T16:06:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.