A new Uncertainty Principle in Machine Learning

• URL: http://arxiv.org/abs/2603.06634v1
• Date: Tue, 24 Feb 2026 12:48:02 GMT
• Title: A new Uncertainty Principle in Machine Learning
• Abstract summary: Many scientific problems in the context of machine learning can be reduced to the search of answers in appropriate variables.<n>What prevents the use of this simple idea is the fatal degeneracy of the Heaviside and sigmoid expansions.<n>This problem can be formulated as a peculiar uncertainty principle -- the sharper the minimum, the smoother the canyons.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many scientific problems in the context of machine learning can be reduced to the search of polynomial answers in appropriate variables. The Hevisidization of arbitrary polynomial is actually provided by one-and-the same two-layer expression. What prevents the use of this simple idea is the fatal degeneracy of the Heaviside and sigmoid expansions, which traps the steepest-descent evolution at the bottom of canyons, close to the starting point, but far from the desired true minimum. This problem is unavoidable and can be formulated as a peculiar uncertainty principle -- the sharper the minimum, the smoother the canyons. It is a direct analogue of the usual one, which is the pertinent property of the more familiar Fourier expansion. Standard machine learning software fights with this problem empirically, for example, by testing evolutions, originated at randomly distributed starting points and then selecting the best one. Surprisingly or not, phenomena and problems, encountered in ML application to science are pure scientific and belong to physics, not to computer science. On the other hand, they sound slightly different and shed new light on the well-known phenomena -- for example, extend the uncertainty principle from Fourier and, later, wavelet analysis to a new peculiar class of nearly singular sigmoid functions.

Related papers

• Data-driven discovery of governing differential equations across physical systems [61.522420798136984]
  We propose a problem-oriented perspective on data-driven differential equation discovery.<n>We first introduce a two-dimensional phase diagram of equation discoverability.<n>We then present the representation-evaluation-optimization (REO) framework as a fundamental abstraction of the discovery process.
  arXiv  Detail & Related papers  (2026-06-08T15:35:06Z)
• Solving a Machine Learning Regression Problem Based on the Theory of Random Functions [0.0]
  This paper studies a machine learning regression problem as a multivariate approximation problem using the framework of the theory of random functions.<n>An ab initio derivation of a regression method is proposed, starting from postulates of indifference.
  arXiv  Detail & Related papers  (2025-12-14T15:12:18Z)
• Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networks [0.913755431537592]
  We study Singular learning theory (SLT) in toy settings relevant to interpretability and phase transitions.<n>Our paper illustrates the many merits of SLT for understanding neural network phase transitions, and poses open research questions for the field.
  arXiv  Detail & Related papers  (2025-11-30T01:39:27Z)
• Learning with Exact Invariances in Polynomial Time [43.81087760363805]
  We study the statistical-computational trade-offs for learning with exact invariances (or symmetries) using kernel regression.<n>Our approach achieves the same excess population risk as the original kernel regression problem.
  arXiv  Detail & Related papers  (2025-02-27T04:49:52Z)
• Riemannian stochastic optimization methods avoid strict saddle points [68.80251170757647]
  We show that policies under study avoid strict saddle points / submanifolds with probability 1. This result provides an important sanity check as it shows that, almost always, the limit state of an algorithm can only be a local minimizer.
  arXiv  Detail & Related papers  (2023-11-04T11:12:24Z)
• Machine learning detects terminal singularities [49.1574468325115]
  Q-Fano varieties are positively curved shapes which have Q-factorial terminal singularities. Despite their importance, the classification of Q-Fano varieties remains unknown. In this paper we demonstrate that machine learning can be used to understand this classification.
  arXiv  Detail & Related papers  (2023-10-31T13:51:24Z)
• Information Bounds on phase transitions in disordered systems [0.0]
  We study phase transitions in models with randomness, such as localization in disordered systems, or random quantum circuits with measurements. We benchmark our method and rederive the well-known Harris criterion, bounding critical exponents in the Anderson localization transition for noninteracting particles. While in real space our critical exponent bound agrees with recent consensus, we find that, somewhat surprisingly, numerical results on Fock-space localization for limited-sized systems do not obey our bounds.
  arXiv  Detail & Related papers  (2023-08-29T18:00:07Z)
• Evolving Scientific Discovery by Unifying Data and Background Knowledge with AI Hilbert [11.56572994087136]
  We show that some famous scientific laws, including Kepler's Third Law of Planetary Motion, can be derived in a principled manner from axioms and data. We model complexity using binary variables and logical constraints, solve optimization problems via mixed-integer linear or semidefinite optimization, and prove the validity of our scientific discoveries.
  arXiv  Detail & Related papers  (2023-08-18T11:19:41Z)
• The Principle of Uncertain Maximum Entropy [0.0]
  The Principle of Maximum Entropy is a rigorous technique for estimating an unknown distribution given partial information.<n>We relax this requirement using a memoryless communication channel as a framework to derive a new, more general principle.<n>We show our new principle provides an upper bound on the entropy of the unknown distribution and the amount of information lost due to the use of a given communications channel is unknown.
  arXiv  Detail & Related papers  (2023-05-17T00:45:41Z)
• Computing non-equilibrium trajectories by a deep learning approach [0.0]
  Predicting the occurence of rare and extreme events in complex systems is a well-known problem in non-equilibrium physics. We propose here a new method which minimizes the geometrical action instead using neural networks. It relies on a natural and simple machine-learning formulation of the classical gMAM approach.
  arXiv  Detail & Related papers  (2022-10-08T14:53:23Z)
• A Random Matrix Perspective on Random Tensors [40.89521598604993]
  We study the spectra of random matrices arising from contractions of a given random tensor. Our technique yields a hitherto unknown characterization of the local maximum of the ML problem. Our approach is versatile and can be extended to other models, such as asymmetric, non-Gaussian and higher-order ones.
  arXiv  Detail & Related papers  (2021-08-02T10:42:22Z)
• Learning a Single Neuron with Bias Using Gradient Descent [53.15475693468925]
  We study the fundamental problem of learning a single neuron with a bias term. We show that this is a significantly different and more challenging problem than the bias-less case.
  arXiv  Detail & Related papers  (2021-06-02T12:09:55Z)
• Conditional physics informed neural networks [85.48030573849712]
  We introduce conditional PINNs (physics informed neural networks) for estimating the solution of classes of eigenvalue problems. We show that a single deep neural network can learn the solution of partial differential equations for an entire class of problems.
  arXiv  Detail & Related papers  (2021-04-06T18:29:14Z)
• Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization [36.182992409810446]
  This paper investigates the importance of noise in non optimization problems. We show that gradient descent can converge to any global form that converges to a global bias that is determined by the injected noise.
  arXiv  Detail & Related papers  (2021-02-24T17:50:17Z)
• Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition [34.235007566913396]
  We describe an interpretable, symmetric decomposition of the variance into terms associated with the labels. We find that the bias decreases monotonically with the network width, but the variance terms exhibit non-monotonic behavior. We also analyze the strikingly rich phenomenology that arises.
  arXiv  Detail & Related papers  (2020-11-04T21:04:02Z)
• Learning Set Functions that are Sparse in Non-Orthogonal Fourier Bases [73.53227696624306]
  We present a new family of algorithms for learning Fourier-sparse set functions. In contrast to other work that focused on the Walsh-Hadamard transform, our novel algorithms operate with recently introduced non-orthogonal Fourier transforms. We demonstrate effectiveness on several real-world applications.
  arXiv  Detail & Related papers  (2020-10-01T14:31:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.