Statistical learning theory and Occam's razor: The core argument

• URL: http://arxiv.org/abs/2312.13842v2
• Date: Fri, 29 Nov 2024 16:02:21 GMT
• Title: Statistical learning theory and Occam's razor: The core argument
• Abstract summary: This paper distills the core argument for simplicity obtainable from statistical learning theory.<n>It is built on the theory's central learning guarantee for the method of empirical risk minimization.
• Score: 1.6317061277457001
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Statistical learning theory is often associated with the principle of Occam's razor, which recommends a simplicity preference in inductive inference. This paper distills the core argument for simplicity obtainable from statistical learning theory, built on the theory's central learning guarantee for the method of empirical risk minimization. This core "means-ends" argument is that a simpler hypothesis class or inductive model is better because it has better learning guarantees; however, these guarantees are model-relative and so the theoretical push towards simplicity is checked by our prior knowledge.

Related papers

• Benign interpolation and Occam's razor [0.688204255655161]
  We show that modern deep learning methods generalize well even when they fit their training data perfectly.<n>We argue that this new appeal to simplicity creates an explanatory gap.<n>Lacking a provable connection to generalization, it is the name "simplicity" that does the work used to do, making a substantive and unargued assumption look like the application of a familiar methodological principle.
  arXiv  Detail & Related papers  (2026-08-04T09:34:47Z)
• Statistical learning theory and Occam's razor: Regularization [0.913755431537592]
  Occam's razor, which instructs us to prefer simplicity in inductive inference, has attracted much scrutiny both in the philosophy of science and in machine learning.<n>I argue that in order to profit from theoretical reliability and "what-you-see-is-what-you-get" guarantees, one must implement a certain preference for simplicity over fit.<n>This is a genuine methodological justification, which neither collapses to a purely pragmatic principle that we prefer simplicity for its own sake, nor to an ontological assumption that the truth is simple.
  arXiv  Detail & Related papers  (2026-08-04T09:17:42Z)
• Learning to Reason with Insight for Informal Theorem Proving [50.054221663394195]
  In this work, we identify a primary bottleneck in informal theorem proving as a lack of insight.<n>We propose a novel framework designed to cultivate this essential reasoning skill and enable LLMs to perform insightful reasoning.<n>Our experiments on challenging mathematical benchmarks demonstrate that this insight-aware generation strategy significantly outperforms baselines.
  arXiv  Detail & Related papers  (2026-04-17T17:36:21Z)
• A Survey of Inductive Reasoning for Large Language Models [55.23215679173251]
  The inductive mode is crucial for knowledge generalization and aligns better with human cognition.<n>Despite the importance of inductive reasoning, there is no systematic summary of it.<n>This paper presents the first comprehensive survey of inductive reasoning for large language models.
  arXiv  Detail & Related papers  (2025-10-11T11:45:38Z)
• Does More Inference-Time Compute Really Help Robustness? [50.47666612618054]
  We show that small-scale, open-source models can benefit from inference-time scaling.<n>We identify an important security risk, intuitively motivated and empirically verified as an inverse scaling law.<n>We urge practitioners to carefully weigh these subtle trade-offs before applying inference-time scaling in security-sensitive, real-world applications.
  arXiv  Detail & Related papers  (2025-07-21T18:08:38Z)
• DeepTheorem: Advancing LLM Reasoning for Theorem Proving Through Natural Language and Reinforcement Learning [67.93945726549289]
  DeepTheorem is a comprehensive informal theorem-proving framework exploiting natural language to enhance mathematical reasoning.<n>DeepTheorem includes a large-scale benchmark dataset consisting of 121K high-quality IMO-level informal theorems and proofs.<n>We devise a novel reinforcement learning strategy (RL-Zero) explicitly tailored to informal theorem proving, leveraging the verified theorem variants to incentivize robust mathematical inference.
  arXiv  Detail & Related papers  (2025-05-29T17:59:39Z)
• Controllable Logical Hypothesis Generation for Abductive Reasoning in Knowledge Graphs [54.596180382762036]
  Abductive reasoning in knowledge graphs aims to generate plausible logical hypotheses from observed entities.<n>Due to a lack of controllability, a single observation may yield numerous plausible but redundant or irrelevant hypotheses.<n>We introduce the task of controllable hypothesis generation to improve the practical utility of abductive reasoning.
  arXiv  Detail & Related papers  (2025-05-27T09:36:47Z)
• Monotonic Learning in the PAC Framework: A New Perspective [8.911102248548206]
  We construct a theoretical risk distribution that approximates a learning algorithm's actual performance.<n>We rigorously prove that this theoretical distribution exhibits monotonicity as sample sizes increase.
  arXiv  Detail & Related papers  (2025-01-09T12:26:11Z)
• The Foundations of Tokenization: Statistical and Computational Concerns [51.370165245628975]
  Tokenization is a critical step in the NLP pipeline. Despite its recognized importance as a standard representation method in NLP, the theoretical underpinnings of tokenization are not yet fully understood. The present paper contributes to addressing this theoretical gap by proposing a unified formal framework for representing and analyzing tokenizer models.
  arXiv  Detail & Related papers  (2024-07-16T11:12:28Z)
• A Theory of Machine Learning [0.0]
  We show that this theory challenges common assumptions in the statistical and the computational learning theories. We briefly discuss some case studies from natural language processing and macroeconomics from the perspective of the new theory.
  arXiv  Detail & Related papers  (2024-07-07T23:57:10Z)
• Machine Learning of the Prime Distribution [49.84018914962972]
  We provide a theoretical argument explaining the experimental observations of Yang-Hui He about the learnability of primes. We also posit that the ErdHos-Kac law would very unlikely be discovered by current machine learning techniques.
  arXiv  Detail & Related papers  (2024-03-19T09:47:54Z)
• Generalising realisability in statistical learning theory under epistemic uncertainty [5.7205053622241735]
  This paper looks into how central notions in statistical learning theory, such as realisability, generalise under the assumption that train and test distribution are issued from the same credal set, i.e., a convex set of probability distributions.
  arXiv  Detail & Related papers  (2024-02-22T18:20:25Z)
• Inference of Abstraction for a Unified Account of Reasoning and Learning [0.0]
  We give a simple theory of probabilistic inference for a unified account of reasoning and learning. We simply model how data cause symbolic knowledge in terms of its satisfiability in formal logic.
  arXiv  Detail & Related papers  (2024-02-14T09:43:35Z)
• Advancing Abductive Reasoning in Knowledge Graphs through Complex Logical Hypothesis Generation [43.26412690886471]
  This paper introduces the task of complex logical hypothesis generation, as an initial step towards abductive logical reasoning with Knowledge Graph. We find that the supervised trained generative model can generate logical hypotheses that are structurally closer to the reference hypothesis. We introduce the Reinforcement Learning from Knowledge Graph (RLF-KG) method, which minimizes differences between observations and conclusions drawn from generated hypotheses according to the KG.
  arXiv  Detail & Related papers  (2023-12-25T08:06:20Z)
• Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement [92.61557711360652]
  Language models (LMs) often fall short on inductive reasoning, despite achieving impressive success on research benchmarks. We conduct a systematic study of the inductive reasoning capabilities of LMs through iterative hypothesis refinement. We reveal several discrepancies between the inductive reasoning processes of LMs and humans, shedding light on both the potentials and limitations of using LMs in inductive reasoning tasks.
  arXiv  Detail & Related papers  (2023-10-12T17:51:10Z)
• A Theory for Emergence of Complex Skills in Language Models [56.947273387302616]
  A major driver of AI products today is the fact that new skills emerge in language models when their parameter set and training corpora are scaled up. This paper takes a different approach, analysing emergence using the famous (and empirical) Scaling Laws of LLMs and a simple statistical framework.
  arXiv  Detail & Related papers  (2023-07-29T09:22:54Z)
• Towards Trustworthy Explanation: On Causal Rationalization [9.48539398357156]
  We propose a new model of rationalization based on two causal desiderata, non-spuriousness and efficiency. The superior performance of the proposed causal rationalization is demonstrated on real-world review and medical datasets.
  arXiv  Detail & Related papers  (2023-06-25T03:34:06Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• A Simple Generative Model of Logical Reasoning and Statistical Learning [0.6853165736531939]
  Statistical learning and logical reasoning are two major fields of AI expected to be unified for human-like machine intelligence. We here propose a simple Bayesian model of logical reasoning and statistical learning. We simply model how data causes symbolic knowledge in terms of its satisfiability in formal logic.
  arXiv  Detail & Related papers  (2023-05-18T16:34:51Z)
• The role of prior information and computational power in Machine Learning [0.0]
  We discuss how prior information and computational power can be employed to solve a learning problem. We argue that employing high computational power has the advantage of a higher performance.
  arXiv  Detail & Related papers  (2022-10-31T20:39:53Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)
• Principled Knowledge Extrapolation with GANs [92.62635018136476]
  We study counterfactual synthesis from a new perspective of knowledge extrapolation. We show that an adversarial game with a closed-form discriminator can be used to address the knowledge extrapolation problem. Our method enjoys both elegant theoretical guarantees and superior performance in many scenarios.
  arXiv  Detail & Related papers  (2022-05-21T08:39:42Z)
• Learning Topic Models: Identifiability and Finite-Sample Analysis [6.181048261489101]
  We propose a maximum likelihood estimator (MLE) of latent topics based on a specific integrated likelihood. We conclude with empirical studies on both simulated and real datasets.
  arXiv  Detail & Related papers  (2021-10-08T16:35:42Z)
• A Low Rank Promoting Prior for Unsupervised Contrastive Learning [108.91406719395417]
  We construct a novel probabilistic graphical model that effectively incorporates the low rank promoting prior into the framework of contrastive learning. Our hypothesis explicitly requires that all the samples belonging to the same instance class lie on the same subspace with small dimension. Empirical evidences show that the proposed algorithm clearly surpasses the state-of-the-art approaches on multiple benchmarks.
  arXiv  Detail & Related papers  (2021-08-05T15:58:25Z)
• A Topological Perspective on Causal Inference [10.965065178451104]
  We show that substantive assumption-free causal inference is possible only in a meager set of structural causal models. Our results show that inductive assumptions sufficient to license valid causal inferences are statistically unverifiable in principle. An additional benefit of our topological approach is that it easily accommodates SCMs with infinitely many variables.
  arXiv  Detail & Related papers  (2021-07-18T23:09:03Z)
• Constrained Learning with Non-Convex Losses [119.8736858597118]
  Though learning has become a core technology of modern information processing, there is now ample evidence that it can lead to biased, unsafe, and prejudiced solutions.
  arXiv  Detail & Related papers  (2021-03-08T23:10:33Z)
• Marginal likelihood computation for model selection and hypothesis testing: an extensive review [66.37504201165159]
  This article provides a comprehensive study of the state-of-the-art of the topic. We highlight limitations, benefits, connections and differences among the different techniques. Problems and possible solutions with the use of improper priors are also described.
  arXiv  Detail & Related papers  (2020-05-17T18:31:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.