Statistical learning theory and Occam's razor: Regularization

• URL: http://arxiv.org/abs/2608.04049v1
• Date: Tue, 04 Aug 2026 09:17:42 GMT
• Title: Statistical learning theory and Occam's razor: Regularization
• Abstract summary: Occam's razor, which instructs us to prefer simplicity in inductive inference, has attracted much scrutiny both in the philosophy of science and in machine learning.<n>I argue that in order to profit from theoretical reliability and "what-you-see-is-what-you-get" guarantees, one must implement a certain preference for simplicity over fit.<n>This is a genuine methodological justification, which neither collapses to a purely pragmatic principle that we prefer simplicity for its own sake, nor to an ontological assumption that the truth is simple.
• Score: 0.913755431537592
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The principle of Occam's razor, which instructs us to prefer simplicity in inductive inference, has attracted much scrutiny both in the philosophy of science and in machine learning. In either field, however, a justification for the principle has been elusive. In this paper, building on an earlier "core argument," I spell out a justification from statistical learning theory for the procedure of regularization: for trading off fit for simplicity. The means-ends argument is that in order to profit from theoretical reliability and "what-you-see-is-what-you-get" guarantees, one must implement a certain preference for simplicity over fit. This is a genuine methodological justification, which neither collapses to a purely pragmatic principle that we prefer simplicity for its own sake, nor to an ontological assumption that the truth is simple.

Related papers

• Benign interpolation and Occam's razor [0.688204255655161]
  We show that modern deep learning methods generalize well even when they fit their training data perfectly.<n>We argue that this new appeal to simplicity creates an explanatory gap.<n>Lacking a provable connection to generalization, it is the name "simplicity" that does the work used to do, making a substantive and unargued assumption look like the application of a familiar methodological principle.
  arXiv  Detail & Related papers  (2026-08-04T09:34:47Z)
• Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs [45.274171760492685]
  We find that enabling reasoning substantially expands the capability boundary of the model's parametric knowledge recall.<n>We show that our insights can be harnessed to directly improve model accuracy by prioritizing reasoning trajectories that contain hallucination-free factual statements.
  arXiv  Detail & Related papers  (2026-03-10T16:59:20Z)
• Newton to Einstein: Axiom-Based Discovery via Game Design [55.30047000068118]
  We propose a game design framework in which scientific inquiry is recast as a rule-evolving system.<n>Unlike conventional ML approaches that operate within fixed assumptions, our method enables the discovery of new theoretical structures.
  arXiv  Detail & Related papers  (2025-09-05T18:59:18Z)
• Reasoning is about giving reasons [55.56111618153049]
  We show that we can identify and extract the logical structure of natural language arguments in three popular reasoning datasets with high accuracies.<n>Our approach supports all forms of reasoning that depend on the logical structure of the natural language argument.
  arXiv  Detail & Related papers  (2025-08-20T07:26:53Z)
• Think When You Need: Self-Adaptive Chain-of-Thought Learning [20.22448368125018]
  Chain of Thought (CoT) reasoning enhances language models' performance but often leads to inefficient "overthinking" on simple problems.<n>We identify that existing approaches directly penalizing reasoning length fail to account for varying problem complexity.<n>Our approach constructs rewards through length and quality comparisons, guided by theoretical assumptions that jointly enhance solution correctness with conciseness.
  arXiv  Detail & Related papers  (2025-04-04T07:34:01Z)
• Advancing Abductive Reasoning in Knowledge Graphs through Complex Logical Hypothesis Generation [43.26412690886471]
  This paper introduces the task of complex logical hypothesis generation, as an initial step towards abductive logical reasoning with Knowledge Graph. We find that the supervised trained generative model can generate logical hypotheses that are structurally closer to the reference hypothesis. We introduce the Reinforcement Learning from Knowledge Graph (RLF-KG) method, which minimizes differences between observations and conclusions drawn from generated hypotheses according to the KG.
  arXiv  Detail & Related papers  (2023-12-25T08:06:20Z)
• Statistical learning theory and Occam's razor: The core argument [1.6317061277457001]
  This paper distills the core argument for simplicity obtainable from statistical learning theory.<n>It is built on the theory's central learning guarantee for the method of empirical risk minimization.
  arXiv  Detail & Related papers  (2023-12-21T13:40:31Z)
• A Simple Generative Model of Logical Reasoning and Statistical Learning [0.6853165736531939]
  Statistical learning and logical reasoning are two major fields of AI expected to be unified for human-like machine intelligence. We here propose a simple Bayesian model of logical reasoning and statistical learning. We simply model how data causes symbolic knowledge in terms of its satisfiability in formal logic.
  arXiv  Detail & Related papers  (2023-05-18T16:34:51Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)
• Quantum realism: axiomatization and quantification [77.34726150561087]
  We build an axiomatization for quantum realism -- a notion of realism compatible with quantum theory. We explicitly construct some classes of entropic quantifiers that are shown to satisfy almost all of the proposed axioms.
  arXiv  Detail & Related papers  (2021-10-10T18:08:42Z)
• Logical Neural Networks [51.46602187496816]
  We propose a novel framework seamlessly providing key properties of both neural nets (learning) and symbolic logic (knowledge and reasoning) Every neuron has a meaning as a component of a formula in a weighted real-valued logic, yielding a highly intepretable disentangled representation. Inference is omni rather than focused on predefined target variables, and corresponds to logical reasoning.
  arXiv  Detail & Related papers  (2020-06-23T16:55:45Z)
• A (Simplified) Supreme Being Necessarily Exists, says the Computer: Computationally Explored Variants of G\"odel's Ontological Argument [0.0]
  Some argument premises are modified, others are dropped, modal collapse is avoided and validity is shown already in weak modal logics K and T. The paper illustrates how modern knowledge representation and reasoning technology for quantified non-classical logics can contribute new knowledge to other disciplines.
  arXiv  Detail & Related papers  (2020-01-14T10:26:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.