Benign interpolation and Occam's razor

• URL: http://arxiv.org/abs/2608.03386v1
• Date: Tue, 04 Aug 2026 09:34:47 GMT
• Title: Benign interpolation and Occam's razor
• Abstract summary: We show that modern deep learning methods generalize well even when they fit their training data perfectly.<n>We argue that this new appeal to simplicity creates an explanatory gap.<n>Lacking a provable connection to generalization, it is the name "simplicity" that does the work used to do, making a substantive and unargued assumption look like the application of a familiar methodological principle.
• Score: 0.688204255655161
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contemporary deep learning methods generalize well even when they fit their training data perfectly, a phenomenon known as benign interpolation. This phenomenon cannot be accounted for by classical statistical learning theory and has prompted a range of attempted new explanations in the statistics and machine learning literature. A common feature of these new proposals is an appeal to a simplicity preference among interpolating models, often presented as a form of Occam's razor. We clarify this debate for a philosophical audience and argue that this new appeal to simplicity creates an explanatory gap. The classical theory offers theorems which connect the simplicity of model classes to good generalization, thus underwriting methodological simplicity norms. The new accounts instead appeal to properties of individual models, which they interpret as a kind of simplicity. Lacking a provable connection to generalization, it is the name "simplicity" that does the work a theorem used to do, making a substantive and unargued assumption look like the application of a familiar methodological principle.

Related papers

• Statistical learning theory and Occam's razor: Regularization [0.913755431537592]
  Occam's razor, which instructs us to prefer simplicity in inductive inference, has attracted much scrutiny both in the philosophy of science and in machine learning.<n>I argue that in order to profit from theoretical reliability and "what-you-see-is-what-you-get" guarantees, one must implement a certain preference for simplicity over fit.<n>This is a genuine methodological justification, which neither collapses to a purely pragmatic principle that we prefer simplicity for its own sake, nor to an ontological assumption that the truth is simple.
  arXiv  Detail & Related papers  (2026-08-04T09:17:42Z)
• Information-Theoretic Framework for Understanding Modern Machine-Learning [4.435094091999926]
  We present an information-theoretic framework that views learning as universal prediction under log loss.<n>We argue that successful architectures possess a broad complexity range, enabling learning in highly over- parameterized model classes.<n>The framework sheds light on the role of inductive biases, the effectiveness of descent gradient, and phenomena such as flat minima.
  arXiv  Detail & Related papers  (2025-06-09T11:32:31Z)
• Deep Learning is Not So Mysterious or Different [54.5330466151362]
  We argue that anomalous generalization behaviour is not distinct to neural networks.<n>We present soft inductive biases as a key unifying principle in explaining these phenomena.<n>We also highlight how deep learning is relatively distinct in other ways, such as its ability for representation learning.
  arXiv  Detail & Related papers  (2025-03-03T22:56:04Z)
• Sheaf theory: from deep geometry to deep learning [0.3749861135832073]
  This paper provides an overview of the applications of sheaf theory in deep learning, data science, and computer science.<n>We describe intuitions and motivations underlying sheaf theory shared by both theoretical researchers and practitioners.<n>We present a new algorithm to compute sheaf cohomology on arbitrary finite posets in response.
  arXiv  Detail & Related papers  (2025-02-21T14:00:25Z)
• Toward Understanding In-context vs. In-weight Learning [50.24035812301655]
  We identify simplified distributional properties that give rise to the emergence and disappearance of in-context learning.<n>We then extend the study to a full large language model, showing how fine-tuning on various collections of natural language prompts can elicit similar in-context and in-weight learning behaviour.
  arXiv  Detail & Related papers  (2024-10-30T14:09:00Z)
• Sequential Representation Learning via Static-Dynamic Conditional Disentanglement [58.19137637859017]
  This paper explores self-supervised disentangled representation learning within sequential data, focusing on separating time-independent and time-varying factors in videos. We propose a new model that breaks the usual independence assumption between those factors by explicitly accounting for the causal relationship between the static/dynamic variables. Experiments show that the proposed approach outperforms previous complex state-of-the-art techniques in scenarios where the dynamics of a scene are influenced by its content.
  arXiv  Detail & Related papers  (2024-08-10T17:04:39Z)
• Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layers [49.80959223722325]
  We study the distinction between feed-forward and attention layers in large language models.<n>We find that feed-forward layers tend to learn simple distributional associations such as bigrams, while attention layers focus on in-context reasoning.
  arXiv  Detail & Related papers  (2024-06-05T08:51:08Z)
• Statistical learning theory and Occam's razor: The core argument [1.6317061277457001]
  This paper distills the core argument for simplicity obtainable from statistical learning theory.<n>It is built on the theory's central learning guarantee for the method of empirical risk minimization.
  arXiv  Detail & Related papers  (2023-12-21T13:40:31Z)
• Learning with Explanation Constraints [91.23736536228485]
  We provide a learning theoretic framework to analyze how explanations can improve the learning of our models. We demonstrate the benefits of our approach over a large array of synthetic and real-world experiments.
  arXiv  Detail & Related papers  (2023-03-25T15:06:47Z)
• The unstable formula theorem revisited via algorithms [18.557423328068122]
  In response to gaps in existing learning models, we introduce a new statistical learning model, called Probably Eventually Correct'' or PEC.<n>We characterize Littlestone (stable) classes in terms of this model.<n>In order to obtain a characterization of Littlestone classes in terms of frequent definitions, we build an equivalence theorem highlighting what is common to many existing approximation algorithms.
  arXiv  Detail & Related papers  (2022-12-09T18:53:34Z)
• General Rough Modeling of Cluster Analysis [0.0]
  The essence of the approach is explained in brief and supported by an example. A new general rough method of analyzing clusterings is invented, and this opens the subject to clearer conceptions and contamination-free proofs.
  arXiv  Detail & Related papers  (2021-06-02T20:54:10Z)
• Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge [96.92252296244233]
  Large pre-trained language models (LMs) acquire some reasoning capacity, but this ability is difficult to control. We show that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements. Our work paves a path towards open-domain systems that constantly improve by interacting with users who can instantly correct a model by adding simple natural language statements.
  arXiv  Detail & Related papers  (2020-06-11T17:02:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.