Marginals Before Conditionals

• URL: http://arxiv.org/abs/2603.10074v1
• Date: Tue, 10 Mar 2026 05:51:01 GMT
• Title: Marginals Before Conditionals
• Abstract summary: We construct a minimal task that isolates conditional learning in neural networks.<n>The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional.
• Score: 0.15229257192293197
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional in a sharp, collective transition. The plateau has a clean decomposition: height = log K (set by ambiguity), duration = f(D) (set by dataset size D, not K). Gradient noise stabilizes the marginal solution: higher learning rates monotonically slow the transition (3.6* across a 7* η range at fixed throughput), and batch-size reduction delays escape, consistent with an entropic force opposing departure from the low-gradient marginal. Internally, a selector-routing head assembles during the plateau, leading the loss transition by ~50% of the waiting time. This is the Type 2 directional asymmetry of Papadopoulos et al. [2024], measured dynamically: we track the excess risk from log K to zero and characterize what stabilizes it, what triggers its collapse, and how long it takes.

Related papers

• Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking [0.0]
  We map the $0.8-to-generalization boundary across 384 configurations of two-hidden generalizations on modular arithmetic.<n>Data complexity is the dominant driver of regime transition, not model capacity.
  arXiv  Detail & Related papers  (2026-09-09T16:47:03Z)
• Constrained Online Learning with Noisy Constraint Values [55.29259818039367]
  Under common feasibility, our LEDGER algorithm attains $O(sqrt T)$ expected regret and $O(sqrtTlog(eT))$ expected budget violation.<n>Neither a Slater condition, independence between feedback channels, nor an absolute constraint-value bound is needed.
  arXiv  Detail & Related papers  (2026-09-07T01:38:41Z)
• Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps [4.422349568747053]
  We study the phase of a warmup-stable-decay (WSD) learning-rate schedule, now a default in large-model pretraining.<n>We give a provable account of which case obtains robustness, and it turns on two properties together: the structure of the gradient noise and whether the structure of the gradient noise normalizes its update.
  arXiv  Detail & Related papers  (2026-07-14T05:21:38Z)
• Sample Complexity of Policy Gradient for Log-Growth Control [23.86784769848408]
  We study the sample complexity of policy gradient for log-growth control.<n>We prove that projected mini-batch policy gradient attains total sample complexity.
  arXiv  Detail & Related papers  (2026-05-26T07:24:14Z)
• SGD at the Edge of Stability: The Stochastic Sharpness Gap [10.176501817419371]
  When training networks with full-batch gradient descent (GD) and step size $$, the largest eigenvalue of the Hessian rises to $2/$ and hovers there, a phenomenon termed the Edge of Stability (EoS)<n>citetdamian2023selfstab showed that this behavior is explained by a self-stabilization mechanism driven by third-order structure of the loss, and that GD implicitly follows projected gradient descent (PGD) on the constraint $ S(boldsymbol)leq 2/$.<n>For mini-batch gradient descent
  arXiv  Detail & Related papers  (2026-04-22T19:02:52Z)
• Conservation Law Breaking at the Edge of Stability: A Spectral Theory of Non-Convex Neural Network Optimization [0.0]
  We show that gradient descent reliably finds good solutions in non-dimensional network optimization.<n>We identify two regimes separated by a width-dependent dynamical transition.<n>All predictions are validated across 23 experiments.
  arXiv  Detail & Related papers  (2026-04-08T10:41:24Z)
• Fast Non-Episodic Finite-Horizon RL with K-Step Lookahead Thresholding [9.43984448422843]
  We introduce a K-step lookahead Q-function that truncates planning to the next K steps.<n>We numerically evaluate the performance of our algorithm under the objective of maximizing reward.<n> Empirical results demonstrate superior cumulative rewards over state-of-the-art RL methods across synthetic MDPs and RL environments.
  arXiv  Detail & Related papers  (2026-01-31T15:44:12Z)
• CAO: Curvature-Adaptive Optimization via Periodic Low-Rank Hessian Sketching [0.0]
  First-order curvatureobs are reliable but slow in sharp, anisotropic regions.<n>We use a curvature-declared method that sketches a Hessian subspace via Hessian products.<n>On CIFAR/100 with ResNet-18/34, the method enters the low-loss threshold (0.75x) substantially earlier.
  arXiv  Detail & Related papers  (2025-11-16T10:57:33Z)
• Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective [66.80315289020487]
  Warmup-Stable-Decay (WSD) schedule uses a constant learning rate to produce a main branch of iterates that can continue indefinitely without a pre-specified compute budget.<n>We show that pretraining loss exhibits a river valley landscape, which resembles a deep valley with a river at its bottom.<n>Inspired by the theory, we introduce WSD-S, a variant of WSD that reuses previous checkpoints' decay phases and keeps only one main branch.
  arXiv  Detail & Related papers  (2024-10-07T16:49:39Z)
• KPZ scaling from the Krylov space [83.88591755871734]
  Recently, a superdiffusion exhibiting the Kardar-Parisi-Zhang scaling in late-time correlators and autocorrelators has been reported. Inspired by these results, we explore the KPZ scaling in correlation functions using their realization in the Krylov operator basis.
  arXiv  Detail & Related papers  (2024-06-04T20:57:59Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)
• Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency [47.8739414267201]
  We consider gradient descent (GD) with a constant stepsize applied to logistic regression with linearly separable data. We show that GD exits this initial oscillatory phase rapidly -- in $mathcalO(eta)$ steps -- and subsequently achieves an $tildemathcalO (1 / (eta t) )$ convergence rate. Our results imply that, given a budget of $T$ steps, GD can achieve an accelerated loss of $tildemathcalO (1/T2)$ with an aggressive stepsize
  arXiv  Detail & Related papers  (2024-02-24T23:10:28Z)
• On the Convergence of Gradient Descent for Large Learning Rates [55.33626480243135]
  We show that convergence is impossible when a fixed step size is used.<n>We provide a proof of this in the case of linear neural networks with a squared loss.<n>We also prove the impossibility of convergence for more general losses without requiring strong assumptions such as Lipschitz continuity for the gradient.
  arXiv  Detail & Related papers  (2024-02-20T16:01:42Z)
• Breaking the Heavy-Tailed Noise Barrier in Stochastic Optimization Problems [56.86067111855056]
  We consider clipped optimization problems with heavy-tailed noise with structured density. We show that it is possible to get faster rates of convergence than $mathcalO(K-(alpha - 1)/alpha)$, when the gradients have finite moments of order. We prove that the resulting estimates have negligible bias and controllable variance.
  arXiv  Detail & Related papers  (2023-11-07T17:39:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.