A Compositional Theory of Curvature in Probabilistic Circuits

• URL: http://arxiv.org/abs/2608.12869v1
• Date: Thu, 13 Aug 2026 06:34:01 GMT
• Title: A Compositional Theory of Curvature in Probabilistic Circuits
• Abstract summary: Probabilistic Circuits (PCs) are generative models that support exact inference.<n>Recent work regularizes this trace globally to bias learning toward flatter, better generalizing optima.<n>We show that treating sharpness as a global regularizer can be misspecified for PCs, whose curvature is inherently compositional.
• Score: 9.164238097900308
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Probabilistic Circuits (PCs) are generative models that support exact inference and, unlike deep neural networks, admit an exact and tractable measure of loss-surface curvature: the trace of the Hessian of the log-likelihood. Recent work regularizes this trace globally to bias learning toward flatter, better generalizing optima. We show that treating sharpness as a global regularizer can be misspecified for PCs, whose curvature is inherently compositional. We prove that each sum node's contribution to the Hessian trace factorizes exactly into its circuit flow, which measures how heavily the node is used, and a local sharpness term determined by its output distribution. This decomposition provides insights into why global sharpness regularization is depth biased and can lead to underfitting. Building on it, we introduce an adaptive sharpness aware regularizer that penalizes nodes based on intrinsic local curvature and preserves closed form EM updates. We also show that empirically, this targeted regularization recovers the generalization that global regularization sacrifices while retaining the robustness and benefits of sharpness aware learning.

Related papers

• Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization [7.792020418343022]
  Divergence-based regularization and Sharpness-Aware Minimization (SAM) are two prominent approaches for improving generalization in deep learning.<n>We show that the two methods are locally consistent under parameter-space perturbations.<n>We also show that divergence-based regularization is defined through transformations of the input.
  arXiv  Detail & Related papers  (2026-09-08T19:01:05Z)
• Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL [7.803284875836737]
  bootstrapping can amplify estimation errors in offline reinforcement learning.<n>We propose Convex Hull Neighborhood Smooth Dual Generalization, which expresses the Bellman backup as an in-sample value target plus a CHN-local correction.<n> Experiments on Gym-MuJoCo and AntMaze show strong aggregate performance and stable value estimation.
  arXiv  Detail & Related papers  (2026-08-04T04:30:02Z)
• Tailoring Strictly Proper Scoring Rules for Downstream Tasks: An Application to Causal Inference [48.78219918881965]
  We propose a principled framework for deriving task-specific strictly proper scoring rules by matching the local curvature of the downstream error metric.<n>We apply this to the Average Treatment Effect (ATE) estimation, deriving a closed-form loss and its corresponding canonical probability mapping.
  arXiv  Detail & Related papers  (2026-06-02T08:41:25Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Why Self-Supervised Encoders Want to Be Normal [0.0]
  We show that the preference for normal distributions in self-supervised encoders is a direct consequence of the Information Bottleneck (IB) principle.<n>Under this framework, we show that latent representations correspond to soft clustering of inputs sharing similar predictive distributions.<n>This perspective unifies a wide range of existing supervised and less-supervised objectives and provides a principled explanation for commonly used regularization schemes.
  arXiv  Detail & Related papers  (2026-04-30T11:33:05Z)
• Explicit Dropout: Deterministic Regularization for Transformer Architectures [55.09895958546215]
  Dropout is a widely used regularization technique in deep learning, but its effects are typically realized through masking.<n>We propose a deterministic formulation that expresses dropout as an additive regularizer directly incorporated into the training loss.
  arXiv  Detail & Related papers  (2026-04-22T12:45:51Z)
• The Inductive Bias of Convolutional Neural Networks: Locality and Weight Sharing Reshape Implicit Regularization [57.37943479039033]
  We study how architectural inductive bias reshapes the implicit regularization induced by the edge-of-stability phenomenon in gradient descent.<n>We show that locality and weight sharing fundamentally change this picture.
  arXiv  Detail & Related papers  (2026-03-05T04:50:51Z)
• Tractable Sharpness-Aware Learning of Probabilistic Circuits [9.353446248109599]
  Probabilistic Circuits (PCs) are a class of generative models that allow exact and tractable inference for a wide range of queries.<n>Recent developments have enabled the learning of deep and expressive PCs, but this increased capacity can often lead to overfitting.<n>Inspired by sharpness aware minimization in neural networks, we propose a Hessian-based regularizer for training PCs.
  arXiv  Detail & Related papers  (2025-08-07T16:13:24Z)
• Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction [36.83448475700536]
  Normalization layers (e.g., Batch Normalization, Layer Normalization) were introduced to help with optimization difficulties in very deep nets. This paper gives mathematical analysis and experiments suggesting that normalization encourages GD to reduce the sharpness of loss surface.
  arXiv  Detail & Related papers  (2022-06-14T18:19:05Z)
• Implicit Regularization in ReLU Networks with the Square Loss [56.70360094597169]
  We show that it is impossible to characterize the implicit regularization with the square loss by any explicit function of the model parameters. Our results suggest that a more general framework may be needed to understand implicit regularization for nonlinear predictors.
  arXiv  Detail & Related papers  (2020-12-09T16:48:03Z)
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [56.69671152009899]
  We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization. We also propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction.
  arXiv  Detail & Related papers  (2020-02-20T15:13:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.