Statistical learning for constrained functional parameters in infinite-dimensional models

• URL: http://arxiv.org/abs/2404.09847v2
• Date: Fri, 18 Jul 2025 22:19:58 GMT
• Title: Statistical learning for constrained functional parameters in infinite-dimensional models
• Authors: Razieh Nabi, Nima S. Hejazi, Mark J. van der Laan, David Benkeser,
• Abstract summary: We develop a framework for estimating function-valued parameters under equality or inequality constraints in infinite-dimensional statistical models.<n>We characterize the solution as the minimizer of a penalized population risk using a Lagrange-type formulation.<n>We demonstrate the flexibility and effectiveness of the proposed method through various examples, simulations, and real-data applications.
• Score: 4.974815773537217
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We develop a general framework for estimating function-valued parameters under equality or inequality constraints in infinite-dimensional statistical models. Such constrained learning problems are common across many areas of statistics and machine learning, where estimated parameters must satisfy structural requirements such as moment restrictions, policy benchmarks, calibration criteria, or fairness considerations. To address these problems, we characterize the solution as the minimizer of a penalized population risk using a Lagrange-type formulation, and analyze it through a statistical functional lens. Central to our approach is a constraint-specific path through the unconstrained parameter space that defines the constrained solutions. For a broad class of constraint-risk pairs, this path admits closed-form expressions and reveals how constraints shape optimal adjustments. When closed forms are unavailable, we derive recursive representations that support tractable estimation. Our results also suggest natural estimators of the constrained parameter, constructed by combining estimates of unconstrained components of the data-generating distribution. Thus, our procedure can be integrated with any statistical learning approach and implemented using standard software. We provide general conditions under which the resulting estimators achieve optimal risk and constraint satisfaction, and we demonstrate the flexibility and effectiveness of the proposed method through various examples, simulations, and real-data applications.

Related papers

• Conformal and kNN Predictive Uncertainty Quantification Algorithms in Metric Spaces [3.637162892228131]
  We develop a conformal prediction algorithm that offers finite-sample coverage guarantees and fast convergence rates of the oracle estimator.<n>In heteroscedastic settings, we forgo these non-asymptotic guarantees to gain statistical efficiency.<n>We demonstrate the practical utility of our approach in personalized--medicine applications involving random response objects.
  arXiv  Detail & Related papers  (2025-07-21T15:54:13Z)
• A Graphical Global Optimization Framework for Parameter Estimation of Statistical Models with Nonconvex Regularization Functions [0.0]
  Problems with linear norm-bound constraints arise in a variety of applications, including portfolio optimization, machine learning, feature selection.<n>We propose a novel graphbased method to globally solve these problems.
  arXiv  Detail & Related papers  (2025-05-06T18:09:54Z)
• Automatic Debiased Machine Learning for Smooth Functionals of Nonparametric M-Estimands [34.30497962430375]
  We propose a unified framework for automatic debiased machine learning (autoDML) to perform inference on smooth functionals of infinite-dimensional M-estimands. We introduce three autoDML estimators based on one-step estimation, targeted minimum loss-based estimation, and the method of sieves. For data-driven model selection, we derive a novel decomposition of model approximation error for smooth functionals of M-estimands.
  arXiv  Detail & Related papers  (2025-01-21T03:50:51Z)
• Fair Risk Minimization under Causal Path-Specific Effect Constraints [3.0232957374216953]
  This paper introduces a framework for estimating fair optimal predictions using machine learning. We derive closed-form solutions for constrained optimization based on mean squared error and cross-entropy risk criteria.
  arXiv  Detail & Related papers  (2024-08-03T02:05:43Z)
• Generalization Bounds of Surrogate Policies for Combinatorial Optimization Problems [61.580419063416734]
  A recent stream of structured learning approaches has improved the practical state of the art for a range of optimization problems. The key idea is to exploit the statistical distribution over instances instead of dealing with instances separately. In this article, we investigate methods that smooth the risk by perturbing the policy, which eases optimization and improves the generalization error.
  arXiv  Detail & Related papers  (2024-07-24T12:00:30Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models [109.06052781040916]
  We introduce a technique to enhance the inference efficiency of parameter-shared language models. We also propose a simple pre-training technique that leads to fully or partially shared models. Results demonstrate the effectiveness of our methods on both autoregressive and autoencoding PLMs.
  arXiv  Detail & Related papers  (2023-10-19T15:13:58Z)
• Online Constraint Tightening in Stochastic Model Predictive Control: A Regression Approach [49.056933332667114]
  No analytical solutions exist for chance-constrained optimal control problems. We propose a data-driven approach for learning the constraint-tightening parameters online during control. Our approach yields constraint-tightening parameters that tightly satisfy the chance constraints.
  arXiv  Detail & Related papers  (2023-10-04T16:22:02Z)
• Efficient Model-Free Exploration in Low-Rank MDPs [76.87340323826945]
  Low-Rank Markov Decision Processes offer a simple, yet expressive framework for RL with function approximation. Existing algorithms are either (1) computationally intractable, or (2) reliant upon restrictive statistical assumptions. We propose the first provably sample-efficient algorithm for exploration in Low-Rank MDPs.
  arXiv  Detail & Related papers  (2023-07-08T15:41:48Z)
• On the Forward Invariance of Neural ODEs [92.07281135902922]
  We propose a new method to ensure neural ordinary differential equations (ODEs) satisfy output specifications. Our approach uses a class of control barrier functions to transform output specifications into constraints on the parameters and inputs of the learning system.
  arXiv  Detail & Related papers  (2022-10-10T15:18:28Z)
• On data-driven chance constraint learning for mixed-integer optimization problems [0.0]
  We develop a Chance Constraint Learning (CCL) methodology with a focus on mixed-integer linear optimization problems. CCL makes use of linearizable machine learning models to estimate conditional quantiles of the learned variables. An open-access software has been developed to be used by practitioners.
  arXiv  Detail & Related papers  (2022-07-08T11:54:39Z)
• A Free Lunch with Influence Functions? Improving Neural Network Estimates with Concepts from Semiparametric Statistics [41.99023989695363]
  We explore the potential for semiparametric theory to be used to improve neural networks and machine learning algorithms. We propose a new neural network method MultiNet, which seeks the flexibility and diversity of an ensemble using a single architecture.
  arXiv  Detail & Related papers  (2022-02-18T09:35:51Z)
• Neural Networks for Parameter Estimation in Intractable Models [0.0]
  We show how to estimate parameters from max-stable processes, where inference is exceptionally challenging. We use data from model simulations as input and train deep neural networks to learn statistical parameters.
  arXiv  Detail & Related papers  (2021-07-29T21:59:48Z)
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories [61.3299263929289]
  Simulation-based inference enables learning the parameters of a model even when its likelihood cannot be computed in practice. One class of methods uses data simulated with different parameters to infer an amortized estimator for the likelihood-to-evidence ratio. We show that this approach can be formulated in terms of mutual information between model parameters and simulated data.
  arXiv  Detail & Related papers  (2021-06-03T12:59:16Z)
• Estimating Structural Target Functions using Machine Learning and Influence Functions [103.47897241856603]
  We propose a new framework for statistical machine learning of target functions arising as identifiable functionals from statistical models. This framework is problem- and model-agnostic and can be used to estimate a broad variety of target parameters of interest in applied statistics. We put particular focus on so-called coarsening at random/doubly robust problems with partially unobserved information.
  arXiv  Detail & Related papers  (2020-08-14T16:48:29Z)
• A Semiparametric Approach to Interpretable Machine Learning [9.87381939016363]
  Black box models in machine learning have demonstrated excellent predictive performance in complex problems and high-dimensional settings. Their lack of transparency and interpretability restrict the applicability of such models in critical decision-making processes. We propose a novel approach to trading off interpretability and performance in prediction models using ideas from semiparametric statistics.
  arXiv  Detail & Related papers  (2020-06-08T16:38:15Z)
• Instability, Computational Efficiency and Statistical Accuracy [101.32305022521024]
  We develop a framework that yields statistical accuracy based on interplay between the deterministic convergence rate of the algorithm at the population level, and its degree of (instability) when applied to an empirical object based on $n$ samples. We provide applications of our general results to several concrete classes of models, including Gaussian mixture estimation, non-linear regression models, and informative non-response models.
  arXiv  Detail & Related papers  (2020-05-22T22:30:52Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)
• The empirical duality gap of constrained statistical learning [115.23598260228587]
  We study the study of constrained statistical learning problems, the unconstrained version of which are at the core of virtually all modern information processing. We propose to tackle the constrained statistical problem overcoming its infinite dimensionality, unknown distributions, and constraints by leveraging finite dimensional parameterizations, sample averages, and duality theory. We demonstrate the effectiveness and usefulness of this constrained formulation in a fair learning application.
  arXiv  Detail & Related papers  (2020-02-12T19:12:29Z)
• Selective machine learning of doubly robust functionals [6.880360838661036]
  We propose a selective machine learning framework for making inferences about a finite-dimensional functional defined on a semiparametric model. We introduce a new selection criterion aimed at bias reduction in estimating the functional of interest based on a novel definition of pseudo-risk.
  arXiv  Detail & Related papers  (2019-11-05T19:00:03Z)
• Orthogonal Statistical Learning [49.55515683387805]
  We provide non-asymptotic excess risk guarantees for statistical learning in a setting where the population risk depends on an unknown nuisance parameter. We show that if the population risk satisfies a condition called Neymanity, the impact of the nuisance estimation error on the excess risk bound achieved by the meta-algorithm is of second order.
  arXiv  Detail & Related papers  (2019-01-25T02:21:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.