Parallel gradient boosting for flexible estimation of conditional distributions

• URL: http://arxiv.org/abs/2607.13550v1
• Date: Wed, 15 Jul 2026 07:56:09 GMT
• Title: Parallel gradient boosting for flexible estimation of conditional distributions
• Abstract summary: We study a modification of the gradient boosting algorithm, which we call parallel gradient boosting.<n>By doing so, only one base model is needed at each iteration, regardless of the number of targets.<n>We show that in such a setting, it provides predictions of similar quality to state-of-the-art boosting libraries.
• Score: 2.453409145803731
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Boosting is one of the most successful learning techniques for standard classification and regression tasks. Its extension to multi-output prediction problems has found an increasing number of applications in recent years. Among them is the prediction of entire conditional distributions rather than single functionals, which can often be framed as a multi-output regression problem, for example multiple quantile regression. Addressing such problems with classical implementations of boosting is computationally challenging, because usually one base model is trained for each target at every iteration. More efficient variants of boosting have been proposed to speed up training, but they tend to be tied to specific loss functions and classes of base learners, usually decision trees. In this work, we study a modification of the gradient boosting algorithm, which we call parallel gradient boosting, designed to circumvent all these limitations. The core idea is to use a common descent direction for all training observations. By doing so, only one base model is needed at each iteration, regardless of the number of targets, which allows for considerable performance gains. We establish sufficient conditions for the convergence of the algorithm, whose practical use is introduced via the multiple quantile regression setting. We show that in such a setting, it provides predictions of similar quality to state-of-the-art boosting libraries such as XGBoost, while being faster by several orders of magnitude. Then, we evaluate the properties of the resulting conditional distribution estimator, which is shown empirically to outperform other nonparametric and semiparametric estimators, especially in high-dimensional settings and in the presence of mixed and/or missing covariates.

Related papers

• Federated stochastic bilevel optimization with fully first-order gradients [57.1147486991903]
  Federated matrices bilevel optimization has been actively studied in recent years due to its widespread applications in machine learning.<n>Most existing federated bilevel optimization algorithms require the computation of second-order Hessian and Jacobian matrices.<n>We propose a novel federated convergence variance-reduced bilevel descent algorithm that relies solely on first-order oracles.
  arXiv  Detail & Related papers  (2026-09-14T21:12:59Z)
• Adaptive Sampled Softmax with Inverted Multi-Index: Methods, Theory and Applications [79.53938312089308]
  The MIDX-Sampler is a novel adaptive sampling strategy based on an inverted multi-index approach.<n>Our method is backed by rigorous theoretical analysis, addressing key concerns such as sampling bias, gradient bias, convergence rates, and generalization error bounds.
  arXiv  Detail & Related papers  (2025-01-15T04:09:21Z)
• From Point to probabilistic gradient boosting for claim frequency and severity prediction [1.3812010983144802]
  We present in a unified notation, and contrast, all the existing point and probabilistic gradient boosting for decision tree algorithms.<n>We compare their performance on five publicly available datasets for claim frequency and severity.<n>We find that there is no trade-off between model adequacy and predictive accuracy: both are achievable simultaneously.
  arXiv  Detail & Related papers  (2024-12-19T14:50:10Z)
• Achieving More with Less: A Tensor-Optimization-Powered Ensemble Method [53.170053108447455]
  Ensemble learning is a method that leverages weak learners to produce a strong learner. We design a smooth and convex objective function that leverages the concept of margin, making the strong learner more discriminative. We then compare our algorithm with random forests of ten times the size and other classical methods across numerous datasets.
  arXiv  Detail & Related papers  (2024-08-06T03:42:38Z)
• Quantile Extreme Gradient Boosting for Uncertainty Quantification [1.7685947618629572]
  Extreme Gradient Boosting (XGBoost) is one of the most popular machine learning (ML) methods. We propose enhancements to XGBoost whereby a modified quantile regression is used as the objective function to estimate uncertainty (QXGBoost) Our proposed method had comparable or better performance than the uncertainty estimates generated for regular and quantile light gradient boosting.
  arXiv  Detail & Related papers  (2023-04-23T19:46:19Z)
• Condensed Gradient Boosting [0.0]
  We propose the use of multi-output regressors as base models to handle the multi-class problem as a single task. An extensive comparison with other multi-ouptut based gradient boosting methods is carried out in terms of generalization and computational efficiency.
  arXiv  Detail & Related papers  (2022-11-26T15:53:19Z)
• Flexible Model Aggregation for Quantile Regression [92.63075261170302]
  Quantile regression is a fundamental problem in statistical learning motivated by a need to quantify uncertainty in predictions. We investigate methods for aggregating any number of conditional quantile models. All of the models we consider in this paper can be fit using modern deep learning toolkits.
  arXiv  Detail & Related papers  (2021-02-26T23:21:16Z)
• Self Normalizing Flows [65.73510214694987]
  We propose a flexible framework for training normalizing flows by replacing expensive terms in the gradient by learned approximate inverses at each layer. This reduces the computational complexity of each layer's exact update from $mathcalO(D3)$ to $mathcalO(D2)$. We show experimentally that such models are remarkably stable and optimize to similar data likelihood values as their exact gradient counterparts.
  arXiv  Detail & Related papers  (2020-11-14T09:51:51Z)
• Multivariate Boosted Trees and Applications to Forecasting and Control [0.0]
  Gradient boosted trees are non-parametric regressors that exploit sequential model fitting and gradient descent to minimize a specific loss function. In this paper, we present a computationally efficient algorithm for fitting multivariate boosted trees.
  arXiv  Detail & Related papers  (2020-03-08T19:26:59Z)
• Robust Boosting for Regression Problems [0.0]
  Gradient boosting algorithms construct a regression predictor using a linear combination of base learners'' The robust boosting algorithm is based on a two-stage approach, similar to boosting is done for robust linear regression. When no atypical observations are present, the robust boosting approach works as well as a standard gradient boosting with a squared loss.
  arXiv  Detail & Related papers  (2020-02-06T01:12:45Z)
• Variance Reduction with Sparse Gradients [82.41780420431205]
  Variance reduction methods such as SVRG and SpiderBoost use a mixture of large and small batch gradients. We introduce a new sparsity operator: The random-top-k operator. Our algorithm consistently outperforms SpiderBoost on various tasks including image classification, natural language processing, and sparse matrix factorization.
  arXiv  Detail & Related papers  (2020-01-27T08:23:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.