Stagewise boosting improves gradient boosting for distributional regression.
problem Vanishing gradient in gradient boosting for distributional regression leads to suboptimal models.
method Proposes a stagewise boosting-type algorithm for distributional regression, combining stagewise regression ideas with gradient boosting and incorporating a novel regularization method, correlation filtering.
result The proposed algorithm provides better results, especially for complex distributions, by reducing the risk of being trapped in a local optimum.
NGBoost boosts probabilistic predictions using natural gradients.
problem Uncertainty estimation in probabilistic predictions.
method Gradient boosting for probabilistic regression with natural gradient correction.
result NGBoost outperforms existing methods for probabilistic prediction.
Robust boosting improves regression accuracy in noisy data.
problem Handling outliers in non-parametric regression.
method Two-stage approach: robust residual scale minimization followed by bounded loss optimization.
result Robust boosting outperforms standard methods in outlier-prone data.
Unified framework for statistical inference in gradient boosting regression.
problem Challenges in statistical inference and uncertainty quantification for gradient boosting.
method Integrates dropout or parallel training with regularization for CLT in boosting.
result Increasing dropout rate and parallel trees enhances signal recovery and performance.
Gradient boosting with randomized trees reduces discontinuities and complexity.
problem Discontinuities in regression functions due to sparse training data.
method Gradient boosting machine with partially randomized decision trees.
result Improves robustness and computational efficiency of gradient boosting.
Proposes GBBHE for efficient large-scale regression.
problem Large-scale regression problems.
method Gradient Boosting with binary histogram partition and ensemble learning.
result Improves computational efficiency and performance on large datasets.
GrowNet uses shallow neural networks for gradient boosting, outperforming existing methods.
problem Improving gradient boosting performance through shallow neural networks.
method Unified gradient boosting framework with shallow neural networks as weak learners, incorporating corrective steps.
result GrowNet outperformed state-of-the-art boosting methods in classification, regression, and learning to rank tasks.
ABHT boosts regression by filtering regions with different smoothness.
problem Improving regression performance through local adaptivity.
method Gradient boosting with adaptive histogram transform.
result ABHT converges faster than PEHT in Hölder continuous spaces.
A method to improve gradient boosting models using stacking.
problem Improving the performance of gradient boosting models.
method Proposes a stacking algorithm to learn a meta-model for ensembles of gradient boosting models.
result The proposed approach can be extended to differentiable combination models like neural networks.
Parallel gradient boosting speeds up multi-output regression.
problem Efficiently predicting entire conditional distributions.
method Modification of gradient boosting to use a common descent direction.
result Significantly faster and comparable performance to state-of-the-art boosting libraries.
Unified probabilistic gradient boosting for entire conditional distribution modeling.
problem Creating accurate probabilistic forecasts from regression tasks.
method Unified probabilistic gradient boosting framework using XGBoost and LightGBM, modeling conditional moments or CDF via Normalizing Flows.
result Achieves state-of-the-art forecast accuracy.
Efficient boosting method for regression with limited feedback.
problem Online boosting for regression tasks with noisy multi-point bandit feedback.
method Efficient regret minimization method with online boosting algorithm and projection-free online convex optimization.
result Improved state-of-the-art guarantees in efficiency.
Gradient boosting for spatial regression models improves prediction accuracy.
problem Spatial data with autoregressive disturbances.
method Model-based gradient boosting algorithm for spatial regression models.
result Improves prediction accuracy on out-of-sample spatial data.
Gradient boosting adapted for multi-label and multi-output tasks.
problem Joint prediction of multiple classification or regression outputs.
method Gradient tree boosting with random output projections.
result Random projection improves adaptation to different output correlation patterns.
Boost-R uses gradient boosted trees for analyzing recurrence data.
problem Analyzing recurrence data with static and dynamic features.
method Gradient boosted additive trees with time-dependent functions.
result Estimates the cumulative intensity function of recurrent event processes.
A new boosting method corrects endogeneity bias in instrumental variable regression.
problem Endogeneity bias in instrumental variable regression.
method Causal Gradient Boosting (boostIV) that builds on gradient boosting algorithm.
result boostIV is consistent and performs well in finite samples compared to other methods.
Python package for ordinal regression using gradient boosting.
problem Handling ordinal variables in machine learning.
method Gradient boosting with latent variable framework.
result Performs joint optimization of latent function and threshold vector.
Excellent ranking power along with well calibrated probability estimates are needed in many classification tasks. In this paper, we introduce a technique, Calibrated Boosting-Forest that captures both. This novel technique is an ensemble of gradient boosting machines that can support both continuous and binary labels. …
AGBoost uses attention weights to improve GBM for regression problems.
problem Improving gradient boosting machine for regression tasks.
method Attention-based modification of GBM with trainable attention weights.
result AGBoost achieves better performance on regression datasets.
Gradient boosting can be seen as Gaussian process inference.
problem Improving uncertainty estimates in out-of-domain detection.
method Gradient boosting reformulated as a kernel method converging to Gaussian process inference.
result Gradient boosting can provide better uncertainty estimates through Monte-Carlo estimation of posterior variance.
This research tackles uncertainty in gradient boosting models using ensemble methods.
problem Quantifying uncertainty in gradient boosting models for high-risk applications.
method Probabilistic ensemble-based framework for gradient boosting classification and regression models.
result Ensembles of gradient boosting models detect anomalous inputs but have limited ability to improve total uncertainty.
Infinitesimal gradient boosting is a new algorithm derived from gradient boosting.
problem Improving the efficiency and smoothness of gradient boosting.
method Introduced a new class of randomized regression trees and used a limit process in vanishing-learning-rate asymptotic.
result Convergence of the stochastic algorithm and characterization of the limiting procedure as a unique solution of a nonlinear ODE.
Hyperboost uses gradient boosting for hyperparameter optimization, outperforming state-of-the-art methods.
problem Hyperparameter tuning for machine learning algorithms
method Gradient boosting surrogate model with quantile regression and distance metric
result Hyperboost outperforms state-of-the-art techniques in empirical tests
Boosting algorithms are frequently used in applied data science and in research. To date, the distinction between boosting with either gradient descent or second-order Newton updates is often not made in both applied and methodological research, and it is thus implicitly assumed that the difference is irrelevant. The g…
Wasserstein gradient boosting predicts probability distributions for supervised learning.
problem Distribution-valued supervised learning where outputs are probability distributions.
method Fits a new weak learner to Wasserstein gradients of loss functionals of probability distributions.
result Superior performance in probabilistic prediction compared to existing methods.
Paper studies ensemble probabilistic regression trees for smooth approximations.
problem Smooth approximations of regression functions.
method Ensemble versions of probabilistic regression trees.
result Ensemble probabilistic regression trees are consistent and perform well.
NGBoost boosts multivariate probabilistic regression.
problem Joint probabilistic regression for multivariate targets.
method Natural Gradient Boosting for nonparametric modeling.
result Competitive performance in oceanographic velocity prediction.
This paper examines a novel gradient boosting framework for regression. We regularize gradient boosted trees by introducing subsampling and employ a modified shrinkage algorithm so that at every boosting stage the estimate is given by an average of trees. The resulting algorithm, titled Boulevard, is shown to converge …
Additive models, such as produced by gradient boosting, and full interaction models, such as classification and regression trees (CART), are widely used algorithms that have been investigated largely in isolation. We show that these models exist along a spectrum, revealing never-before-known connections between these t…
Study on L2-boosting behavior as learning rate approaches zero.
problem Understanding the asymptotic behavior of L2-boosting algorithms with vanishing learning rates.
method Analyzes L2-boosting for regression with linear base learners, proving a deterministic limit and characterizing it as a solution to a linear differential equation.
result Proves the existence of a unique solution to the limit problem and analyzes the training and test error.
A new method for automatic gradient tree boosting using information theory.
problem Automatic selection of tree complexity and number in gradient boosting.
method Optimism of greedy leaf splitting procedure modeled as a Cox-Ingersoll-Ross process, leading to an information criterion for model selection.
result The method achieves significant speedups (10-1400) compared to xgboost without sacrificing predictive power.
Boosted Frank-Wolfe accelerates optimization for nonconvex problems.
problem Optimizing nonconvex and quasar-convex objectives efficiently.
method Developed a novel step size strategy for stochastic Frank-Wolfe, extending it to various gradient estimators.
result Boosted Frank-Wolfe achieves faster convergence rates than non-boosted Frank-Wolfe.
GWRBoost improves GWR for better spatial relationship quantification.
problem Underfitting in GWR for complex data and lack of explainable quantification.
method Geographically weighted gradient boosting model using localized additive model and gradient boosting optimization.
result Significant improvement in RMSE and AICc compared to classic GWR.
New statistical methods improve explainability of boosting models.
problem Uncertainty quantification for boosting models is computationally intensive and hard to interpret.
method Derive methods for statistical inference using gradient boosting and Boulevard regularization.
result Achieve asymptotically normal predictions with theoretical guarantees and runtime independent of data size.
ScoreStop uses gradient tests to stop gradient boosting early.
problem Overfitting in gradient boosted decision trees.
method ScoreStop uses a functional score test based on gradients to stop boosting.
result ScoreStop is competitive with loss-based early stopping methods.
Enhances GBDT robustness with one-hot encoding and regularization.
problem Low robustness of GBDT models against covariate perturbation.
method One-hot encoding to linear framework, risk decomposition, L1 or L2 regularization. result Regularization enhances GBDT robustness.
Scaling regression to large datasets is a common problem in many application areas. We propose a two step approach to scaling regression to large datasets. Using a regression tree (CART) to segment the large dataset constitutes the first step of this approach. The second step of this approach is to develop a suitable r…
Predicts arterial road incident duration with extreme gradient boosting.
problem Predicting incident duration on arterial roads, especially with limited data.
method Bi-level framework combining classification and regression models.
result Extreme gradient boosting outperformed other models by 53%.
Gradient Boosted Mixed Models estimate mean and variance components for clustered data.
problem Limited flexibility in linear mixed models for complex settings.
method Gradient Boosting extended to mixed models with likelihood-based gradients and flexible base learners.
result Accurate recovery of variance components and improved predictive accuracy.
Ability for accurate hospital case cost modelling and prediction is critical for efficient health care financial management and budgetary planning. A variety of regression machine learning algorithms are known to be effective for health care cost predictions. The purpose of this experiment was to build an Azure Machine…
Gradient boosting of regression trees is a competitive procedure for learning predictive models of continuous data that fits the data with an additive non-parametric model. The classic version of gradient boosting assumes that the data is independent and identically distributed. However, relational data with interdepen…
PathBoost boosts graph-level predictions using path-based features.
problem Graph-level classification and regression challenges.
method Gradient tree boosting method for graph-level prediction.
result PathBoost outperforms graph neural networks and graph kernel approaches in many cases.
Boosting methods for interval-censored data improve predictive accuracy in survival analysis.
problem Handling interval-censored data in survival analysis and time-to-event studies.
method Nonparametric boosting methods using censoring unbiased transformations and functional gradient descent.
result Effective boosting methods for regression and classification with interval-censored data, offering robust performance.
Boosting method adapted for non-square loss functions.
problem Applying gradient boosting to non-square loss functions.
method Introducing column measure ν^(L_2) to adapt L2-Boosting for different loss functions.
result L2-Boosting can be modified to select variables according to different loss functions.
Gradient boosting estimates Riesz representer for causal inference.
problem Estimating causal quantities using traditional methods is challenging and prone to variance issues.
method Gradient boosting algorithm to directly estimate Riesz representer.
result Gradient boosting performs similarly or better than traditional methods in estimating causal quantities.
Multivariate boosted trees improve forecasting and control by capturing correlated predictions.
problem Capturing multivariate target cross-correlations and applying structured penalties to predictions.
method A computationally efficient algorithm for fitting multivariate boosted trees.
result Multivariate trees outperform univariate counterparts in correlated prediction scenarios.
BoostForest combines multiple BoostTree models for improved accuracy.
problem Improving ensemble learning performance.
method BoostTree uses gradient boosting and random cut-points. BoostForest bootstraps training data and randomly samples parameters.
result BoostForest outperforms classical ensemble methods on 35 datasets.
Combines coarse learners for nonparametric probabilistic regression.
problem Predicting full probability density functions without strong assumptions.
method Combining gradient boosted forests trained on coarsened target values.
result Prediction intervals have high fidelity and provide valuable insights.