A new gradient tree boosting framework reduces variance and accelerates performance.
problem High variance in stochastic gradient boosting.
method Combining gradient tree boosting with importance sampling and a regularizer.
result Achieves a linear convergence rate on logistic loss and 2.5x--18x acceleration on LogitBoost and LambdaMART.
Survey of gradient boosting algorithms and their applications.
problem Improving machine learning models through boosting techniques.
method Detailed mathematical frameworks of gradient boosting algorithms.
result Illustration of gradient boosting's role in model optimization and ranking.
Wavelet boosts gradient boosting, improving accuracy, especially in imbalanced data.
problem Improving gradient boosting performance, especially in imbalanced data.
method Wavelet decomposition of trees in gradient boosting.
result Wavelet-based gradient boosting outperforms existing methods, especially in imbalanced data.
A scalable framework for gradient boosting using TensorFlow.
problem Training gradient boosted trees efficiently on large datasets.
method Distributed training architecture, automatic loss differentiation, layer-by-layer boosting, multi-class handling, regularization.
result Faster prediction and smaller ensembles compared to traditional methods.
This research compares gradient and Newton boosting methods in classification and regression.
problem The distinction between gradient descent and Newton updates in boosting algorithms is not well understood.
method Presented a unified framework for gradient and Newton boosting, and compared them with tree base learners.
result Newton boosting outperforms gradient and hybrid boosting in predictive accuracy on most datasets.
Proximal boosting improves gradient boosting for non-differentiable losses.
problem Minimizing non-differentiable losses in prediction models.
method Proximal point algorithm applied to gradient boosting.
result Proximal boosting outperforms gradient boosting in convergence rate and accuracy.
CatBoost boosts performance on datasets with categorical features.
problem Handling categorical features in gradient boosting.
method Gradient boosting library with GPU and CPU implementations.
result Outperforms existing implementations on popular datasets.
This research tackles uncertainty in gradient boosting models using ensemble methods.
problem Quantifying uncertainty in gradient boosting models for high-risk applications.
method Probabilistic ensemble-based framework for gradient boosting classification and regression models.
result Ensembles of gradient boosting models detect anomalous inputs but have limited ability to improve total uncertainty.
Accelerated Gradient Boosting improves performance and sparsity of predictions.
problem Improving prediction accuracy and sparsity in machine learning models.
method Combining gradient boosting with Nesterov's accelerated descent.
result Accelerated Gradient Boosting (AGB) outperforms traditional gradient boosting in terms of sparsity and sensitivity to shrinkage parameters.
SGLB boosts machine learning with Langevin diffusion for multimodal loss functions.
problem Dealing with multimodal loss functions in machine learning.
method Stochastic Gradient Langevin Boosting (SGLB) based on Langevin diffusion equation.
result SGLB guarantees global convergence for multimodal loss functions.
Gradient boosting with randomized trees reduces discontinuities and complexity.
problem Discontinuities in regression functions due to sparse training data.
method Gradient boosting machine with partially randomized decision trees.
result Improves robustness and computational efficiency of gradient boosting.
NGBoost boosts probabilistic predictions using natural gradients.
problem Uncertainty estimation in probabilistic predictions.
method Gradient boosting for probabilistic regression with natural gradient correction.
result NGBoost outperforms existing methods for probabilistic prediction.
Wasserstein gradient boosting predicts probability distributions for supervised learning.
problem Distribution-valued supervised learning where outputs are probability distributions.
method Fits a new weak learner to Wasserstein gradients of loss functionals of probability distributions.
result Superior performance in probabilistic prediction compared to existing methods.
A method to improve gradient boosting models using stacking.
problem Improving the performance of gradient boosting models.
method Proposes a stacking algorithm to learn a meta-model for ensembles of gradient boosting models.
result The proposed approach can be extended to differentiable combination models like neural networks.
Tree-structured boosting connects gradient boosted stumps and full decision trees.
problem Connecting gradient boosted stumps and full decision trees.
method Introducing tree-structured boosting to create a single decision tree.
result Tree-structured boosting produces models equivalent to CART or gradient boosted stumps at the extremes.
Infinitesimal gradient boosting is a new algorithm derived from gradient boosting.
problem Improving the efficiency and smoothness of gradient boosting.
method Introduced a new class of randomized regression trees and used a limit process in vanishing-learning-rate asymptotic.
result Convergence of the stochastic algorithm and characterization of the limiting procedure as a unique solution of a nonlinear ODE.
VeriGB verifies robustness of gradient boosted models.
problem Proving robustness of gradient boosted models.
method VeriGB encodes model and robustness as SMT formula for verification.
result VeriGB proves robustness of large gradient boosted models.
Functional central limit theorem for kernel gradient flow and infinitesimal gradient boosting
problem Fluctuations of boosting processes around their deterministic limit
method Stochastic perturbation analysis of ODEs in Banach spaces
result Rescaled deviations converge to a Gaussian process
Unified framework for statistical inference in gradient boosting regression.
problem Challenges in statistical inference and uncertainty quantification for gradient boosting.
method Integrates dropout or parallel training with regularization for CLT in boosting.
result Increasing dropout rate and parallel trees enhances signal recovery and performance.
GrowNet uses shallow neural networks for gradient boosting, outperforming existing methods.
problem Improving gradient boosting performance through shallow neural networks.
method Unified gradient boosting framework with shallow neural networks as weak learners, incorporating corrective steps.
result GrowNet outperformed state-of-the-art boosting methods in classification, regression, and learning to rank tasks.
New algorithm improves convergence of gradient boosting trees.
problem Global convergence of Newton boosting in tabular machine learning.
method Introduces Gradient Regularized Newton Descent for GBDTs, proving linear convergence for smooth, strongly convex losses and O ( 1 k 2 ) \mathcal{O}(\frac{1}{k^2}) O ( k 2 1 ) rate for general convex losses. result Achieves globally convergent second-order GBDT algorithm with rate matching first-order boosting.
Boost-R uses gradient boosted trees for analyzing recurrence data.
problem Analyzing recurrence data with static and dynamic features.
method Gradient boosted additive trees with time-dependent functions.
result Estimates the cumulative intensity function of recurrent event processes.
Boulevard boosts trees with regularization, converging to a normal distribution.
problem Improving regression model accuracy and uncertainty quantification.
method Regularized stochastic gradient boosting trees with subsampling and shrinkage.
result The algorithm converges to a normal distribution, enabling uncertainty characterization.
Gradient boosting can be seen as Gaussian process inference.
problem Improving uncertainty estimates in out-of-domain detection.
method Gradient boosting reformulated as a kernel method converging to Gaussian process inference.
result Gradient boosting can provide better uncertainty estimates through Monte-Carlo estimation of posterior variance.
FGBoost boosts gradient boosting for complex data.
problem Gradient boosting struggles with non-Euclidean data.
method Introduces FGBoost for geodesic metric spaces.
result FGBoost performs well on complex data.
Gradient boosting method enforced with individual fairness.
problem Enforcing fairness in machine learning models.
method Functional gradient descent on robust loss function.
result Algorithm converges globally and generalizes.
Stagewise boosting improves gradient boosting for distributional regression.
problem Vanishing gradient in gradient boosting for distributional regression leads to suboptimal models.
method Proposes a stagewise boosting-type algorithm for distributional regression, combining stagewise regression ideas with gradient boosting and incorporating a novel regularization method, correlation filtering.
result The proposed algorithm provides better results, especially for complex distributions, by reducing the risk of being trapped in a local optimum.
Gradient boosting improved with lassoed trees achieves faster convergence.
problem Improving gradient boosting convergence in large nonparametric spaces.
method Lassoed gradient boosted trees with early stopping.
result Achieves faster than n − 1 / 4 n^{-1/4} n − 1/4 L2 convergence rate. sGBM speeds up gradient boosting by parallelizing and adapting base learners.
problem Infeasibility of parallelizing GBM training and sub-optimal performance in online settings.
method Integrates multiple differentiable base learners, jointly optimizing them with linear speed-up.
result sGBM achieves higher time efficiency and better accuracy than traditional GBM.
Boosted Frank-Wolfe accelerates optimization for nonconvex problems.
problem Optimizing nonconvex and quasar-convex objectives efficiently.
method Developed a novel step size strategy for stochastic Frank-Wolfe, extending it to various gradient estimators.
result Boosted Frank-Wolfe achieves faster convergence rates than non-boosted Frank-Wolfe.
New GPU algorithm boosts machine learning with larger datasets.
problem Limited GPU memory restricts training data size.
method Out-of-core GPU gradient boosting algorithm.
result Training larger datasets on GPUs without accuracy loss.
Gradient boosting adapted for vector inputs.
problem Applying gradient boosting to multi-class classification problems.
method Extended gradient boosting framework to vector inputs using histogram-based decision trees.
result Efficient algorithm for vector-valued objectives.
Unified probabilistic gradient boosting for entire conditional distribution modeling.
problem Creating accurate probabilistic forecasts from regression tasks.
method Unified probabilistic gradient boosting framework using XGBoost and LightGBM, modeling conditional moments or CDF via Normalizing Flows.
result Achieves state-of-the-art forecast accuracy.
Python library for boosting statistical relational models.
problem Expressing learning and inference problems in statistical relational models.
method Adapting scikit-learn interface for boosted statistical relational models.
result Provides examples for using srlearn.
Two extensions improve gradient boosting for multiclass problems.
problem Multiclass classification with small models.
method Two extensions to gradient boosting: vector-valued trees and layer-by-layer boosting.
result Extensions lead to faster convergence and more compact ensembles.
The paper provides convergence guarantees for multicalibration gradient boosting.
problem Understanding the convergence properties of multicalibration gradient boosting.
method Computational guarantees for multicalibration gradient boosting algorithms, including adaptive variants.
result The magnitude of successive prediction updates decays at O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) , leading to convergence in empirical multicalibration error. Calibrated Boosting-Forest improves ranking and probability calibration in classification tasks.
problem Need for superior ranking power and well-calibrated probability estimates in classification tasks.
method Ensemble of gradient boosting machines supporting both continuous and binary labels.
result Calibrated Boosting-Forest achieves significant improvements in ranking and probability calibration compared to state-of-the-art models.
GBMixed boosts mixed models for clustered data, estimating mean and variance flexibly.
problem Flexible estimation of mean and variance components in clustered data.
method Gradient Boosting framework for linear mixed models with likelihood-based gradients.
result GBMixed accurately recovers complex nonlinear fixed effects and covariances.
Develops a fair tree boosting method for tabular data.
problem Lack of fair classifiers based on decision trees in tabular data.
method Adversarial gradient tree boosting that minimizes adversarial neural network's ability to predict sensitive attributes.
result Achieves higher accuracy while maintaining fairness.
ABHT boosts regression by filtering regions with different smoothness.
problem Improving regression performance through local adaptivity.
method Gradient boosting with adaptive histogram transform.
result ABHT converges faster than PEHT in Hölder continuous spaces.
StructureBoost improves gradient boosting for complex categorical variables efficiently.
problem Efficiently handling complex categorical variables with known structure.
method Two methods to overcome computational obstacles in SCDT enumeration for structured categorical variables.
result StructureBoost outperforms existing packages on complex categorical problems.
Gradient Boosted Feature Selection (GBFS) selects features reliably and scales well.
problem Feature selection in complex datasets.
method Gradient Boosted Trees modification.
result GBFS outperforms other feature selection algorithms on real-world data.
Infinitesimal boosting converges to a deterministic process in large sample limit.
problem Characterizing the asymptotic behavior of infinitesimal gradient boosting in large sample sizes.
method Proving convergence to a deterministic process using large sample theory and differential equations.
result The test error decreases over time in the population limit.
Residual Networks are shown to be equivalent to boosting feature representation.
problem Improving feature representation in deep learning models.
method Proved ResNet's equivalence to Online Gradient Boosting and proposed decision tree residual modules.
result ResNet can achieve Online Gradient Boosting regret bounds through architectural changes.
Gradient Boosted Mixed Models estimate mean and variance components for clustered data.
problem Limited flexibility in linear mixed models for complex settings.
method Gradient Boosting extended to mixed models with likelihood-based gradients and flexible base learners.
result Accurate recovery of variance components and improved predictive accuracy.
Unified comparison of gradient boosting algorithms for insurance claims.
problem Improving predictive accuracy and computational efficiency in insurance claim prediction.
method Unified notation and comprehensive numerical study comparing 12 gradient boosting algorithms on 5 datasets.
result No trade-off between model adequacy and predictive accuracy.
dboost optimizes prediction models for convex cone problems.
problem Optimizing prediction models for decision-making.
method Gradient boosting with implicit differentiation for convex quadratic cone programming.
result dboost reduces out-of-sample decision regret.
Gradient Boosted Normalizing Flows improve flexibility of NFs without increasing complexity.
problem Improving flexibility of normalizing flows without increasing complexity.
method Gradient Boosting applied to normalizing flows to create a mixture model structure.
result GBNFs outperform non-boosted NFs and produce better results with simpler components.