Gradient boosting with randomized trees reduces discontinuities and complexity.
problem Discontinuities in regression functions due to sparse training data.
method Gradient boosting machine with partially randomized decision trees.
result Improves robustness and computational efficiency of gradient boosting.
Boost-R uses gradient boosted trees for analyzing recurrence data.
problem Analyzing recurrence data with static and dynamic features.
method Gradient boosted additive trees with time-dependent functions.
result Estimates the cumulative intensity function of recurrent event processes.
Paper studies ensemble probabilistic regression trees for smooth approximations.
problem Smooth approximations of regression functions.
method Ensemble versions of probabilistic regression trees.
result Ensemble probabilistic regression trees are consistent and perform well.
Infinitesimal gradient boosting is a new algorithm derived from gradient boosting.
problem Improving the efficiency and smoothness of gradient boosting.
method Introduced a new class of randomized regression trees and used a limit process in vanishing-learning-rate asymptotic.
result Convergence of the stochastic algorithm and characterization of the limiting procedure as a unique solution of a nonlinear ODE.
AGBoost uses attention weights to improve GBM for regression problems.
problem Improving gradient boosting machine for regression tasks.
method Attention-based modification of GBM with trainable attention weights.
result AGBoost achieves better performance on regression datasets.
Additive models, such as produced by gradient boosting, and full interaction models, such as classification and regression trees (CART), are widely used algorithms that have been investigated largely in isolation. We show that these models exist along a spectrum, revealing never-before-known connections between these t…
This paper examines a novel gradient boosting framework for regression. We regularize gradient boosted trees by introducing subsampling and employ a modified shrinkage algorithm so that at every boosting stage the estimate is given by an average of trees. The resulting algorithm, titled Boulevard, is shown to converge …
A new method for automatic gradient tree boosting using information theory.
problem Automatic selection of tree complexity and number in gradient boosting.
method Optimism of greedy leaf splitting procedure modeled as a Cox-Ingersoll-Ross process, leading to an information criterion for model selection.
result The method achieves significant speedups (10-1400) compared to xgboost without sacrificing predictive power.
In many applications of supervised learning, multiple classification or regression outputs have to be predicted jointly. We consider several extensions of gradient boosting to address such problems. We first propose a straightforward adaptation of gradient boosting exploiting multiple output regression trees as base le…
Multivariate boosted trees improve forecasting and control by capturing correlated predictions.
problem Capturing multivariate target cross-correlations and applying structured penalties to predictions.
method A computationally efficient algorithm for fitting multivariate boosted trees.
result Multivariate trees outperform univariate counterparts in correlated prediction scenarios.
GrowNet uses shallow neural networks for gradient boosting, outperforming existing methods.
problem Improving gradient boosting performance through shallow neural networks.
method Unified gradient boosting framework with shallow neural networks as weak learners, incorporating corrective steps.
result GrowNet outperformed state-of-the-art boosting methods in classification, regression, and learning to rank tasks.
Scaling regression to large datasets is a common problem in many application areas. We propose a two step approach to scaling regression to large datasets. Using a regression tree (CART) to segment the large dataset constitutes the first step of this approach. The second step of this approach is to develop a suitable r…
Enhances GBDT robustness with one-hot encoding and regularization.
problem Low robustness of GBDT models against covariate perturbation.
method One-hot encoding to linear framework, risk decomposition, L1 or L2 regularization. result Regularization enhances GBDT robustness.
Unified framework for statistical inference in gradient boosting regression.
problem Challenges in statistical inference and uncertainty quantification for gradient boosting.
method Integrates dropout or parallel training with regularization for CLT in boosting.
result Increasing dropout rate and parallel trees enhances signal recovery and performance.
Boosting algorithms are frequently used in applied data science and in research. To date, the distinction between boosting with either gradient descent or second-order Newton updates is often not made in both applied and methodological research, and it is thus implicitly assumed that the difference is irrelevant. The g…
Proposes GBBHE for efficient large-scale regression.
problem Large-scale regression problems.
method Gradient Boosting with binary histogram partition and ensemble learning.
result Improves computational efficiency and performance on large datasets.
Robust boosting improves regression accuracy in noisy data.
problem Handling outliers in non-parametric regression.
method Two-stage approach: robust residual scale minimization followed by bounded loss optimization.
result Robust boosting outperforms standard methods in outlier-prone data.
Unified probabilistic gradient boosting for entire conditional distribution modeling.
problem Creating accurate probabilistic forecasts from regression tasks.
method Unified probabilistic gradient boosting framework using XGBoost and LightGBM, modeling conditional moments or CDF via Normalizing Flows.
result Achieves state-of-the-art forecast accuracy.
DBT combines diffusion models and boosting for supervised learning.
problem Supervised learning problems.
method Diffusion Boosting paradigm and Diffusion Boosted Trees (DBT).
result DBT outperforms deep neural network-based diffusion models and is effective on real-world classification tasks.
Treeffuser predicts tabular data distributions using gradient-boosted trees.
problem Probabilistic prediction with flexible, non-parametric models.
method Gradient-boosted trees for score estimation in conditional diffusion model.
result Treeffuser outperforms existing methods in probabilistic prediction tasks.
In this paper, we introduce a new machine learning (ML) model for nonlinear regression called the Boosted Smooth Transition Regression Trees (BooST), which is a combination of boosting algorithms with smooth transition regression trees. The main advantage of the BooST model is the estimation of the derivatives (partial…
A new gradient tree boosting framework reduces variance and accelerates performance.
problem High variance in stochastic gradient boosting.
method Combining gradient tree boosting with importance sampling and a regularizer.
result Achieves a linear convergence rate on logistic loss and 2.5x--18x acceleration on LogitBoost and LambdaMART.
Develops RKHS framework for analyzing tree ensembles.
problem Analyzing the theoretical properties of tree ensembles.
method Reproducing Kernel Hilbert Spaces (RKHS) for tree ensembles.
result Characterizes Random Forest predictor as minimizer of a penalized empirical risk functional in RKHS.
BoostForest combines multiple BoostTree models for improved accuracy.
problem Improving ensemble learning performance.
method BoostTree uses gradient boosting and random cut-points. BoostForest bootstraps training data and randomly samples parameters.
result BoostForest outperforms classical ensemble methods on 35 datasets.
We introduce a novel boosting algorithm called `KTBoost' which combines kernel boosting and tree boosting. In each boosting iteration, the algorithm adds either a regression tree or reproducing kernel Hilbert space (RKHS) regression function to the ensemble of base learners. Intuitively, the idea is that discontinuous …
LoBoost improves local conformal prediction for gradient-boosted trees without extra data splits.
problem Quantifying uncertainty in gradient-boosted tree predictions.
method Model-native local conformal prediction using leaf structure.
result Competitive interval quality and improved test MSE with large calibration speedups.
Gradient boosting can be seen as Gaussian process inference.
problem Improving uncertainty estimates in out-of-domain detection.
method Gradient boosting reformulated as a kernel method converging to Gaussian process inference.
result Gradient boosting can provide better uncertainty estimates through Monte-Carlo estimation of posterior variance.
Bayesian GBMs improve predictive uncertainty calibration for tabular data.
problem Lack of well-calibrated predictive uncertainties in gradient boosting machines.
method Variational inference with soft decision trees.
result Variational soft GBMs provide useful uncertainty estimates and maintain good predictive performance.
New trees-based models handle correlated data better.
problem Standard trees-based models ignore correlation structure.
method Explicitly accounts for correlation structure in splitting criterion, stopping rules, and fitted values.
result New approach superior to standard models in simulations and real data.
New statistical methods improve explainability of boosting models.
problem Uncertainty quantification for boosting models is computationally intensive and hard to interpret.
method Derive methods for statistical inference using gradient boosting and Boulevard regularization.
result Achieve asymptotically normal predictions with theoretical guarantees and runtime independent of data size.
Ability for accurate hospital case cost modelling and prediction is critical for efficient health care financial management and budgetary planning. A variety of regression machine learning algorithms are known to be effective for health care cost predictions. The purpose of this experiment was to build an Azure Machine…
SCORE improves tree-based predictions with boosted residual extraTrees.
problem Improving tree-based prediction models with reduced errors.
method Inspired by representation learning, SCORE uses boosting, regularized regression, and variable selection.
result SCORE provides comparable or superior performance compared to other models.
AF improves classification models by adaptively weighting trees.
problem Improving classification model performance.
method AF combines OP2T for input-dependent weights and MIO for dynamic refinement.
result AF consistently outperforms RF, XGBoost, and other weighted RF.
Parallel gradient boosting speeds up multi-output regression.
problem Efficiently predicting entire conditional distributions.
method Modification of gradient boosting to use a common descent direction.
result Significantly faster and comparable performance to state-of-the-art boosting libraries.
Develops a fair tree boosting method for tabular data.
problem Lack of fair classifiers based on decision trees in tabular data.
method Adversarial gradient tree boosting that minimizes adversarial neural network's ability to predict sensitive attributes.
result Achieves higher accuracy while maintaining fairness.
The paper proposes blending gradient boosted trees and neural networks for hierarchical time series forecasting.
problem Point and probabilistic forecasting of hierarchical time series.
method A blending methodology of gradient boosted trees and neural networks, with feature engineering and diverse model selection.
result Ranked within the gold medal range in both Accuracy and Uncertainty tracks of the M5 Competition.
This research tackles uncertainty in gradient boosting models using ensemble methods.
problem Quantifying uncertainty in gradient boosting models for high-risk applications.
method Probabilistic ensemble-based framework for gradient boosting classification and regression models.
result Ensembles of gradient boosting models detect anomalous inputs but have limited ability to improve total uncertainty.
SnapBoost uses random base hypothesis classes to improve gradient boosting performance.
problem Improving gradient boosting performance.
method Heterogeneous Newton Boosting Machine (HNBM) with variable base hypothesis classes.
result SnapBoost achieves better generalization loss than competing frameworks.
TF Boosted Trees (TFBT) is a new open-sourced frame-work for the distributed training of gradient boosted trees. It is based on TensorFlow, and its distinguishing features include a novel architecture, automatic loss differentiation, layer-by-layer boosting that results in smaller ensembles and faster prediction, princ…
Gradient boosting improved with lassoed trees achieves faster convergence.
problem Improving gradient boosting convergence in large nonparametric spaces.
method Lassoed gradient boosted trees with early stopping.
result Achieves faster than n−1/4 L2 convergence rate. agtboost speeds up gradient tree boosting with automatic complexity adjustment.
problem Speeding up and simplifying gradient tree boosting computations.
method Adaptive gradient tree boosting with automatic complexity adjustment and feature importance.
result Significant decrease in computation time and simplification of model complexity.
pGMM kernel outperforms ordinary ridge regression and RBF kernel ridge regression without tuning.
problem Comparing pGMM kernel regression with other ridge regression methods.
method Implemented and compared pGMM kernel regression with ordinary ridge regression and RBF kernel ridge regression.
result pGMM kernel performs well without tuning and can match boosted trees with parameter tuning.
Wasserstein gradient boosting predicts probability distributions for supervised learning.
problem Distribution-valued supervised learning where outputs are probability distributions.
method Fits a new weak learner to Wasserstein gradients of loss functionals of probability distributions.
result Superior performance in probabilistic prediction compared to existing methods.
We propose to use boosted regression trees as a way to compute human-interpretable solutions to reinforcement learning problems. Boosting combines several regression trees to improve their accuracy without significantly reducing their inherent interpretability. Prior work has focused independently on reinforcement lear…
Proposes a simple neural network model similar to gradient boosted decision trees.
problem Building a neural network equivalent to gradient boosted decision trees.
method Converts an ensemble of decision trees to a neural network, relaxes properties, and trains a simple neural network model.
result The proposed Hammock model achieves similar performance to gradient boosted decision trees.
New algorithm improves convergence of gradient boosting trees.
problem Global convergence of Newton boosting in tabular machine learning.
method Introduces Gradient Regularized Newton Descent for GBDTs, proving linear convergence for smooth, strongly convex losses and O(k21) rate for general convex losses. result Achieves globally convergent second-order GBDT algorithm with rate matching first-order boosting.
Gradient tree boosting is a prediction algorithm that sequentially produces a model in the form of linear combinations of decision trees, by solving an infinite-dimensional optimization problem. We combine gradient boosting and Nesterov's accelerated descent to design a new algorithm, which we call AGB (for Accelerated…
Technology and collaboration enable dramatic increases in the size of psychological and psychiatric data collections, but finding structure in these large data sets with many collected variables is challenging. Decision tree ensembles like random forests (Strobl, Malley, and Tutz, 2009) are a useful tool for finding st…