Additive models, such as produced by gradient boosting, and full interaction models, such as classification and regression trees (CART), are widely used algorithms that have been investigated largely in isolation. We show that these models exist along a spectrum, revealing never-before-known connections between these t…
Gradient boosted decision trees are a popular machine learning technique, in part because of their ability to give good accuracy with small models. We describe two extensions to the standard tree boosting algorithm designed to increase this advantage. The first improvement extends the boosting formalism from scalar-val…
Boost-R uses gradient boosted trees for analyzing recurrence data.
problem Analyzing recurrence data with static and dynamic features.
method Gradient boosted additive trees with time-dependent functions.
result Estimates the cumulative intensity function of recurrent event processes.
In this paper, we introduce a new machine learning (ML) model for nonlinear regression called the Boosted Smooth Transition Regression Trees (BooST), which is a combination of boosting algorithms with smooth transition regression trees. The main advantage of the BooST model is the estimation of the derivatives (partial…
Boosting trees predict Twitch subscriptions from user activity.
problem Predicting Twitch user subscriptions from activity data.
method Used boosting trees and target-encodings for high cardinality categoricals.
result User activity can be better predicted than content alone.
DBT combines diffusion models and boosting for supervised learning.
problem Supervised learning problems.
method Diffusion Boosting paradigm and Diffusion Boosted Trees (DBT).
result DBT outperforms deep neural network-based diffusion models and is effective on real-world classification tasks.
SHAP explains boosted trees with additively modeled features.
problem Explaining predictions of boosted trees models with additively modeled features.
method SHAP values for additively modeled features in boosted trees models.
result SHAP dependence plot matches partial dependence plot for additively modeled features.
SMAC method optimizes tree-boosting hyperparameters best.
problem Optimizing hyperparameters for tree-boosting to improve model accuracy.
method Compared and evaluated various hyperparameter optimization methods.
result SMAC method outperforms other methods for hyperparameter tuning.
Two algorithms for interpreting and boosting tree-based models using rule covering.
problem Interpreting and boosting tree-based ensemble methods.
method Mathematical programming models constructed from decision tree rules.
result Selects a few rules that closely match the accuracy of the model.
agtboost speeds up gradient tree boosting with automatic complexity adjustment.
problem Speeding up and simplifying gradient tree boosting computations.
method Adaptive gradient tree boosting with automatic complexity adjustment and feature importance.
result Significant decrease in computation time and simplification of model complexity.
A new method for automatic gradient tree boosting using information theory.
problem Automatic selection of tree complexity and number in gradient boosting.
method Optimism of greedy leaf splitting procedure modeled as a Cox-Ingersoll-Ross process, leading to an information criterion for model selection.
result The method achieves significant speedups (10-1400) compared to xgboost without sacrificing predictive power.
Multivariate boosted trees improve forecasting and control by capturing correlated predictions.
problem Capturing multivariate target cross-correlations and applying structured penalties to predictions.
method A computationally efficient algorithm for fitting multivariate boosted trees.
result Multivariate trees outperform univariate counterparts in correlated prediction scenarios.
Boosted tree method improves MTL in heterogeneous domains.
problem Improving MTL in diverse, domain-specific tasks.
method Two-stage approach: common model for shared features, specific models for task-specific instances.
result Enhanced multi-task learning performance with interpretability.
Classifier evasion consists in finding for a given instance x the nearest instance x′ such that the classifier predictions of x and x′ are different. We present two novel algorithms for systematically computing evasions for tree ensembles such as boosted trees and random forests. Our first algorithm uses a Mixe…
Gradient boosting with randomized trees reduces discontinuities and complexity.
problem Discontinuities in regression functions due to sparse training data.
method Gradient boosting machine with partially randomized decision trees.
result Improves robustness and computational efficiency of gradient boosting.
This paper examines a novel gradient boosting framework for regression. We regularize gradient boosted trees by introducing subsampling and employ a modified shrinkage algorithm so that at every boosting stage the estimate is given by an average of trees. The resulting algorithm, titled Boulevard, is shown to converge …
Boosting meta-trees improve decision tree performance.
problem Overfitting in decision trees.
method Boosting approach to construct multiple meta-trees.
result Ensembles of meta-trees prevent overfitting.
We introduce a novel boosting algorithm called `KTBoost' which combines kernel boosting and tree boosting. In each boosting iteration, the algorithm adds either a regression tree or reproducing kernel Hilbert space (RKHS) regression function to the ensemble of base learners. Intuitively, the idea is that discontinuous …
TF Boosted Trees (TFBT) is a new open-sourced frame-work for the distributed training of gradient boosted trees. It is based on TensorFlow, and its distinguishing features include a novel architecture, automatic loss differentiation, layer-by-layer boosting that results in smaller ensembles and faster prediction, princ…
Tree ensembles are flexible predictive models that can capture relevant variables and to some extent their interactions in a compact and interpretable manner. Most algorithms for obtaining tree ensembles are based on versions of boosting or Random Forest. Previous work showed that boosting algorithms exhibit a cyclic b…
New methods improve prediction performance and reduce computation time in boosting and random forest models.
problem Improving prediction performance and reducing computation time in boosting and random forest models.
method Random tree depth injection approach for Boosting and Random Forests.
result The new methods can improve prediction performance and reduce computation time by up to 40%.
Infinitesimal gradient boosting is a new algorithm derived from gradient boosting.
problem Improving the efficiency and smoothness of gradient boosting.
method Introduced a new class of randomized regression trees and used a limit process in vanishing-learning-rate asymptotic.
result Convergence of the stochastic algorithm and characterization of the limiting procedure as a unique solution of a nonlinear ODE.
Gradient tree boosting is a prediction algorithm that sequentially produces a model in the form of linear combinations of decision trees, by solving an infinite-dimensional optimization problem. We combine gradient boosting and Nesterov's accelerated descent to design a new algorithm, which we call AGB (for Accelerated…
Boosted decision trees typically yield good accuracy, precision, and ROC area. However, because the outputs from boosting are not well calibrated posterior probabilities, boosting yields poor squared error and cross-entropy. We empirically demonstrate why AdaBoost predicts distorted probabilities and examine three cali…
SBPMT combines bagging and boosting for improved classification.
problem Improving classification performance in machine learning.
method Designing SBPMT, a hybrid of bagging and boosting, with PMT as base classifiers.
result SBPMT is consistent and can reduce generalization error with more subagging iterations.
DBDT uses deep boosting decision trees for fraud detection.
problem Fraud detection in imbalanced data.
method Gradient boosting with neural networks (SDT), AUC maximization.
result DBDT significantly improves fraud detection performance.
As an adaptive, interpretable, robust, and accurate meta-algorithm for arbitrary differentiable loss functions, gradient tree boosting is one of the most popular machine learning techniques, though the computational expensiveness severely limits its usage. Stochastic gradient boosting could be adopted to accelerates gr…
Gradient boosting improved with lassoed trees achieves faster convergence.
problem Improving gradient boosting convergence in large nonparametric spaces.
method Lassoed gradient boosted trees with early stopping.
result Achieves faster than n−1/4 L2 convergence rate. AGBoost uses attention weights to improve GBM for regression problems.
problem Improving gradient boosting machine for regression tasks.
method Attention-based modification of GBM with trainable attention weights.
result AGBoost achieves better performance on regression datasets.
A hybrid strategy forecasts short-term loads using Warm-start Gradient Tree Boosting.
problem Lack of effective short-term load forecasting methods.
method Hybrid strategy integrating four different inference models: tree-based ensemble method Warm-start Gradient Tree Boosting (WGTB).
result Demonstrates effectiveness of hybrid strategy on real datasets.
We show that Residual Networks (ResNet) is equivalent to boosting feature representation, without any modification to the underlying ResNet training algorithm. A regret bound based on Online Gradient Boosting theory is proved and suggests that ResNet could achieve Online Gradient Boosting regret bounds through neural n…
Proposes new attribution methods for trees with regularization.
problem Feature attribution for trees trained with regularization.
method Prediction Decomposition Attribution (PreDecomp) and TreeInner.
result TreeInner shows state-of-the-art feature selection performance.
This study extends verifiable learning to boosted tree ensembles, enabling efficient security verification.
problem Efficiently verifying the robustness of boosted tree ensembles against norm-based attackers.
method Formal verification of robustness for large-spread boosted tree ensembles, considering L∞-norm and pseudo-polynomial time for Lp-norm verification. result Polynomial time verification for L∞-norm attackers, NP-hard for other norms, and pseudo-polynomial time for Lp-norm verification. Treeffuser predicts tabular data distributions using gradient-boosted trees.
problem Probabilistic prediction with flexible, non-parametric models.
method Gradient-boosted trees for score estimation in conditional diffusion model.
result Treeffuser outperforms existing methods in probabilistic prediction tasks.
Tuning SVM and boosting models using optimization algorithms.
problem Tuning parameters for SVM and boosting models across various datasets.
method Used grid search to identify parameter ranges and optimization algorithms to select models.
result Optimization algorithms outperformed grid search in selecting well-performing models.
New algorithm improves convergence of gradient boosting trees.
problem Global convergence of Newton boosting in tabular machine learning.
method Introduces Gradient Regularized Newton Descent for GBDTs, proving linear convergence for smooth, strongly convex losses and O(k21) rate for general convex losses. result Achieves globally convergent second-order GBDT algorithm with rate matching first-order boosting.
UnmaskingTrees improves tabular data imputation and generation using gradient-boosted decision trees.
problem Traditional methods outperform advanced deep learning techniques on tabular data imputation benchmarks.
method UnmaskingTrees employs gradient-boosted decision trees to incrementally unmask features for imputation and generation.
result UnmaskingTrees outperforms state-of-the-art methods on tabular imputation and generation benchmarks.
Fair classification has become an important topic in machine learning research. While most bias mitigation strategies focus on neural networks, we noticed a lack of work on fair classifiers based on decision trees even though they have proven very efficient. In an up-to-date comparison of state-of-the-art classificatio…
GAMI-Tree uses model-based trees to fit low-order fANOVA models.
problem Fitting interpretable fANOVA models with low-order interactions.
method GAMI-Tree uses model-based trees and a new interaction filtering method.
result GAMI-Tree outperforms EBM and GAMI-Net in predictive performance and interpretability.
SCORE improves tree-based predictions with boosted residual extraTrees.
problem Improving tree-based prediction models with reduced errors.
method Inspired by representation learning, SCORE uses boosting, regularized regression, and variable selection.
result SCORE provides comparable or superior performance compared to other models.
Hollow-tree Super resolves feature importance in large datasets.
problem Lack of effective scaling for large feature numbers in boosted tree models.
method Hollow-tree Super (HOTS) methodology for feature importance visualization.
result HOTS effectively resolves feature importance and directionality in high-dimensional neuroscientific data.
Boosting trees can test necessary conditions for regression model calibration.
problem Testing calibration and auto-calibration in regression models.
method Using boosting trees to test calibration and auto-calibration.
result Boosting trees prove to be very powerful in testing calibration and auto-calibration in large insurance datasets.
We consider the problem of learning a forest of nonlinear decision rules with general loss functions. The standard methods employ boosted decision trees such as Adaboost for exponential loss and Friedman's gradient boosting for general loss. In contrast to these traditional boosting algorithms that treat a tree learner…
New test improves tree ensemble pruning for better model performance.
problem Lack of robust theoretical justification for penalty terms in tree ensembles.
method Developed a novel hypothesis test for tree ensemble split quality.
result Significant reduction in out-of-sample loss using the new test.
VisRuler simplifies decision extraction from bagged and boosted trees.
problem Complexity and lack of interpretability in ensemble models.
method Visual analytics tool for selecting robust models, important features, and essential decisions.
result Users successfully extracted and explained decisions from ensemble models.
Credit scoring plays a vital role in the field of consumer finance. Survival analysis provides an advanced solution to the credit-scoring problem by quantifying the probability of survival time. In order to deal with highly heterogeneous industrial data collected in Chinese market of consumer finance, we propose a nonp…
Gradient boosted trees outperform other models in predicting corporate bankruptcy.
problem Predicting financial distress of publicly traded U.S. firms.
method Benchmarked various machine learning models using a comprehensive sample of bankruptcies.
result Gradient boosted trees outperform other models in one-year-ahead forecasts.
Forest tree species mapped with high accuracy using satellite data.
problem Classifying dominant tree species in Swedish forests.
method Extreme gradient boosting model with Bayesian optimization, combining Sentinel-1/2 satellite data and field observations.
result Overall accuracy of 85%, F1 score of 0.82, Matthews correlation coefficient of 0.81.