Tree-structured boosting connects gradient boosted stumps and full decision trees.
problem Connecting gradient boosted stumps and full decision trees.
method Introducing tree-structured boosting to create a single decision tree.
result Tree-structured boosting produces models equivalent to CART or gradient boosted stumps at the extremes.
Proposes a simple neural network model similar to gradient boosted decision trees.
problem Building a neural network equivalent to gradient boosted decision trees.
method Converts an ensemble of decision trees to a neural network, relaxes properties, and trains a simple neural network model.
result The proposed Hammock model achieves similar performance to gradient boosted decision trees.
Residual Networks are shown to be equivalent to boosting feature representation.
problem Improving feature representation in deep learning models.
method Proved ResNet's equivalence to Online Gradient Boosting and proposed decision tree residual modules.
result ResNet can achieve Online Gradient Boosting regret bounds through architectural changes.
Gradient boosting with randomized trees reduces discontinuities and complexity.
problem Discontinuities in regression functions due to sparse training data.
method Gradient boosting machine with partially randomized decision trees.
result Improves robustness and computational efficiency of gradient boosting.
Proposes mGBDTs for learning hierarchical representations in gradient boosting decision trees.
problem Inability of gradient boosting decision trees to learn hierarchical representations.
method Introduces multi-layered GBDT forest (mGBDTs) with explicit emphasis on hierarchical learning.
result Jointly trained mGBDTs can learn hierarchical representations effectively without backpropagation.
New algorithm improves convergence of gradient boosting trees.
problem Global convergence of Newton boosting in tabular machine learning.
method Introduces Gradient Regularized Newton Descent for GBDTs, proving linear convergence for smooth, strongly convex losses and O(k21) rate for general convex losses. result Achieves globally convergent second-order GBDT algorithm with rate matching first-order boosting.
DBDT uses deep boosting decision trees for fraud detection.
problem Fraud detection in imbalanced data.
method Gradient boosting with neural networks (SDT), AUC maximization.
result DBDT significantly improves fraud detection performance.
AGBoost uses attention weights to improve GBM for regression problems.
problem Improving gradient boosting machine for regression tasks.
method Attention-based modification of GBM with trainable attention weights.
result AGBoost achieves better performance on regression datasets.
Gradient boosting adapted for vector inputs.
problem Applying gradient boosting to multi-class classification problems.
method Extended gradient boosting framework to vector inputs using histogram-based decision trees.
result Efficient algorithm for vector-valued objectives.
SketchBoost accelerates GBDT for multioutput problems up to 40x.
problem Efficiently training GBDT for multioutput problems with high-dimensional outputs.
method Approximate computation of scoring function for faster decision tree splitting.
result SketchBoost speeds up GBDT training by up to 40 times.
We consider the problem of learning a forest of nonlinear decision rules with general loss functions. The standard methods employ boosted decision trees such as Adaboost for exponential loss and Friedman's gradient boosting for general loss. In contrast to these traditional boosting algorithms that treat a tree learner…
sGBM speeds up gradient boosting by parallelizing and adapting base learners.
problem Infeasibility of parallelizing GBM training and sub-optimal performance in online settings.
method Integrates multiple differentiable base learners, jointly optimizing them with linear speed-up.
result sGBM achieves higher time efficiency and better accuracy than traditional GBM.
Develops a new method for decision trees using categorical variable structure.
problem Lack of structure in treating categorical variables as predictors.
method Introduces a mathematical framework to represent categorical structure and generalizes decision trees to utilize this structure.
result Improves prediction accuracy on weather data using the new method.
Accelerated Gradient Boosting improves performance and sparsity of predictions.
problem Improving prediction accuracy and sparsity in machine learning models.
method Combining gradient boosting with Nesterov's accelerated descent.
result Accelerated Gradient Boosting (AGB) outperforms traditional gradient boosting in terms of sparsity and sensitivity to shrinkage parameters.
Converts GBDT trees to neural networks for online updates.
problem Performance loss in converting GBDT trees to neural networks.
method Converts existing GBDT implementations to neural network architectures, allowing online updates of decision splits.
result Learning bounds for neural network architecture with updated splits.
We generate counterfactual explanations for tree-based boosting ensembles.
problem Understanding how tree-based models make predictions.
method Extending a method for random forests to GBDTs, accounting for tree sequential dependency and negative gradients.
result A method to generate counterfactual explanations for GBDTs.
GPU acceleration for tree boosting improves speed and scalability.
problem Scalability and performance issues in GPU-based tree building algorithms.
method Histogram-based algorithm for approximate split finding on GPUs.
result 7-8 times faster training on CPU and 25 times faster on Xeon server.
Two extensions improve gradient boosting for multiclass problems.
problem Multiclass classification with small models.
method Two extensions to gradient boosting: vector-valued trees and layer-by-layer boosting.
result Extensions lead to faster convergence and more compact ensembles.
Develops a fair tree boosting method for tabular data.
problem Lack of fair classifiers based on decision trees in tabular data.
method Adversarial gradient tree boosting that minimizes adversarial neural network's ability to predict sensitive attributes.
result Achieves higher accuracy while maintaining fairness.
FPGA-based logic architecture speeds up GBDT training 259x.
problem Training efficiency and power consumption in GBDT models.
method Implemented logic architecture on FPGA, compared with software libraries.
result Training speed 26-259x faster, power efficiency 90-1,104x higher.
agtboost speeds up gradient tree boosting with automatic complexity adjustment.
problem Speeding up and simplifying gradient tree boosting computations.
method Adaptive gradient tree boosting with automatic complexity adjustment and feature importance.
result Significant decrease in computation time and simplification of model complexity.
Improves GBDT accuracy with differential privacy.
problem Balancing privacy and accuracy in GBDT models.
method Adaptive gradient control and novel boosting framework for privacy budget allocation.
result Achieves better model accuracy with differential privacy.
StructureBoost improves gradient boosting for complex categorical variables efficiently.
problem Efficiently handling complex categorical variables with known structure.
method Two methods to overcome computational obstacles in SCDT enumeration for structured categorical variables.
result StructureBoost outperforms existing packages on complex categorical problems.
New sampling technique improves boosting model accuracy.
problem Improving generalization performance and learning time in stochastic gradient boosting.
method Formulated optimization problem to maximize estimation accuracy, leading to Minimal Variance Sampling (MVS).
result MVS significantly increases model quality and reduces the number of examples needed.
DBT combines diffusion models and boosting for supervised learning.
problem Supervised learning problems.
method Diffusion Boosting paradigm and Diffusion Boosted Trees (DBT).
result DBT outperforms deep neural network-based diffusion models and is effective on real-world classification tasks.
Bayesian GBMs improve predictive uncertainty calibration for tabular data.
problem Lack of well-calibrated predictive uncertainties in gradient boosting machines.
method Variational inference with soft decision trees.
result Variational soft GBMs provide useful uncertainty estimates and maintain good predictive performance.
Method finds influential training samples for GBDT models efficiently.
problem Finding influential training samples for GBDT models.
method Leave-one-out retraining, extending to non-parametric GBDT ensembles, and approximations.
result Efficiently finds influential training samples for GBDT models.
The paper proposes a method to assess and improve data quality using GBDT training dynamics.
problem Improving data quality in datasets with noisy labels and varying contributions.
method Metrics computed from training dynamics of Gradient Boosting Decision Trees (GBDTs).
result The method achieved the best results compared to other approaches.
Asynch-SGBDT speeds up GBDT training on Parameter Server.
problem Training GBDT on Parameter Server is slow due to computational resources and time.
method Developed asynch-SGBDT, an asynchronous parallel method for GBDT training.
result Asynch-SGBDT achieves linear speedup in asynchronous parallel training.
Decision trees perform well in complex interactions, even when interactions are not fully accounted for.
problem Interpreting complex interactions in machine learning models.
method Experiments on datasets and two methods for robust GLMs.
result Tree depth compensates for model misspecification, enhancing performance in complex scenarios.
Improves decision tree methods for high-dimensional, sparse input spaces.
problem Scalable supervised learning for high-dimensional, sparse inputs and large datasets.
method Random Forest and Gradient Boosting with random projections and sparsity.
result Improved accuracy and efficiency in multi-label and multi-output learning.
UnmaskingTrees improves tabular data imputation and generation using gradient-boosted decision trees.
problem Traditional methods outperform advanced deep learning techniques on tabular data imputation benchmarks.
method UnmaskingTrees employs gradient-boosted decision trees to incrementally unmask features for imputation and generation.
result UnmaskingTrees outperforms state-of-the-art methods on tabular imputation and generation benchmarks.
Enhances GBDT robustness with one-hot encoding and regularization.
problem Low robustness of GBDT models against covariate perturbation.
method One-hot encoding to linear framework, risk decomposition, L1 or L2 regularization. result Regularization enhances GBDT robustness.
Dynamic CBDT improves treatment effect estimation in clinical data.
problem Estimating heterogeneous treatment effects in observational data with high accuracy and interpretability.
method Dynamic Regularized Causal Boosted Decision Trees (CBDT) integrating variance regularization and calibration.
result Significantly improved estimation accuracy and reliable coverage of true treatment effects.
SnapBoost uses random base hypothesis classes to improve gradient boosting performance.
problem Improving gradient boosting performance.
method Heterogeneous Newton Boosting Machine (HNBM) with variable base hypothesis classes.
result SnapBoost achieves better generalization loss than competing frameworks.
ENTMOOT integrates tree models for better decision-making and optimization.
problem Difficult optimization of tree models and lack of reliable uncertainty measures.
method Integrates already trained tree models into optimization problems with a reliable uncertainty measure.
result Proves globally optimal solutions for optimization problems.
Boosting meta-trees improve decision tree performance.
problem Overfitting in decision trees.
method Boosting approach to construct multiple meta-trees.
result Ensembles of meta-trees prevent overfitting.
Unified comparison of gradient boosting algorithms for insurance claims.
problem Improving predictive accuracy and computational efficiency in insurance claim prediction.
method Unified notation and comprehensive numerical study comparing 12 gradient boosting algorithms on 5 datasets.
result No trade-off between model adequacy and predictive accuracy.
Tree prototypes improve tree ensemble interpretability.
problem Making tree ensembles interpretable.
method Introducing prototypes, a new distance for GBTs, and adaptive selection methods.
result Prototypes can perform as well as or better than original tree ensembles when used as nearest-prototype classifiers.
DOFEN improves DNN performance on tabular data benchmarks.
problem DOFEN tackles the performance gap between DNNs and tree-based models on tabular data.
method DOFEN uses a two-level rODT forest ensembling process inspired by oblivious decision trees.
result DOFEN achieves state-of-the-art results on the Tabular Benchmark.
Research creates a machine learning model for predicting TAVI patient mortality.
problem Lack of robust risk scores for TAVI patients.
method Gradient boosting on decision trees, feature analysis and selection, model validation.
result Model outperforms existing risk scores with AUC of 0.83.
GrowNet uses shallow neural networks for gradient boosting, outperforming existing methods.
problem Improving gradient boosting performance through shallow neural networks.
method Unified gradient boosting framework with shallow neural networks as weak learners, incorporating corrective steps.
result GrowNet outperformed state-of-the-art boosting methods in classification, regression, and learning to rank tasks.
Multivariate tree boosting finds structure in large data sets with multiple outcomes.
problem Challenges in finding structure in large, multivariate data sets.
method Gradient Boosted Regression Trees (mvtboost) for continuous, multivariate outcomes.
result Identifies predictors with non-linear effects and interactions without specification.
New algorithm speeds up robustness verification for tree-based models.
problem Formal robustness verification of tree-based models, especially ensembles.
method Reformulated as max-clique problem on a multi-partite graph with bounded boxicity; developed efficient multi-level verification algorithm.
result Tight lower bounds on robustness of decision tree ensembles, hundreds of times faster than previous approach.
Unified method for local GBDT feature contributions.
problem Need for local model interpretation for GBDT.
method Unified computation mechanism for instance-level feature contributions.
result Unified computation mechanism for GBDT feature contributions.
Paper improves GBDT accuracy in federated learning.
problem Efficiency and accuracy issues in federated GBDT training.
method Uses locality-sensitive hashing for tree boosting, with relaxed privacy constraints.
result Significantly improves predictive accuracy compared to local data training.
LoBoost improves local conformal prediction for gradient-boosted trees without extra data splits.
problem Quantifying uncertainty in gradient-boosted tree predictions.
method Model-native local conformal prediction using leaf structure.
result Competitive interval quality and improved test MSE with large calibration speedups.
Gradient boosting can be seen as Gaussian process inference.
problem Improving uncertainty estimates in out-of-domain detection.
method Gradient boosting reformulated as a kernel method converging to Gaussian process inference.
result Gradient boosting can provide better uncertainty estimates through Monte-Carlo estimation of posterior variance.