arXiv research

Boosted additive models reveal new insights and potential pathologies.

SHAP explains boosted trees with additively modeled features.

Boosting method for causal SEMs from observational data.

Boost-R uses gradient boosted trees for analyzing recurrence data.

A method interprets black-box models using an ensemble of gradient boosting machines.

We propose a statistical inference framework for the component-wise functional gradient descent algorithm (CFGD) under normality assumption for model errors, also known as $L_2$-Boosting. The CFGD is one of the most versatile tools to analyze data, because it scales well to high-dimensional data sets, allows for a very…

We present a new algorithm for boosting generalized additive models for location, scale and shape (GAMLSS) that allows to incorporate stability selection, an increasingly popular way to obtain stable sets of covariates while controlling the per-family error rate (PFER). The model is fitted repeatedly to subsampled data…

Additive models, such as produced by gradient boosting, and full interaction models, such as classification and regression trees (CART), are widely used algorithms that have been investigated largely in isolation. We show that these models exist along a spectrum, revealing never-before-known connections between these t…

In this paper, we derive a novel probabilistic model of boosting as a Product of Experts. We re-derive the boosting algorithm as a greedy incremental model selection procedure which ensures that addition of new experts to the ensemble does not decrease the likelihood of the data. These learning rules lead to a generic …

New methods improve prediction performance and reduce computation time in boosting and random forest models.

FGBoost boosts gradient boosting for complex data.

Combines boosting with Gaussian process and mixed effects models.

We present Natural Gradient Boosting (NGBoost), an algorithm for generic probabilistic prediction via gradient boosting. Typical regression models return a point estimate, conditional on covariates, but probabilistic regression models output a full probability distribution over the outcome space, conditional on the cov…

Algorithm infers sampling distribution from i.i.d. samples without supervision.

Gradient Boosted Normalizing Flows improve flexibility of NFs without increasing complexity.

Gradient Boosted Mixed Models estimate mean and variance components for clustered data.

Boosting strategies for merging vs. ensembling studies analyzed.

We propose a black-box variational inference method to approximate intractable distributions with an increasingly rich approximating class. Our method, termed variational boosting, iteratively refines an existing variational approximation by solving a sequence of optimization problems, allowing the practitioner to trad…

Boosting algorithms are frequently used in applied data science and in research. To date, the distinction between boosting with either gradient descent or second-order Newton updates is often not made in both applied and methodological research, and it is thus implicitly assumed that the difference is irrelevant. The g…

Generalized additive models (GAMs) are favored in many regression and binary classification problems because they are able to fit complex, nonlinear functions while still remaining interpretable. In the first part of this paper, we generalize a state-of-the-art GAM learning algorithm based on boosted trees to the multi…

Boosting improves accuracy with fewer calls to weak learners for certain concept classes.

We show that Residual Networks (ResNet) is equivalent to boosting feature representation, without any modification to the underlying ResNet training algorithm. A regret bound based on Online Gradient Boosting theory is proved and suggests that ResNet could achieve Online Gradient Boosting regret bounds through neural n…

A new gradient boosting method improves interpretability of probabilistic models.

EBM improves car insurance claim severity and frequency prediction while maintaining interpretability.

Stagewise boosting improves gradient boosting for distributional regression.

Boosting is a popular way to derive powerful learners from simpler hypothesis classes. Following previous work (Mason et al., 1999; Friedman, 2000) on general boosting frameworks, we analyze gradient-based descent algorithms for boosting with respect to any convex objective and introduce a new measure of weak learner p…

Boosting algorithms improve estimation of personalized treatment rules.

FAST optimizes additive segmentation for faster, more interpretable models.

This research adapts scoring rules for training survival models, improving predictive performance.

In this tutorial paper, we first define mean squared error, variance, covariance, and bias of both random variables and classification/predictor models. Then, we formulate the true and generalization errors of the model for both training and validation/test instances where we make use of the Stein's Unbiased Risk Estim…

Gradient boosting of regression trees is a competitive procedure for learning predictive models of continuous data that fits the data with an additive non-parametric model. The classic version of gradient boosting assumes that the data is independent and identically distributed. However, relational data with interdepen…

The paper provides convergence guarantees for multicalibration gradient boosting.

ProtoNAM models tabular data with neural networks, making predictions transparent.

Paper presents robust boosting methods for label noise.

Paper proposes ensemble methods to prevent forgetting in neural networks.

Boosting with unlabeled data achieves optimal sample complexity in agnostic settings.

LoBoost improves local conformal prediction for gradient-boosted trees without extra data splits.

Random forest and deep neural network are two schools of effective classification methods in machine learning. While the random forest is robust irrespective of the data domain, the deep neural network has advantages in handling high dimensional data. In view that a differentiable neural decision forest can be added to…

A robust model for time series forecasting is highly important in many domains, including but not limited to financial forecast, air temperature and electricity consumption. To improve forecasting performance, traditional approaches usually require additional feature sets. However, adding more feature sets from differe…

ParamBoost uses gradient boosting to create interpretable non-linear models with constraints.

sGBM speeds up gradient boosting by parallelizing and adapting base learners.

XGBoost is a scalable ensemble technique based on gradient boosting that has demonstrated to be a reliable and efficient machine learning challenge solver. This work proposes a practical analysis of how this novel technique works in terms of training speed, generalization performance and parameter setup. In addition, a…

Enhanced Tweedie model for insurance claims using CatBoost.

Credit scoring plays a vital role in the field of consumer finance. Survival analysis provides an advanced solution to the credit-scoring problem by quantifying the probability of survival time. In order to deal with highly heterogeneous industrial data collected in Chinese market of consumer finance, we propose a nonp…

The paper forecasts joint electricity demand across 14 British regions using additive models.

Paper proposes a boosting method with fast learning rates and early stopping.

Boosted CVaR Classification improves tail performance in classification tasks.

pGMM kernel outperforms ordinary ridge regression and RBF kernel ridge regression without tuning.