We present Natural Gradient Boosting (NGBoost), an algorithm for generic probabilistic prediction via gradient boosting. Typical regression models return a point estimate, conditional on covariates, but probabilistic regression models output a full probability distribution over the outcome space, conditional on the cov…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Natural gradient boosting”
Boost-R uses gradient boosted trees for analyzing recurrence data.
problem Analyzing recurrence data with static and dynamic features.
method Gradient boosted additive trees with time-dependent functions.
result Estimates the cumulative intensity function of recurrent event processes.
A functional central limit theorem for kernel gradient flow and infinitesimal gradient boostingmath.PR
Functional central limit theorem for kernel gradient flow and infinitesimal gradient boosting
problem Fluctuations of boosting processes around their deterministic limit
method Stochastic perturbation analysis of ODEs in Banach spaces
result Rescaled deviations converge to a Gaussian process
Gradient boosting adapted for vector inputs.
problem Applying gradient boosting to multi-class classification problems.
method Extended gradient boosting framework to vector inputs using histogram-based decision trees.
result Efficient algorithm for vector-valued objectives.
A scalable model estimates revenue uncertainty for SMEs.
problem Estimating revenue uncertainty for SMEs to manage credit limits.
method Scalable Natural Gradient Boosting Machines.
result The method distinguishes accurate from inaccurate revenue forecasts.
RoNGBa: A Robustly Optimized Natural Gradient Boosting Training Approach with Leaf Number Clippingcs.LG
Natural gradient has been recently introduced to the field of boosting to enable the generic probabilistic predication capability. Natural gradient boosting shows promising performance improvements on small datasets due to better training dynamics, but it suffers from slow training speed overhead especially for large d…
GBMixed boosts mixed models for clustered data, estimating mean and variance flexibly.
problem Flexible estimation of mean and variance components in clustered data.
method Gradient Boosting framework for linear mixed models with likelihood-based gradients.
result GBMixed accurately recovers complex nonlinear fixed effects and covariances.
Stagewise boosting improves gradient boosting for distributional regression.
problem Vanishing gradient in gradient boosting for distributional regression leads to suboptimal models.
method Proposes a stagewise boosting-type algorithm for distributional regression, combining stagewise regression ideas with gradient boosting and incorporating a novel regularization method, correlation filtering.
result The proposed algorithm provides better results, especially for complex distributions, by reducing the risk of being trapped in a local optimum.
Gradient boosting for spatial regression models improves prediction accuracy.
problem Spatial data with autoregressive disturbances.
method Model-based gradient boosting algorithm for spatial regression models.
result Improves prediction accuracy on out-of-sample spatial data.
Develop gradient boosting for estimating covariate-dependent GP distributions in insurance.
problem Estimating covariate-dependent Generalized Pareto distributions in insurance.
method Developing a statistical learning theory for gradient boosting.
result Deriving non-asymptotic error bounds for the boosting estimator.
NGBoost boosts multivariate probabilistic regression.
problem Joint probabilistic regression for multivariate targets.
method Natural Gradient Boosting for nonparametric modeling.
result Competitive performance in oceanographic velocity prediction.
A feature selection algorithm should ideally satisfy four conditions: reliably extract relevant features; be able to identify non-linear feature interactions; scale linearly with the number of features and dimensions; allow the incorporation of known sparsity structure. In this work we propose a novel feature selection…
Gradient boosting algorithm for spatial panel models improves estimation in high-dimensional settings.
problem Estimation failure in high-dimensional spatial panel models.
method Model-based gradient boosting algorithm for spatial panel models with random and fixed effects.
result Feasibility and interpretability in both low- and high-dimensional settings.
As an adaptive, interpretable, robust, and accurate meta-algorithm for arbitrary differentiable loss functions, gradient tree boosting is one of the most popular machine learning techniques, though the computational expensiveness severely limits its usage. Stochastic gradient boosting could be adopted to accelerates gr…
This research tackles uncertainty in gradient boosting models using ensemble methods.
problem Quantifying uncertainty in gradient boosting models for high-risk applications.
method Probabilistic ensemble-based framework for gradient boosting classification and regression models.
result Ensembles of gradient boosting models detect anomalous inputs but have limited ability to improve total uncertainty.
Gradient boosting with randomized trees reduces discontinuities and complexity.
problem Discontinuities in regression functions due to sparse training data.
method Gradient boosting machine with partially randomized decision trees.
result Improves robustness and computational efficiency of gradient boosting.
A method to improve gradient boosting models using stacking.
problem Improving the performance of gradient boosting models.
method Proposes a stacking algorithm to learn a meta-model for ensembles of gradient boosting models.
result The proposed approach can be extended to differentiable combination models like neural networks.
Wasserstein gradient boosting predicts probability distributions for supervised learning.
problem Distribution-valued supervised learning where outputs are probability distributions.
method Fits a new weak learner to Wasserstein gradients of loss functionals of probability distributions.
result Superior performance in probabilistic prediction compared to existing methods.
In this survey, we discuss several different types of gradient boosting algorithms and illustrate their mathematical frameworks in detail: 1. introduction of gradient boosting leads to 2. objective function optimization, 3. loss function estimations, and 4. model constructions. 5. application of boosting in ranking.
Infinitesimal gradient boosting is a new algorithm derived from gradient boosting.
problem Improving the efficiency and smoothness of gradient boosting.
method Introduced a new class of randomized regression trees and used a limit process in vanishing-learning-rate asymptotic.
result Convergence of the stochastic algorithm and characterization of the limiting procedure as a unique solution of a nonlinear ODE.
Boosting algorithms are frequently used in applied data science and in research. To date, the distinction between boosting with either gradient descent or second-order Newton updates is often not made in both applied and methodological research, and it is thus implicitly assumed that the difference is irrelevant. The g…
Unified framework for statistical inference in gradient boosting regression.
problem Challenges in statistical inference and uncertainty quantification for gradient boosting.
method Integrates dropout or parallel training with regularization for CLT in boosting.
result Increasing dropout rate and parallel trees enhances signal recovery and performance.
GrowNet uses shallow neural networks for gradient boosting, outperforming existing methods.
problem Improving gradient boosting performance through shallow neural networks.
method Unified gradient boosting framework with shallow neural networks as weak learners, incorporating corrective steps.
result GrowNet outperformed state-of-the-art boosting methods in classification, regression, and learning to rank tasks.
This paper introduces Stochastic Gradient Langevin Boosting (SGLB) - a powerful and efficient machine learning framework that may deal with a wide range of loss functions and has provable generalization guarantees. The method is based on a special form of the Langevin diffusion equation specifically designed for gradie…
New algorithm improves convergence of gradient boosting trees.
problem Global convergence of Newton boosting in tabular machine learning.
method Introduces Gradient Regularized Newton Descent for GBDTs, proving linear convergence for smooth, strongly convex losses and rate for general convex losses.
result Achieves globally convergent second-order GBDT algorithm with rate matching first-order boosting.
In this paper we present CatBoost, a new open-sourced gradient boosting library that successfully handles categorical features and outperforms existing publicly available implementations of gradient boosting in terms of quality on a set of popular publicly available datasets. The library has a GPU implementation of lea…
Gradient tree boosting is a prediction algorithm that sequentially produces a model in the form of linear combinations of decision trees, by solving an infinite-dimensional optimization problem. We combine gradient boosting and Nesterov's accelerated descent to design a new algorithm, which we call AGB (for Accelerated…
MFAI uses gradient boosted trees to leverage auxiliary info for scalable Bayesian matrix factorization.
problem Matrix factorization struggles with poor data quality, especially high sparsity and low SNR.
method Integrates gradient boosted trees into probabilistic matrix factorization framework.
result MFAI effectively leverages auxiliary information, improving model performance.
Gradient boosting is a prediction method that iteratively combines weak learners to produce a complex and accurate model. From an optimization point of view, the learning procedure of gradient boosting mimics a gradient descent on a functional variable. This paper proposes to build upon the proximal point algorithm, wh…
Gradient boosting can be seen as Gaussian process inference.
problem Improving uncertainty estimates in out-of-domain detection.
method Gradient boosting reformulated as a kernel method converging to Gaussian process inference.
result Gradient boosting can provide better uncertainty estimates through Monte-Carlo estimation of posterior variance.
Fréchet Geodesic Boostingstat.ML
FGBoost boosts gradient boosting for complex data.
problem Gradient boosting struggles with non-Euclidean data.
method Introduces FGBoost for geodesic metric spaces.
result FGBoost performs well on complex data.
Gradient boosting method enforced with individual fairness.
problem Enforcing fairness in machine learning models.
method Functional gradient descent on robust loss function.
result Algorithm converges globally and generalizes.
Lassoed Tree Boostingstat.ML
Gradient boosting improved with lassoed trees achieves faster convergence.
problem Improving gradient boosting convergence in large nonparametric spaces.
method Lassoed gradient boosted trees with early stopping.
result Achieves faster than L2 convergence rate.
sGBM speeds up gradient boosting by parallelizing and adapting base learners.
problem Infeasibility of parallelizing GBM training and sub-optimal performance in online settings.
method Integrates multiple differentiable base learners, jointly optimizing them with linear speed-up.
result sGBM achieves higher time efficiency and better accuracy than traditional GBM.
Boosted Frank-Wolfe accelerates optimization for nonconvex problems.
problem Optimizing nonconvex and quasar-convex objectives efficiently.
method Developed a novel step size strategy for stochastic Frank-Wolfe, extending it to various gradient estimators.
result Boosted Frank-Wolfe achieves faster convergence rates than non-boosted Frank-Wolfe.
The study develops a machine learning system for credit scoring and default prediction.
problem Developing a robust credit rating and default prediction system.
method Combines NLP, AE, GBM, DE, and SHAP/LIME for model interpretability.
result Obtained excellent out-of-sample performance in credit rating and default prediction.
New GPU algorithm boosts machine learning with larger datasets.
problem Limited GPU memory restricts training data size.
method Out-of-core GPU gradient boosting algorithm.
result Training larger datasets on GPUs without accuracy loss.
TF Boosted Trees (TFBT) is a new open-sourced frame-work for the distributed training of gradient boosted trees. It is based on TensorFlow, and its distinguishing features include a novel architecture, automatic loss differentiation, layer-by-layer boosting that results in smaller ensembles and faster prediction, princ…
Unified probabilistic gradient boosting for entire conditional distribution modeling.
problem Creating accurate probabilistic forecasts from regression tasks.
method Unified probabilistic gradient boosting framework using XGBoost and LightGBM, modeling conditional moments or CDF via Normalizing Flows.
result Achieves state-of-the-art forecast accuracy.
Calibrated Boosting-Foreststat.ML
Excellent ranking power along with well calibrated probability estimates are needed in many classification tasks. In this paper, we introduce a technique, Calibrated Boosting-Forest that captures both. This novel technique is an ensemble of gradient boosting machines that can support both continuous and binary labels. …
The paper provides convergence guarantees for multicalibration gradient boosting.
problem Understanding the convergence properties of multicalibration gradient boosting.
method Computational guarantees for multicalibration gradient boosting algorithms, including adaptive variants.
result The magnitude of successive prediction updates decays at , leading to convergence in empirical multicalibration error.
ABHT boosts regression by filtering regions with different smoothness.
problem Improving regression performance through local adaptivity.
method Gradient boosting with adaptive histogram transform.
result ABHT converges faster than PEHT in Hölder continuous spaces.
StructureBoost improves gradient boosting for complex categorical variables efficiently.
problem Efficiently handling complex categorical variables with known structure.
method Two methods to overcome computational obstacles in SCDT enumeration for structured categorical variables.
result StructureBoost outperforms existing packages on complex categorical problems.
Infinitesimal boosting converges to a deterministic process in large sample limit.
problem Characterizing the asymptotic behavior of infinitesimal gradient boosting in large sample sizes.
method Proving convergence to a deterministic process using large sample theory and differential equations.
result The test error decreases over time in the population limit.
Tree-Structured Boosting: Connections Between Gradient Boosted Stumps and Full Decision Treesstat.ML
Additive models, such as produced by gradient boosting, and full interaction models, such as classification and regression trees (CART), are widely used algorithms that have been investigated largely in isolation. We show that these models exist along a spectrum, revealing never-before-known connections between these t…
Gradient Boosted Mixed Models estimate mean and variance components for clustered data.
problem Limited flexibility in linear mixed models for complex settings.
method Gradient Boosting extended to mixed models with likelihood-based gradients and flexible base learners.
result Accurate recovery of variance components and improved predictive accuracy.
Unified comparison of gradient boosting algorithms for insurance claims.
problem Improving predictive accuracy and computational efficiency in insurance claim prediction.
method Unified notation and comprehensive numerical study comparing 12 gradient boosting algorithms on 5 datasets.
result No trade-off between model adequacy and predictive accuracy.
dboost optimizes prediction models for convex cone problems.
problem Optimizing prediction models for decision-making.
method Gradient boosting with implicit differentiation for convex quadratic cone programming.
result dboost reduces out-of-sample decision regret.