Tree ensembles, such as random forests and boosted trees, are renowned for their high prediction performance. However, their interpretability is critically limited due to the enormous complexity. In this study, we present a method to make a complex tree ensemble interpretable by simplifying the model. Specifically, we …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,694 papers · 148 categories
Trend · papers per month
48 results for “Bayesian tree ensemble”
We propose a novel "tree-averaging" model that utilizes the ensemble of classification and regression trees (CART). Each constituent tree is estimated with a subset of similar data. We treat this grouping of subsets as Bayesian ensemble trees (BET) and model them as an infinite mixture Dirichlet process. We show that B…
GBEST model improves survival analysis for small datasets.
problem Challenges in survival analysis, especially with small data.
method Bayesian bootstrap and Beta Stacy bootstrap methods integrated into bagging tree models.
result GBEST model outperforms classical survival models in predictive performance and stability.
Bayesian tree ensemble model for estimating treatment effects in high-dimensional survival data.
problem Estimating heterogeneous treatment effects in censored survival data with many covariates.
method Developed a Bayesian tree ensemble model with a horseshoe prior for adaptive shrinkage.
result Accurately estimates treatment effects in high-dimensional covariate spaces and non-linear functions.
Ensemble of regression trees have become popular statistical tools for the estimation of conditional mean given a set of predictors. However, quantile regression trees and their ensembles have not yet garnered much attention despite the increasing popularity of the linear quantile regression model. This work proposes a…
Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spacesstat.ML
Tree ensemble kernels improve Bayesian optimization for mixed features and constraints.
problem Optimizing over mixed-feature spaces with known constraints.
method Kernel interpretation of tree ensembles as Gaussian Process prior, compatible optimization formulation for acquisition function, integration of known constraints.
result Framework outperforms state-of-the-art methods for mixed-feature spaces and constraints.
Oblique BART improves tree-based predictions.
problem Axis-aligned decision rules in BART can be suboptimal.
method Developed an oblique version of BART using data-adaptive hyperplane partitions.
result Oblique BART outperformed axis-aligned BART and other tree methods on benchmarks.
This paper develops a novel stochastic tree ensemble method for nonlinear regression, which we refer to as XBART, short for Accelerated Bayesian Additive Regression Trees. By combining regularization and stochastic search strategies from Bayesian modeling with computationally efficient techniques from recursive partiti…
GP-BART improves BART's predictive performance by incorporating Gaussian process priors.
problem Lack of smoothness and explicit covariance structure in BART.
method GP-BART extends BART with Gaussian process priors for tree predictions.
result GP-BART outperforms traditional models in various applications.
Tree ensembles like RF and GBT can be seen as kernels, improving regression and classification performance.
problem Improving kernel methods for tree ensemble based models.
method Investigation of RF and GBT kernels in simulation and real data.
result RF and GBT kernels are competitive to their respective ensembles in higher dimensions, particularly with noisy features.
REALITrees uses a Rashomon ensemble approach for active learning in sparse decision trees.
problem Active learning reduces labeling costs by selecting informative samples, but current methods often sacrifice model diversity and direct characterization of the hypothesis space.
method REALITrees constructs a committee of all near-optimal sparse decision tree models using a Rashomon Set and a Gibbs posterior to weight them by empirical risk.
result REALITrees outperforms randomized ensembles, especially in noisy environments, by leveraging expanded model multiplicity.
GPU-accelerated BART speeds up Bayesian regression.
problem Long running time of BART makes it impractical for large datasets.
method GPU-enabled implementation of BART.
result BART is now 200x faster on GPUs.
An Infinite BART modelstat.CO
Infinite BART model selects number of trees and allows different functions for clusters.
problem Regression and classification analysis with automatic tree selection and cluster-specific functions.
method Incorporates an Indian Buffet process prior to select a subset of decision trees for each observation.
result Infinite BART model outperforms classic BART on simulated and real datasets.
In this paper we present decomposable priors, a family of priors over structure and parameters of tree belief nets for which Bayesian learning with complete observations is tractable, in the sense that the posterior is also decomposable and can be completely determined analytically in polynomial time. This follows from…
Unified framework for selecting variables with uncertainty quantification.
problem Uncertainty in nonlinear variable selection for various models.
method Develops a unified framework using integrated partial derivatives for quantifying variable importance and uncertainty.
result The approach provides a principled method for quantifying variable selection uncertainty and is generalizable to non-differentiable models.
BART is extended to handle various response variables.
problem Modeling nonlinear regression functions for diverse response types.
method Generalized Bayesian Additive Regression Trees (GBART) for exponential family distributions.
result The posterior concentrates at a minimax rate for certain response distributions.
Bayesian Additive Regression Trees (BART) is a fully Bayesian approach to modeling with ensembles of trees. BART can uncover complex regression functions with high dimensional regressors in a fairly automatic way and provide Bayesian quantification of the uncertainty through the posterior. However, BART assumes IID nor…
This study shows how social insects and machine learning methods share a common mathematical framework.
problem Understanding how decentralized systems achieve optimal decision-making.
method Developed a rigorous mathematical framework to show isomorphism between ant colonies and ensemble machine learning.
result Demonstrated that ant colony decision-making and random forest learning implement identical variance reduction strategies through decorrelation of identical units.
Bayesian Additive Distribution Regression (DistBART) predicts distributions from grouped data.
problem Predicting distributions from grouped data with varying characteristics.
method Bayesian nonparametric approach using BART for modeling the regression function.
result Empirical and theoretical evidence supports DistBART's effectiveness in learning from low-dimensional marginals.
ForestPrune optimizes tree ensemble pruning for compactness and speed.
problem Large tree ensembles in predictive models consume excessive memory and reduce interpretability.
method Developed a specialized optimization algorithm to efficiently prune tree ensembles by depth layers.
result ForestPrune produces compact, high-performing models that outperform existing post-processing methods.
This work presents an approach to automatically induction for non-greedy decision trees constructed from neural network architecture. This construction can be used to transfer weights when growing or pruning a decision tree, allowing non-greedy decision tree algorithms to automatically learn and adapt to the ideal arch…
Bayesian CART models improve insurance claims frequency prediction and interpretation.
problem Improving accuracy and interpretability in insurance pricing models.
method Introducing Bayesian CART models for claims frequency, implementing MCMC algorithm for posterior tree exploration, and using DIC for model selection.
result Bayesian CART models can better classify policy-holders into risk groups.
Paper studies ensemble probabilistic regression trees for smooth approximations.
problem Smooth approximations of regression functions.
method Ensemble versions of probabilistic regression trees.
result Ensemble probabilistic regression trees are consistent and perform well.
Optimal survival trees ensemble reduces tree count and improves predictive performance.
problem Improving predictive performance in survival analysis.
method Grows a forest of optimal survival trees by ranking and selecting the best trees based on out-of-bag error.
result Reduces the number of trees in the ensemble while improving predictive performance.
We develop a Bayesian "sum-of-trees" model where each tree is constrained by a regularization prior to be a weak learner, and fitting and inference are accomplished via an iterative Bayesian backfitting MCMC algorithm that generates samples from a posterior. Effectively, BART is a nonparametric Bayesian regression appr…
Flexible tree ensemble learning framework supports arbitrary loss functions and multi-task learning.
problem Limited modeling capabilities of existing tree ensemble learning toolkits.
method Differentiable tree ensembles with tensor-based formulation for efficient training.
result Our framework leads to 100x more compact and 23% more expressive tree ensembles.
Tree ensembles, such as random forest and boosted trees, are renowned for their high prediction performance, whereas their interpretability is critically limited. In this paper, we propose a post processing method that improves the model interpretability of tree ensembles. After learning a complex tree ensembles in a s…
A framework detects nonlinear and interaction effects in epidemiological data with uncertainty quantification.
problem Lack of reliable inference for ML-discovered nonlinearities and interactions in epidemiological data.
method Combines Bayesian sparse regression, tree ensembles, and Shapley values.
result Valid uncertainty quantification for feature effects at the individual level.
Boosting meta-trees improve decision tree performance.
problem Overfitting in decision trees.
method Boosting approach to construct multiple meta-trees.
result Ensembles of meta-trees prevent overfitting.
Tree ensembles such as random forests and boosted trees are accurate but difficult to understand, debug and deploy. In this work, we provide the inTrees (interpretable trees) framework that extracts, measures, prunes and selects rules from a tree ensemble, and calculates frequent variable interactions. An rule-based le…
BFTS uses Bayesian Additive Regression Trees for improved personalized mobile health interventions.
problem Adapting to complex, non-linear user behaviors in personalized mobile health interventions.
method Bayesian Forest Thompson Sampling (BFTS) integrates Bayesian Additive Regression Trees (BART) into the exploration loop of contextual bandits.
result BFTS achieves state-of-the-art regret on tabular benchmarks and improves engagement rates by over 30% in a behavioral intervention study.
Proposes spBART for risk prediction using epigenetic signatures and covariates.
problem Complex high-dimensional epigenetic data and low-dimensional covariates for risk prediction.
method Semi-parametric Bayesian Additive Regression Trees (spBART) with cross-validation for variable selection.
result Achieves strong out-of-sample discrimination (AUC = 0.96) in held-out validation set.
This paper introduces TNTK to study infinite soft tree ensembles.
problem Understanding the behavior of infinite soft tree ensembles.
method Introduced Tree Neural Tangent Kernel (TNTK) to analyze infinite soft tree ensembles.
result Identified several non-trivial properties of infinite soft tree ensembles.
TREX explains tree ensembles by identifying key training examples.
problem Identifying which training examples most influence tree ensemble predictions.
method TREX builds a surrogate model using a kernel that captures tree ensemble structure, approximating the original model.
result TREX provides accurate and effective explanations for tree ensembles.
A new method creates simpler, more interpretable decision trees from complex ensembles.
problem Complex tree ensembles reduce interpretability and control over machine learning models.
method Dynamic-programming based algorithm for finding a minimum-size decision tree.
result Optimal born-again trees are simpler and more interpretable than original ensembles.
New framework uses tree ensembles for contextual bandits.
problem Optimizing decisions in dynamic environments with contextual information.
method Adapts Upper Confidence Bound and Thompson Sampling to tree ensemble methods.
result Tree ensemble methods outperform traditional methods in regret minimization and runtime.
HATT improves online decision tree ensembles by using a more eager splitting strategy.
problem Improving the efficiency of online decision tree ensembles.
method Replacing Hoeffding Tree's split strategy with HATT, which uses the Hoeffding Test for candidate splits.
result HATT outperforms Hoeffding Tree in online bagging and boosting ensembles, as shown by significant performance improvements in various testbenches.
Novel analysis improves weighted majority vote in multiclass classification.
problem Improving the performance of weighted majority vote in multiclass classification.
method Analyzes expected risk of weighted majority vote, considering prediction correlations and provides a bound for efficient minimization.
result Minimization of the new bound typically does not degrade the test error of the ensemble.
Ensembles of decision trees perform well on many problems, but are not interpretable. In contrast to existing approaches in interpretability that focus on explaining relationships between features and predictions, we propose an alternative approach to interpret tree ensemble classifiers by surfacing representative poin…
Shallow trees in ensemble models make models more interpretable and sometimes better.
problem Lack of transparency in high-performing tree ensemble models.
method Developed an interpretation algorithm to convert tree ensembles into functional ANOVA representations. Proposed strategies to enhance interpretability.
result Shallow trees in ensemble models can lead to better generalization performance and improved interpretability.
A Framework for an Assessment of the Kernel-target Alignment in Tree Ensemble Kernel Learningstat.ML
The study investigates kernel-target alignment in tree ensemble kernels.
problem The degree of kernel-target alignment affects the performance of tree ensemble kernels in kernel learning.
method Eigenanalysis of the kernel matrix and sensitivity analysis via landmark learning.
result Good performance of tree ensemble kernels is associated with strong kernel-target alignment.
Bayesian Additive Regression Networks use neural networks for regression tasks.
problem Regression tasks with small neural networks and ensemble learning.
method Bayesian Additive Regression Tree principles applied to small neural networks, Gibbs sampling for ensemble learning.
result BARN provides more consistent and often more accurate results than shallow neural networks, BART, and ordinary least squares.
Due to their accuracies, methods based on ensembles of regression trees are a popular approach for making predictions. Some common examples include Bayesian additive regression trees, boosting and random forests. This paper focuses on honest random forests, which add honesty to the original form of random forests and a…
Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothersstat.ML
Paper explains how tree ensembles improve predictions by smoothing and regulating smoothness.
problem Understanding why tree ensembles perform well despite their complexity.
method Interpreting tree ensembles as adaptive and self-regularizing smoothers.
result Ensemble trees make more smooth predictions than individual trees and adjust smoothness based on input dissimilarity.
Paper analyzes soft tree ensembles using NTK, finding only leaf count matters.
problem Understanding impact of various tree architectures in ensemble learning.
method Formulated and analyzed Neural Tangent Kernel (NTK) for soft tree ensembles.
result Only the number of leaves at each depth is relevant for tree architecture in ensemble learning.
On Theory for BARTstat.ML
Ensemble learning is a statistical paradigm built on the premise that many weak learners can perform exceptionally well when deployed collectively. The BART method of Chipman et al. (2010) is a prominent example of Bayesian ensemble learning, where each learner is a tree. Due to its impressive performance, BART has rec…
New approach verifies tree ensembles efficiently, making them more robust.
problem Verifying robustness of decision tree ensembles against evasion attacks is NP-hard.
method Identified large-spread ensembles, proposed verifiable learning, designed training algorithm.
result Large-spread ensembles can be verified in polynomial time and are more robust.
Using ensemble methods for regression has been a large success in obtaining high-accuracy prediction. Examples are Bagging, Random forest, Boosting, BART (Bayesian additive regression tree), and their variants. In this paper, we propose a new perspective named variable grouping to enhance the predictive performance. Th…