TREX explains tree ensembles by identifying key training examples.
problem Identifying which training examples most influence tree ensemble predictions.
method TREX builds a surrogate model using a kernel that captures tree ensemble structure, approximating the original model.
result TREX provides accurate and effective explanations for tree ensembles.
Proposes a deep tree-ensemble model for multi-output prediction.
problem Lack of efficient solutions for multi-output prediction.
method Integrates tree-embeddings into deep tree-ensembles for structured output prediction.
result Superior performance in multi-label classification and multi-target regression tasks.
ForestPrune optimizes tree ensemble pruning for compactness and speed.
problem Large tree ensembles in predictive models consume excessive memory and reduce interpretability.
method Developed a specialized optimization algorithm to efficiently prune tree ensembles by depth layers.
result ForestPrune produces compact, high-performing models that outperform existing post-processing methods.
Tree ensembles, such as random forests and boosted trees, are renowned for their high prediction performance. However, their interpretability is critically limited due to the enormous complexity. In this study, we present a method to make a complex tree ensemble interpretable by simplifying the model. Specifically, we …
Flexible tree ensemble learning framework supports arbitrary loss functions and multi-task learning.
problem Limited modeling capabilities of existing tree ensemble learning toolkits.
method Differentiable tree ensembles with tensor-based formulation for efficient training.
result Our framework leads to 100x more compact and 23% more expressive tree ensembles.
Tree ensembles, such as random forest and boosted trees, are renowned for their high prediction performance, whereas their interpretability is critically limited. In this paper, we propose a post processing method that improves the model interpretability of tree ensembles. After learning a complex tree ensembles in a s…
New framework uses tree ensembles for contextual bandits.
problem Optimizing decisions in dynamic environments with contextual information.
method Adapts Upper Confidence Bound and Thompson Sampling to tree ensemble methods.
result Tree ensemble methods outperform traditional methods in regret minimization and runtime.
Tree ensembles such as random forests and boosted trees are accurate but difficult to understand, debug and deploy. In this work, we provide the inTrees (interpretable trees) framework that extracts, measures, prunes and selects rules from a tree ensemble, and calculates frequent variable interactions. An rule-based le…
The study investigates kernel-target alignment in tree ensemble kernels.
problem The degree of kernel-target alignment affects the performance of tree ensemble kernels in kernel learning.
method Eigenanalysis of the kernel matrix and sensitivity analysis via landmark learning.
result Good performance of tree ensemble kernels is associated with strong kernel-target alignment.
Integrates differentiable decision trees into neural networks for faster training and inference.
problem Combining differentiability and conditional computation in tree ensembles for neural networks.
method Sparse activation function and specialized forward/backward propagation algorithms for efficient training and inference.
result 10x speed-ups and 20x reduction in parameters compared to existing methods, while maintaining performance.
New method tests tree models without causing computational pressure.
problem Incompatibility of adversarial robustness testing with tree ensembles.
method Smooths tree ensembles with sigmoid functions and uses gradient descent.
result Successfully reveals adversarial vulnerability of tree ensemble models.
Paper analyzes soft tree ensembles using NTK, finding only leaf count matters.
problem Understanding impact of various tree architectures in ensemble learning.
method Formulated and analyzed Neural Tangent Kernel (NTK) for soft tree ensembles.
result Only the number of leaves at each depth is relevant for tree architecture in ensemble learning.
Tree ensembles like RF and GBT can be seen as kernels, improving regression and classification performance.
problem Improving kernel methods for tree ensemble based models.
method Investigation of RF and GBT kernels in simulation and real data.
result RF and GBT kernels are competitive to their respective ensembles in higher dimensions, particularly with noisy features.
Ensembles of decision trees perform well on many problems, but are not interpretable. In contrast to existing approaches in interpretability that focus on explaining relationships between features and predictions, we propose an alternative approach to interpret tree ensemble classifiers by surfacing representative poin…
Tree ensemble kernels improve Bayesian optimization for mixed features and constraints.
problem Optimizing over mixed-feature spaces with known constraints.
method Kernel interpretation of tree ensembles as Gaussian Process prior, compatible optimization formulation for acquisition function, integration of known constraints.
result Framework outperforms state-of-the-art methods for mixed-feature spaces and constraints.
A single slow-growing tree matches Random Forest's performance.
problem Matching Random Forest's performance with a single tree.
method SGT uses a learning rate to tame CART's greedy algorithm, improving on greedy ML algorithms.
result SGT and tree ensembles like Booging, BT, and RF improve performance.
Paper explains how tree ensembles improve predictions by smoothing and regulating smoothness.
problem Understanding why tree ensembles perform well despite their complexity.
method Interpreting tree ensembles as adaptive and self-regularizing smoothers.
result Ensemble trees make more smooth predictions than individual trees and adjust smoothness based on input dissimilarity.
Variable selection for high-dimensional linear models has received a lot of attention lately, mostly in the context of l1-regularization. Part of the attraction is the variable selection effect: parsimonious models are obtained, which are very suitable for interpretation. In terms of predictive power, however, these re…
A new method creates simpler, more interpretable decision trees from complex ensembles.
problem Complex tree ensembles reduce interpretability and control over machine learning models.
method Dynamic-programming based algorithm for finding a minimum-size decision tree.
result Optimal born-again trees are simpler and more interpretable than original ensembles.
TreeHFD algorithm explains tree ensemble models through hierarchical orthogonality.
problem Difficulty in explaining black-box tree ensemble models.
method TreeHFD algorithm using hierarchical orthogonality constraints.
result TreeHFD estimates Hoeffding decomposition from data samples.
New methods improve tree ensemble models by compressing them while maintaining accuracy.
problem Theoretical understanding and practical compression of tree ensembles like random forests and gradient boosting machines.
method Spectral perspective on tree ensembles, deriving minimax rates and developing compression schemes.
result Leading eigenfunctions/singular vectors capture dominant predictive directions, leading to smaller, competitive models.
This study extends verifiable learning to boosted tree ensembles, enabling efficient security verification.
problem Efficiently verifying the robustness of boosted tree ensembles against norm-based attackers.
method Formal verification of robustness for large-spread boosted tree ensembles, considering L∞-norm and pseudo-polynomial time for Lp-norm verification. result Polynomial time verification for L∞-norm attackers, NP-hard for other norms, and pseudo-polynomial time for Lp-norm verification. Optimizes feature shifts for tree ensemble reclassification.
problem Improving tree ensemble reclassification accuracy through feature perturbation.
method Mathematical optimization of feature shifts to maximize reclassification probability.
result Effective feature ranking and optimization for tree ensemble reclassification.
New approach verifies tree ensembles efficiently, making them more robust.
problem Verifying robustness of decision tree ensembles against evasion attacks is NP-hard.
method Identified large-spread ensembles, proposed verifiable learning, designed training algorithm.
result Large-spread ensembles can be verified in polynomial time and are more robust.
We study the robustness verification problem for tree-based models, including decision trees, random forests (RFs) and gradient boosted decision trees (GBDTs). Formal robustness verification of decision tree ensembles involves finding the exact minimal adversarial perturbation or a guaranteed lower bound of it. Existin…
Bayesian tree ensemble model for estimating treatment effects in high-dimensional survival data.
problem Estimating heterogeneous treatment effects in censored survival data with many covariates.
method Developed a Bayesian tree ensemble model with a horseshoe prior for adaptive shrinkage.
result Accurately estimates treatment effects in high-dimensional covariate spaces and non-linear functions.
ControlBurn selects few features from tree ensembles for better model interpretability.
problem Improving model interpretability in machine learning models.
method Sparse tree ensembles with lasso optimization.
result ControlBurn selects feature-sparse subsets for better model interpretability.
We propose a method to extract interpretable rules from tree ensembles.
problem Tree ensembles are accurate but hard to interpret.
method Propose an estimator to extract compact sets of decision rules from tree ensembles.
result Our estimator improves accuracy and reveals useful relationships in the data.
Tree ensembles are flexible predictive models that can capture relevant variables and to some extent their interactions in a compact and interpretable manner. Most algorithms for obtaining tree ensembles are based on versions of boosting or Random Forest. Previous work showed that boosting algorithms exhibit a cyclic b…
Shallow trees in ensemble models make models more interpretable and sometimes better.
problem Lack of transparency in high-performing tree ensemble models.
method Developed an interpretation algorithm to convert tree ensembles into functional ANOVA representations. Proposed strategies to enhance interpretability.
result Shallow trees in ensemble models can lead to better generalization performance and improved interpretability.
Tree ensemble method tackles multi-objective constrained optimization in energy systems.
problem Complex, multi-objective, and constrained optimization problems in energy systems.
method Data-driven tree ensemble approach for black-box problems with heterogeneous variable spaces.
result Competitive performance and sampling efficiency compared to state-of-the-art tools.
LionForests interprets random forests for better understanding of predictions.
problem Interpreting black-box tree ensemble models like random forests.
method Combining unsupervised learning and a similarity metric to explain tree ensembles.
result LionForests provides transparent rules for interpreting random forest predictions.
Tree ensembles such as Random Forests have achieved impressive empirical success across a wide variety of applications. To understand how these models make predictions, people routinely turn to feature importance measures calculated from tree ensembles. It has long been known that Mean Decrease Impurity (MDI), one of t…
FIRE extracts interpretable rules from tree ensembles.
problem Building sparse and interpretable rule sets from tree ensembles.
method Optimization-based framework with fusion regularization and sparsity-inducing penalties.
result FIRE outperforms state-of-the-art rule ensemble algorithms.
New framework detects model weaknesses in decision tree ensembles.
problem Detecting feature sensitivity in decision tree ensembles.
method Data-aware sensitivity framework using MILP and SMT.
result Realistic and interpretable examples of model weaknesses.
Recent advances in machine learning and artificial intelligence are now being considered in safety-critical autonomous systems where software defects may cause severe harm to humans and the environment. Design organizations in these domains are currently unable to provide convincing arguments that their systems are saf…
Classifier evasion consists in finding for a given instance x the nearest instance x′ such that the classifier predictions of x and x′ are different. We present two novel algorithms for systematically computing evasions for tree ensembles such as boosted trees and random forests. Our first algorithm uses a Mixe…
Auto-encoding is an important task which is typically realized by deep neural networks (DNNs) such as convolutional neural networks (CNN). In this paper, we propose EncoderForest (abbrv. eForest), the first tree ensemble based auto-encoder. We present a procedure for enabling forests to do backward reconstruction by ut…
Develops RKHS framework for analyzing tree ensembles.
problem Analyzing the theoretical properties of tree ensembles.
method Reproducing Kernel Hilbert Spaces (RKHS) for tree ensembles.
result Characterizes Random Forest predictor as minimizer of a penalized empirical risk functional in RKHS.
Current deep learning models are mostly build upon neural networks, i.e., multiple layers of parameterized differentiable nonlinear modules that can be trained by backpropagation. In this paper, we explore the possibility of building deep models based on non-differentiable modules. We conjecture that the mystery behind…
Tree ensemble models such as random forests and boosted trees are among the most widely used and practically successful predictive models in applied machine learning and business analytics. Although such models have been used to make predictions based on exogenous, uncontrollable independent variables, they are increas…
This paper introduces TNTK to study infinite soft tree ensembles.
problem Understanding the behavior of infinite soft tree ensembles.
method Introduced Tree Neural Tangent Kernel (TNTK) to analyze infinite soft tree ensembles.
result Identified several non-trivial properties of infinite soft tree ensembles.
We present the Network-based Biased Tree Ensembles (NetBiTE) method for drug sensitivity prediction and drug sensitivity biomarker identification in cancer using a combination of prior knowledge and gene expression data. Our devised method consists of a biased tree ensemble that is built according to a probabilistic bi…
New test improves tree ensemble pruning for better model performance.
problem Lack of robust theoretical justification for penalty terms in tree ensembles.
method Developed a novel hypothesis test for tree ensemble split quality.
result Significant reduction in out-of-sample loss using the new test.
Unified framework for selecting variables with uncertainty quantification.
problem Uncertainty in nonlinear variable selection for various models.
method Develops a unified framework using integrated partial derivatives for quantifying variable importance and uncertainty.
result The approach provides a principled method for quantifying variable selection uncertainty and is generalizable to non-differentiable models.
We introduce canonical correlation forests (CCFs), a new decision tree ensemble method for classification and regression. Individual canonical correlation trees are binary decision trees with hyperplane splits based on local canonical correlation coefficients calculated during training. Unlike axis-aligned alternatives…
Despite its success and popularity, machine learning is now recognized as vulnerable to evasion attacks, i.e., carefully crafted perturbations of test inputs designed to force prediction errors. In this paper we focus on evasion attacks against decision tree ensembles, which are among the most successful predictive mod…
Paper introduces XBART for nonlinear regression, outperforming XGBoost.
problem Nonlinear regression problems, especially in speed and accuracy.
method Combines Bayesian modeling and recursive partitioning for efficient, accurate predictions.
result XBART provides faster and more accurate predictions than XGBoost.