arXiv research

Alpha-trimming prunes trees in random forests to improve predictive performance.

ForestPrune optimizes tree ensemble pruning for compactness and speed.

We consider the problem of learning decision rules for prediction with feature budget constraint. In particular, we are interested in pruning an ensemble of decision trees to reduce expected feature cost while maintaining high prediction accuracy for any test example. We propose a novel 0-1 integer program formulation …

New test improves tree ensemble pruning for better model performance.

New method prunes classification trees for biased data.

This paper improves forest pruning to balance accuracy and interpretability.

This study argues for pruning trees in random forests to improve performance in low signal-to-noise scenarios.

The study analyzes decision trees on real and categorical features, deriving bounds on their VC dimension and proposing improved pruning algorithms.

New pruning method for sparse additive models speeds up causal structure learning.

A new method creates simpler, more interpretable decision trees from complex ensembles.

For a density $f$ on ${\mathbb R}^d$, a {\it high-density cluster} is any connected component of $\{x: f(x) \geq λ\}$, for some $λ> 0$. The set of all high-density clusters forms a hierarchy called the {\it cluster tree} of $f$. We present two procedures for estimating the cluster tree given samples from $f$. The first…

Random Forests automatically prune a latent 'true' tree, explaining their overfitting without tuning.

Covariance-Driven Regression Trees reduce overfitting in CART.

Paper addresses limitations of traditional hierarchical clustering methods.

Random forests perform bootstrap-aggregation by sampling the training samples with replacement. This enables the evaluation of out-of-bag error which serves as a internal cross-validation mechanism. Our motivation lies in using the unsampled training samples to improve each decision tree in the ensemble. We study the e…

PILOT is a fast algorithm for linear model trees that outperforms existing methods.

Bayesian Decision Trees are known for their probabilistic interpretability. However, their construction can sometimes be costly. In this article we present a general Bayesian Decision Tree algorithm applicable to both regression and classification problems. The algorithm does not apply Markov Chain Monte Carlo and does…

Tree ensembles such as random forests and boosted trees are accurate but difficult to understand, debug and deploy. In this work, we provide the inTrees (interpretable trees) framework that extracts, measures, prunes and selects rules from a tree ensemble, and calculates frequent variable interactions. An rule-based le…

Predict accuracy of neural architectures using non-neural models.

Nearest neighbor (k-NN) graphs are widely used in machine learning and data mining applications, and our aim is to better understand what they reveal about the cluster structure of the unknown underlying distribution of points. Moreover, is it possible to identify spurious structures that might arise due to sampling va…

The paper studies statistical properties of CART regression trees.

This work presents an approach to automatically induction for non-greedy decision trees constructed from neural network architecture. This construction can be used to transfer weights when growing or pruning a decision tree, allowing non-greedy decision tree algorithms to automatically learn and adapt to the ideal arch…

Deep neural networks have been proven powerful at processing perceptual data, such as images and audio. However for tabular data, tree-based models are more popular. A nice property of tree-based models is their natural interpretability. In this work, we present Deep Neural Decision Trees (DNDT) -- tree models realised…

Random Forests (RF) is one of the algorithms of choice in many supervised learning applications, be it classification or regression. The appeal of such tree-ensemble methods comes from a combination of several characteristics: a remarkable accuracy in a variety of tasks, a small number of parameters to tune, robustness…

A new pruning method finds sparse minimizers in flat regions of deep neural networks.

We consider the classical sparse regression problem of recovering a sparse signal $x_0$ given a measurement vector $y = Φx_0+w$. We propose a tree search algorithm driven by the deep neural network for sparse regression (TSN). TSN improves the signal reconstruction performance of the deep neural network designed for sp…

Automatically jailbreaks LLMs with black-box access.

A cluster tree provides a highly-interpretable summary of a density function by representing the hierarchy of its high-density clusters. It is estimated using the empirical tree, which is the cluster tree constructed from a density estimator. This paper addresses the basic question of quantifying our uncertainty by ass…

Proves a conjecture about 3-manifold invariants using trees and asymptotic formulas.

To deal with various datasets over different complexity, this paper presents an self-adaptive learning model that combines the proposed Dynamic Connected Neural Decision Networks (DNDN) and a new pruning method--Dynamic Soft Pruning (DSP). DNDN is a combination of random forests and deep neural networks that enjoys bot…

New method learns binary decision trees efficiently.

Shallow trees in ensemble models make models more interpretable and sometimes better.

In this paper we study predictive pattern mining problems where the goal is to construct a predictive model based on a subset of predictive patterns in the database. Our main contribution is to introduce a novel method called safe pattern pruning (SPP) for a class of predictive pattern mining problems. The SPP method a…

Boosted decision trees enjoy popularity in a variety of applications; however, for large-scale datasets, the cost of training a decision tree in each round can be prohibitively expensive. Inspired by ideas from the multi-arm bandit literature, we develop a highly efficient algorithm for computing exact greedy-optimal d…

Develops regression trees for estimating cumulative incidence curves in competing risks.

Enhanced survival trees improve computational efficiency and inference.

New techniques optimize decision trees for interpretable machine learning.

Risk bounds for Classification and Regression Trees (CART, Breiman et. al. 1984) classifiers are obtained under a margin condition in the binary supervised classification framework. These risk bounds are obtained conditionally on the construction of the maximal deep binary tree and permit to prove that the linear penal…

New methods improve tree ensemble models by compressing them while maintaining accuracy.

Bayesian optimization algorithm reduces regret with efficient region pruning.

Decision trees can be biased towards minority class, contrary to belief.

Stochastic Gradient TreeBoost is often found in many winning solutions in public data science challenges. Unfortunately, the best performance requires extensive parameter tuning and can be prone to overfitting. We propose PaloBoost, a Stochastic Gradient TreeBoost model that uses novel regularization techniques to guar…

The paper challenges the use of decision trees for pointwise inference due to slow convergence rates.

Simultaneous Latent Budget Trees for stratified classification

Regression trees are becoming increasingly popular as omnibus predicting tools and as the basis of numerous modern statistical learning ensembles. Part of their popularity is their ability to create a regression prediction without ever specifying a structure for the mean model. However, the method implicitly assumes ho…

HS improves tree-based models' accuracy and interpretability without changing their structure.

Nonparametric estimation of the conditional distribution of a response given high-dimensional features is a challenging problem. It is important to allow not only the mean but also the variance and shape of the response density to change flexibly with features, which are massive-dimensional. We propose a multiscale dic…

Depression is one of the most common mental health disorders, and a large number of depressed people commit suicide each year. Potential depression sufferers usually do not consult psychological doctors because they feel ashamed or are unaware of any depression, which may result in severe delay of diagnosis and treatme…