Since its inception in the 1980s, ID3 has become one of the most successful and widely used algorithms for learning decision trees. However, its theoretical properties remain poorly understood. In this work, we introduce a novel metric of a decision tree algorithm's performance, called mean iteration statistical consis…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “wide trees”
Boosted decision trees enjoy popularity in a variety of applications; however, for large-scale datasets, the cost of training a decision tree in each round can be prohibitively expensive. Inspired by ideas from the multi-arm bandit literature, we develop a highly efficient algorithm for computing exact greedy-optimal d…
Latent tree modelsmath.ST
Latent tree models are graphical models defined on trees, in which only a subset of variables is observed. They were first discussed by Judea Pearl as tree-decomposable distributions to generalise star-decomposable distributions such as the latent class model. Latent tree models, or their submodels, are widely used in:…
A new random forest algorithm improves tree construction for optimal performance.
problem Improving the performance of random forests, especially in complex and smooth scenarios.
method Adaptive split-balancing method using permutation-based splitting criterion.
result Achieves minimax optimality under various Lipschitz and Hölder classes.
PhyloVAE learns tree topologies without supervision.
problem Learning accurate tree representations from evolutionary data.
method Unsupervised learning via variational autoencoders with efficient tree generation.
result PhyloVAE generates high-resolution tree topologies efficiently.
Towards Scalable Meta-Learning of near-optimal Interpretable Models via Synthetic Model Generationscs.LG
Meta-learning interpretable decision trees with synthetic data.
problem Lack of efficient, scalable methods for generating synthetic data for decision tree meta-learning.
method Synthetic generation of near-optimal decision trees using the MetaTree transformer architecture.
result Meta-learning of decision trees achieves performance comparable to real-world data or optimal decision trees, with significant computational cost reduction.
We introduce a novel boosting algorithm called `KTBoost' which combines kernel boosting and tree boosting. In each boosting iteration, the algorithm adds either a regression tree or reproducing kernel Hilbert space (RKHS) regression function to the ensemble of base learners. Intuitively, the idea is that discontinuous …
Analyzing CARTstat.ML
Decision trees with binary splits are popularly constructed using Classification and Regression Trees (CART) methodology. For binary classification and regression models, this approach recursively divides the data into two near-homogenous daughter nodes according to a split point that maximizes the reduction in sum of …
New framework uses tree ensembles for contextual bandits.
problem Optimizing decisions in dynamic environments with contextual information.
method Adapts Upper Confidence Bound and Thompson Sampling to tree ensemble methods.
result Tree ensemble methods outperform traditional methods in regret minimization and runtime.
OPLTs online train label trees for multi-label and multi-class classification.
problem Online multi-label and multi-class classification challenges.
method Fully online training of label trees without prior knowledge.
result Strong theoretical guarantees and low complexity.
Flexible tree ensemble learning framework supports arbitrary loss functions and multi-task learning.
problem Limited modeling capabilities of existing tree ensemble learning toolkits.
method Differentiable tree ensembles with tensor-based formulation for efficient training.
result Our framework leads to 100x more compact and 23% more expressive tree ensembles.
Tree-Structured Boosting: Connections Between Gradient Boosted Stumps and Full Decision Treesstat.ML
Additive models, such as produced by gradient boosting, and full interaction models, such as classification and regression trees (CART), are widely used algorithms that have been investigated largely in isolation. We show that these models exist along a spectrum, revealing never-before-known connections between these t…
Tree Ensemble Explainability through the Hoeffding Functional Decomposition and TreeHFD Algorithmstat.ML
TreeHFD algorithm explains tree ensemble models through hierarchical orthogonality.
problem Difficulty in explaining black-box tree ensemble models.
method TreeHFD algorithm using hierarchical orthogonality constraints.
result TreeHFD estimates Hoeffding decomposition from data samples.
We consider the classical sparse regression problem of recovering a sparse signal given a measurement vector . We propose a tree search algorithm driven by the deep neural network for sparse regression (TSN). TSN improves the signal reconstruction performance of the deep neural network designed for sp…
Glitches cause unreliable AI decisions with steep boundaries.
problem Glitches impair the reliability of AI models with steep decision boundaries.
method Formal definition of glitches, algorithmic search using MILP encoding.
result Glitches are widespread and indicate potential model inconsistencies.
User preferences for items can be inferred from either explicit feedback, such as item ratings, or implicit feedback, such as rental histories. Research in collaborative filtering has concentrated on explicit feedback, resulting in the development of accurate and scalable models. However, since explicit feedback is oft…
Study a risk model with tree-structured Poisson-Markov random field for rainfall events.
problem Dependence between rainfall frequencies in insurance portfolios.
method Tree-structured Markov random field with Poisson marginals.
result Asymptotic results for portfolio risk and risk allocation.
Algorithm recovers permutations of high-dimensional Gaussian vectors with constant correlation.
problem Recovering permutations of high-dimensional Gaussian vectors with constant correlation.
method Computing and comparing weighted counts of specially chosen wide trees.
result Polynomial-time algorithm for exact recovery at constant correlation.
ENTMOOT integrates tree models for better decision-making and optimization.
problem Difficult optimization of tree models and lack of reliable uncertainty measures.
method Integrates already trained tree models into optimization problems with a reliable uncertainty measure.
result Proves globally optimal solutions for optimization problems.
Tree ensembles such as Random Forests have achieved impressive empirical success across a wide variety of applications. To understand how these models make predictions, people routinely turn to feature importance measures calculated from tree ensembles. It has long been known that Mean Decrease Impurity (MDI), one of t…
HATT improves online decision tree ensembles by using a more eager splitting strategy.
problem Improving the efficiency of online decision tree ensembles.
method Replacing Hoeffding Tree's split strategy with HATT, which uses the Hoeffding Test for candidate splits.
result HATT outperforms Hoeffding Tree in online bagging and boosting ensembles, as shown by significant performance improvements in various testbenches.
Obtaining accurate and well calibrated probability estimates from classifiers is useful in many applications, for example, when minimising the expected cost of classifications. Existing methods of calibrating probability estimates are applied globally, ignoring the potential for improvements by applying a more fine-gra…
Tree tensor networks balance model complexity and empirical risk for high-dimensional function approximation.
problem Selecting optimal tree structure and ranks for high-dimensional function approximation.
method Proposes a complexity-based model selection method for tree tensor networks in empirical risk minimization.
result Demonstrates near-minimax adaptive performance across various smoothness classes.
AF improves classification models by adaptively weighting trees.
problem Improving classification model performance.
method AF combines OP2T for input-dependent weights and MIO for dynamic refinement.
result AF consistently outperforms RF, XGBoost, and other weighted RF.
Shallow trees in ensemble models make models more interpretable and sometimes better.
problem Lack of transparency in high-performing tree ensemble models.
method Developed an interpretation algorithm to convert tree ensembles into functional ANOVA representations. Proposed strategies to enhance interpretability.
result Shallow trees in ensemble models can lead to better generalization performance and improved interpretability.
JSRT improves regression tree performance by incorporating global node information.
problem Regression tree performance relies on local node means, ignoring global node information.
method Proposes JSRT by integrating global mean information from different nodes.
result Demonstrates superior performance and efficiency compared to other regression tree methods.
Optimal decision trees learned via dynamic programming and search.
problem Finding the best classification trees that are both accurate and compact.
method Dynamic programming and search techniques.
result Our algorithm learns optimal classification trees significantly faster and scales better than existing methods.
Improved Random Forests detect pure interactions better.
problem Random Forests struggle with certain pure interactions.
method Alternative partitioning schemes during tree construction.
result Improved Random Forests enhance fitting ability in scenarios with pure interactions.
The paper introduces staged event trees for transparent treatment effect estimation.
problem Estimating treatment effects from observational data.
method Staged event trees framework for causal inference.
result Enhanced treatment effect estimation with improved interpretability.
Proposes new attribution methods for trees with regularization.
problem Feature attribution for trees trained with regularization.
method Prediction Decomposition Attribution (PreDecomp) and TreeInner.
result TreeInner shows state-of-the-art feature selection performance.
Improved Shapley Values for tree-based models, more accurate than existing methods.
problem Inaccurate Shapley Values in tree-based models leading to poor explanations.
method Introduced two new estimators exploiting tree structure, derived correct approach for categorical variables.
result More accurate Shapley Values for tree-based models, demonstrated through simulations.
Integrates differentiable decision trees into neural networks for faster training and inference.
problem Combining differentiability and conditional computation in tree ensembles for neural networks.
method Sparse activation function and specialized forward/backward propagation algorithms for efficient training and inference.
result 10x speed-ups and 20x reduction in parameters compared to existing methods, while maintaining performance.
Decision trees are consistent for regression and classification tasks even with many predictors.
problem Consistency of decision trees with many predictors.
method CART and C4.5 methodology, oracle inequality, sparsity constraints.
result Decision trees and random forests are consistent for various types of data.
New test improves tree ensemble pruning for better model performance.
problem Lack of robust theoretical justification for penalty terms in tree ensembles.
method Developed a novel hypothesis test for tree ensemble split quality.
result Significant reduction in out-of-sample loss using the new test.
This paper develops a novel stochastic tree ensemble method for nonlinear regression, which we refer to as XBART, short for Accelerated Bayesian Additive Regression Trees. By combining regularization and stochastic search strategies from Bayesian modeling with computationally efficient techniques from recursive partiti…
New method tests tree models without causing computational pressure.
problem Incompatibility of adversarial robustness testing with tree ensembles.
method Smooths tree ensembles with sigmoid functions and uses gradient descent.
result Successfully reveals adversarial vulnerability of tree ensemble models.
Max-Cut decision tree improves classification accuracy and reduces computation time.
problem Improving decision tree accuracy and efficiency for complex classification tasks.
method Alternative splitting metric (max cut) and PCA-based feature selection at each node.
result 49% improvement in accuracy with 94% reduction in CPU time on CIFAR-100 data.
In machine learning ensemble methods have demonstrated high accuracy for the variety of problems in different areas. Two notable ensemble methods widely used in practice are gradient boosting and random forests. In this paper we present InfiniteBoost - a novel algorithm, which combines important properties of these two…
Tree structured graphical models are powerful at expressing long range or hierarchical dependency among many variables, and have been widely applied in different areas of computer science and statistics. However, existing methods for parameter estimation, inference, and structure learning mainly rely on the Gaussian or…
Covariance-Driven Regression Trees reduce overfitting in CART.
problem Overfitting in CART decision trees, especially with small sample sizes.
method Covariance-driven splitting criterion for regression trees (CovRT).
result CovRT achieves superior prediction accuracy compared to CART in simulations and real-world tasks.
A new metric for comparing measures on tree systems reduces computational burden.
problem Heavy computation in Optimal Transport problems.
method Introducing tree systems and a novel metric (Tree-Sliced Wasserstein distance on Systems of Lines, TSW-SL).
result TSW-SL performs favorably compared to Sliced Wasserstein and its variants.
Forest Garrotestat.ML
Variable selection for high-dimensional linear models has received a lot of attention lately, mostly in the context of l1-regularization. Part of the attraction is the variable selection effect: parsimonious models are obtained, which are very suitable for interpretation. In terms of predictive power, however, these re…
SMAC method optimizes tree-boosting hyperparameters best.
problem Optimizing hyperparameters for tree-boosting to improve model accuracy.
method Compared and evaluated various hyperparameter optimization methods.
result SMAC method outperforms other methods for hyperparameter tuning.
A novel tree algorithm improves time series forecasting accuracy.
problem Improving accuracy in non-linear time series forecasting.
method Developed a hierarchical TAR model as a regression tree that trains globally across series, introducing a forecasting-specific tree algorithm with cross-series learning.
result Significantly higher accuracy than state-of-the-art tree-based algorithms and benchmarks across four metrics.
HS improves tree-based models' accuracy and interpretability without changing their structure.
problem Overfitting in tree-based models.
method Hierarchical Shrinkage (HS) post-hoc algorithm that shrinks tree predictions towards ancestor means.
result HS significantly improves predictive performance and interpretability of decision trees and RFs.
Gaussian Recombining Split Treeq-fin.CP
A new tree model, GRST, improves option pricing without log-normality assumptions.
problem Limitations of CRR binomial trees in valuing securities with early exercise characteristics.
method Gaussian Recombining Split Tree (GRST) that generates a discrete probability mass function approximating a Gaussian distribution.
result Option prices from GRST align closely with market prices.
In AI research and industry, machine learning is the most widely used tool. One of the most important machine learning algorithms is Gradient Boosting Decision Tree, i.e. GBDT whose training process needs considerable computational resources and time. To shorten GBDT training time, many works tried to apply GBDT on Par…
New methods improve tree ensemble models by compressing them while maintaining accuracy.
problem Theoretical understanding and practical compression of tree ensembles like random forests and gradient boosting machines.
method Spectral perspective on tree ensembles, deriving minimax rates and developing compression schemes.
result Leading eigenfunctions/singular vectors capture dominant predictive directions, leading to smaller, competitive models.