Novel algorithm optimizes decision trees for nonlinear metrics.
problem Optimizing decision trees for nonlinear metrics like F1-score.
method Bi-objective optimisation approach to find optimal trees on Pareto frontier.
result The optimal tree for nonlinear metrics lies on the Pareto frontier.
We consider the problem of learning a forest of nonlinear decision rules with general loss functions. The standard methods employ boosted decision trees such as Adaboost for exponential loss and Friedman's gradient boosting for general loss. In contrast to these traditional boosting algorithms that treat a tree learner…
As data collections become larger, exploratory regression analysis becomes more important but more challenging. When observations are hierarchically clustered the problem is even more challenging because model selection with mixed effect models can produce misleading results when nonlinear effects are not included into…
Despite outstanding contribution to the significant progress of Artificial Intelligence (AI), deep learning models remain mostly black boxes, which are extremely weak in explainability of the reasoning process and prediction results. Explainability is not only a gateway between AI and society but also a powerful tool t…
In this paper we propose a synergistic melting of neural networks and decision trees (DT) we call neural decision trees (NDT). NDT is an architecture a la decision tree where each splitting node is an independent multilayer perceptron allowing oblique decision functions or arbritrary nonlinear decision function if more…
This paper uses NLDT to find interpretable control rules from complex DRL policies.
problem Complex, non-interpretable policies from black-box AI methods.
method Evolutionary optimization of NLDT for hierarchical control rules.
result Interpretable control rules with similar performance to black-box DRL.
Decision Machines embeds decision trees into vector spaces for improved optimization.
problem Overfitting and difficulty in finding optimal decision tree structure.
method Embedding Boolean tests into a binary vector space and representing tree structure as matrices.
result Optimized decision trees with enhanced predictive power.
Boosting meta-trees improve decision tree performance.
problem Overfitting in decision trees.
method Boosting approach to construct multiple meta-trees.
result Ensembles of meta-trees prevent overfitting.
Attention augments forest for tabular data accuracy.
problem Training tabular data models with high accuracy and efficiency.
method Tree Attention Block (TAB) in differentiable forest framework.
result Attention augmented differentiable forest achieves comparable and sometimes higher accuracy than GBDT models.
The paper studies statistical properties of CART regression trees.
problem Understanding the statistical properties of CART regression trees.
method The paper constructs a prior distribution on split points and solves a nonlinear optimization problem to bound the Pearson correlation between the optimal decision stump and response data.
result CART with cost-complexity pruning achieves an optimal complexity/goodness-of-fit tradeoff when the depth scales with the logarithm of the sample size.
Decision tree learning is a popular classification technique most commonly used in machine learning applications. Recent work has shown that decision trees can be used to represent provably-correct controllers concisely. Compared to representations using lookup tables or binary decision diagrams, decision trees are sma…
The study analyzes when Bayesian averaging over decision trees is reliable.
problem When do Bayesian model averaging weights over decision trees provide reliable information?
method Closed-form solution for Bayesian decision trees with Catalan-exponential priors.
result Established a complete non-asymptotic theory of rational commitment thresholds.
Electricity is bought and sold in wholesale markets at prices that fluctuate significantly. Short-term forecasting of electricity prices is an important endeavor because it helps electric utilities control risk and because it influences competitive strategy for generators. As the "smart grid" grows, short-term price fo…
Ability for accurate hospital case cost modelling and prediction is critical for efficient health care financial management and budgetary planning. A variety of regression machine learning algorithms are known to be effective for health care cost predictions. The purpose of this experiment was to build an Azure Machine…
Trinary decision tree improves handling of missing data in machine learning.
problem Improving accuracy in decision tree algorithms when dealing with missing data.
method Introduces Trinary decision tree, which does not assume missing values contain information about the response.
result Trinary decision tree outperforms other algorithms in Missing Completely at Random settings, especially when data is only missing out-of-sample.
Meta-learning interpretable decision trees with synthetic data.
problem Lack of efficient, scalable methods for generating synthetic data for decision tree meta-learning.
method Synthetic generation of near-optimal decision trees using the MetaTree transformer architecture.
result Meta-learning of decision trees achieves performance comparable to real-world data or optimal decision trees, with significant computational cost reduction.
The paper improves decision tree stability for health care applications.
problem Stability of decision trees in health care applications.
method Introducing a new distance metric to determine tree stability and proposing a novel training methodology.
result On average, a 4.6% decrease in predictive power yields a 38% increase in model stability.
Convex polytope trees expand decision trees with interpretable boundaries.
problem High accuracy often requires many nodes in decision trees, reducing interpretability.
method CPT uses logical disjunction of weighted linear decision-makers, geometrically a convex polytope.
result CPT achieves high accuracy with fewer nodes compared to existing methods.
FoLDTree improves oblique decision trees with ULDA, enhancing accuracy and feature selection.
problem Axis-orthogonal splits limit traditional decision trees' performance on oblique decision boundaries.
method Integrates ULDA into decision tree structure for efficient oblique splits, feature selection, and handling missing values.
result FoLDTree outperforms other methods in accuracy and feature selection, comparable to random forest.
Paper introduces algorithms for private decision tree learning.
problem Private decision tree learning in distributed settings.
method Proposes DP-TopDown, NoisyCounts, and LocalRNM.
result First utility guarantees for private decision tree learning.
Enhanced ODT with Feature Concatenation boosts learning efficiency.
problem Insufficient learning efficiency of ODT due to linear projections not being transmitted to child nodes.
method Feature Concatenation ( exttt{FC-ODT}) to transmit linear projections along decision paths.
result Experiments show exttt{FC-ODT} outperforms state-of-the-art decision trees with a limited tree depth.
Max-Cut decision tree improves classification accuracy and reduces computation time.
problem Improving decision tree accuracy and efficiency for complex classification tasks.
method Alternative splitting metric (max cut) and PCA-based feature selection at each node.
result 49% improvement in accuracy with 94% reduction in CPU time on CIFAR-100 data.
Transform ANNs into interpretable decision trees.
problem Lack of interpretability in ANNs.
method Developed two MDT algorithms: EC-DT and Extended C-Net.
result Extended C-Net generates the most compact and effective trees.
New techniques optimize decision trees for interpretable machine learning.
problem Optimizing decision trees for interpretable machine learning.
method General framework for decision tree optimization addressing imbalanced data and continuous variables.
result Proves optimal decision trees for various objectives.
This work presents an approach to automatically induction for non-greedy decision trees constructed from neural network architecture. This construction can be used to transfer weights when growing or pruning a decision tree, allowing non-greedy decision tree algorithms to automatically learn and adapt to the ideal arch…
This paper improves Bayesian decision tree learning using HMC.
problem Bayesian decision tree learning is challenging due to a large parameter space.
method Develops and compares HMC-based algorithms for exploring Bayesian decision tree posteriors.
result HMC-based methods outperform existing methods in predictive accuracy and tree complexity.
Decision trees are an extremely popular machine learning technique. Unfortunately, overfitting in decision trees still remains an open issue that sometimes prevents achieving good performance. In this work, we present a novel approach for the construction of decision trees that avoids the overfitting by design, without…
Decision trees improve performance in various fields.
problem Improving performance in machine learning and data mining.
method Introduce tree-based models and training techniques.
result Highlight probabilistic characteristics and practical benefits.
Novel approach for creating interpretable classifiers using bilevel optimization of split-rules in NLDTs.
problem Creating highly accurate and easily interpretable classifiers for practical applications.
method Representing classifiers as assemblies of simple mathematical rules using NLDTs with evolutionary bilevel optimization.
result The approach ensures interpretability while achieving high accuracy on various classification problems.
Decision trees improve decision-making by optimizing predictions of unknown parameters.
problem Optimizing decisions based on predicted unknown parameters.
method SPO Trees (SPOTs) for training decision trees under the SPO loss function.
result SPOTs provide higher quality decisions and significantly lower model complexity compared to other machine learning approaches.
Decision trees can be biased towards minority class, contrary to belief.
problem Bias in decision trees towards minority class in imbalanced datasets.
method Critical evaluation of past literature, specific conditions analysis, tree-fitting adjustments, and post-hoc calibration methods.
result Decision trees can be biased towards minority class under specific conditions, not always towards majority.
We show how neural models can be used to realize piece-wise constant functions such as decision trees. The proposed architecture, which we call locally constant networks, builds on ReLU networks that are piece-wise linear and hence their associated gradients with respect to the inputs are locally constant. We formally …
We introduce a novel incremental decision tree learning algorithm, Hoeffding Anytime Tree, that is statistically more efficient than the current state-of-the-art, Hoeffding Tree. We demonstrate that an implementation of Hoeffding Anytime Tree---"Extremely Fast Decision Tree", a minor modification to the MOA implementat…
Decision trees are ubiquitous in machine learning for their ease of use and interpretability. Yet, these models are not typically employed in reinforcement learning as they cannot be updated online via stochastic gradient descent. We overcome this limitation by allowing for a gradient update over the entire tree that i…
In this paper, we introduce a new machine learning (ML) model for nonlinear regression called the Boosted Smooth Transition Regression Trees (BooST), which is a combination of boosting algorithms with smooth transition regression trees. The main advantage of the BooST model is the estimation of the derivatives (partial…
Algorithm learns decision trees from noisy data.
problem Learning stochastic decision trees from corrupted samples.
method Quasipolynomial-time algorithm for adversarial noise.
result Returns a hypothesis with error within 2η+ε of optimal. Rectified decision trees improve machine learning interpretability and effectiveness.
problem Combining interpretability and effectiveness in machine learning models.
method Knowledge distillation and modified decision tree splitting criteria.
result Soft labels improve model performance and reduce model size.
Study explores loss design for decision trees to improve robustness against noisy labels.
problem Improving decision tree robustness to noisy labels.
method Investigated loss correction and symmetric losses, found ineffective.
result Other loss design directions need exploration for robust decision trees.
Develops a new method for decision trees using categorical variable structure.
problem Lack of structure in treating categorical variables as predictors.
method Introduces a mathematical framework to represent categorical structure and generalizes decision trees to utilize this structure.
result Improves prediction accuracy on weather data using the new method.
A novel gradient-based method optimizes decision trees for complex tasks.
problem Training decision trees with arbitrary differentiable loss functions.
method Gradient-based optimization using first and second derivatives of loss functions.
result Improves accuracy and flexibility in decision tree optimization.
Decision trees improve intraday trading strategies for NIFTY50 stocks.
problem Creating optimal trading rules for individual stocks.
method Using decision trees to generate unique trading rules for each stock based on technical indicators.
result Decision tree strategies outperform simple buy-and-hold for many stocks.
This paper solves the open problem of computing Bayes optimal prediction for decision trees using a Markov chain Monte Carlo method.
problem Computing the Bayes optimal prediction for decision trees is infeasible due to an infeasible summation over all division patterns of a feature space.
method Solved the open problem using a Markov chain Monte Carlo method with adaptively tuned step size.
result Computed the Bayes optimal prediction for decision trees using a Markov chain Monte Carlo method.
In this paper, we present a novel sequential paradigm for classification in crowdsourcing systems. Considering that workers are unreliable and they perform the tests with errors, we study the construction of decision trees so as to minimize the probability of mis-classification. By exploiting the connection between pro…
Optimal decision trees learned via dynamic programming and search.
problem Finding the best classification trees that are both accurate and compact.
method Dynamic programming and search techniques.
result Our algorithm learns optimal classification trees significantly faster and scales better than existing methods.
Decision tree algorithms have been among the most popular algorithms for interpretable (transparent) machine learning since the early 1980's. The problem that has plagued decision tree algorithms since their inception is their lack of optimality, or lack of guarantees of closeness to optimality: decision tree algorithm…
Decision tree learning heuristics fail even in smoothed analysis for complex targets.
problem Greedy decision tree learning heuristics fail for complex target functions in the smoothed analysis model.
method Construct counterexamples and analyze the behavior of heuristics in the smoothed setting and agnostic setting.
result Greedy decision tree learning heuristics can build trees of exponential depth before achieving high accuracy for certain complex target functions.
A new decision tree induction method using MIP for faster optimization.
problem Optimizing decision tree split rules for better classification performance.
method Mixed-integer programming (MIP) for Gini reduction maximization, efficient search algorithm.
result bsnsing trees outperform other tree models in new case discrimination.
In this paper we propose a method to build a neural network that is similar to an ensemble of decision trees. We first illustrate how to convert a learned ensemble of decision trees to a single neural network with one hidden layer and an input transformation. We then relax some properties of this network such as thresh…