Paper studies ensemble probabilistic regression trees for smooth approximations.
problem Smooth approximations of regression functions.
method Ensemble versions of probabilistic regression trees.
result Ensemble probabilistic regression trees are consistent and perform well.
SBT model uses randomized sharding and sub-models to improve Bayesian Additive Regression Trees.
problem Improving efficiency and accuracy of Bayesian Additive Regression Trees.
method Randomized sharding, sub-models, intersection tree structure, optimal design.
result Theoretical optimal weights and worst-case complexity of SBT model.
A new tree method for tensor data improves regression accuracy.
problem Efficiently modeling tensor data for regression problems.
method Scalar-output regression tree models for scalar-on-tensor problems, and tensor-on-tensor problems using additive tree ensemble approaches.
result The tensor-input tree (TT) method outperforms tensor-input GP models in efficiency and accuracy.
Regression trees learn gradients of differentiable functions.
problem Understanding gradients of differentiable functions using regression trees.
method Developed a method to estimate gradients of differentiable functions using regression trees and exposed quantities from tree learning libraries.
result Gradient estimates from regression trees can be used to improve predictive analysis and solve tasks in uncertainty quantification.
Regression Trees analyze stock returns, revealing market excess return as the most informative factor.
problem Understanding informational content of three factors in stock returns.
method Joint regression tree analysis of daily stock return data for 5 major US corporations.
result The market excess return factor is always the most informative in all cases (solo and joint).
Ensemble of regression trees have become popular statistical tools for the estimation of conditional mean given a set of predictors. However, quantile regression trees and their ensembles have not yet garnered much attention despite the increasing popularity of the linear quantile regression model. This work proposes a…
SMART combines decision trees and MARS for better regression modeling.
problem High variance in decision trees for continuous relationships, poor performance in MARS for discontinuities.
method SMART uses a decision tree to identify subsets with distinct continuous relationships, then applies MARS to fit these relationships independently.
result SMART improves regression performance over state-of-the-art methods in capturing discontinuities and continuous relationships.
JSRT improves regression tree performance by incorporating global node information.
problem Regression tree performance relies on local node means, ignoring global node information.
method Proposes JSRT by integrating global mean information from different nodes.
result Demonstrates superior performance and efficiency compared to other regression tree methods.
New trees-based models handle correlated data better.
problem Standard trees-based models ignore correlation structure.
method Explicitly accounts for correlation structure in splitting criterion, stopping rules, and fitted values.
result New approach superior to standard models in simulations and real data.
GP-BART improves BART's predictive performance by incorporating Gaussian process priors.
problem Lack of smoothness and explicit covariance structure in BART.
method GP-BART extends BART with Gaussian process priors for tree predictions.
result GP-BART outperforms traditional models in various applications.
Enhances time-series regression trees with latent factors for robust financial analysis.
problem Handling predictors with measurement error, trends, seasonality, and missing data.
method Integrates latent stationary factors extracted via state-space methods into time-series regression trees.
result Factor-augmented trees provide a reliable approach for macro-finance problems, exemplified by the lead-lag effect between equity volatility and the business cycle.
We study the convergence of the predictive surface of regression trees and forests. To support our analysis we introduce a notion of adaptive concentration for regression trees. This approach breaks tree training into a model selection phase in which we pick the tree splits, followed by a model fitting phase where we f…
A new type of distributional regression tree uses soft split rules for better predictive performance.
problem Estimating complete conditional distributions in regression.
method Distributional adaptive soft regression trees using multivariate soft split rules.
result The method outperforms various benchmark methods, especially in complex non-linear interactions.
This study converts BART to Gaussian process regression, revealing its limitations and potential improvements.
problem Understanding the Gaussian process limit of BART and its implications.
method Deriving and computing BART's prior covariance function, implementing the infinite trees limit as GP regression, and tuning hyperparameters.
result The Gaussian process limit of BART is inferior to standard BART but can be made competitive with proper hyperparameter tuning.
The paper uses regression trees/random forests to price Bermudan options more efficiently.
problem Pricing Bermudan options with conditional expectation estimation.
method Estimates conditional expectations using regression trees or random forests instead of traditional regression methods.
result Regression trees/random forests provide better results in high dimensions.
We consider the classical sparse regression problem of recovering a sparse signal x0 given a measurement vector y=Φx0+w. We propose a tree search algorithm driven by the deep neural network for sparse regression (TSN). TSN improves the signal reconstruction performance of the deep neural network designed for sp…
In this paper, we investigate adaptive nonlinear regression and introduce tree based piecewise linear regression algorithms that are highly efficient and provide significantly improved performance with guaranteed upper bounds in an individual sequence manner. We use a tree notion in order to partition the space of regr…
In this paper, we introduce a new machine learning (ML) model for nonlinear regression called the Boosted Smooth Transition Regression Trees (BooST), which is a combination of boosting algorithms with smooth transition regression trees. The main advantage of the BooST model is the estimation of the derivatives (partial…
flexBART improves BART for categorical predictors by creating flexible tree partitions.
problem Limitation of BART in handling categorical predictors with one-hot encoding.
method flexBART re-implements BART with regression trees that can assign multiple levels to both branches of a decision tree node, and proposes a new decision rule prior for spatial data.
result flexBART often yields improved predictive performance and scales better to larger datasets than existing BART implementations.
BART and MOTR-BART improve tree-based predictions with local linear models.
problem Non-linearity and high-order interactions in data.
method Bayesian Additive Regression Trees (BART) and Model Trees BART (MOTR-BART) using piecewise linear functions.
result MOTR-BART achieves equal or better performance with fewer trees than BART.
Oblique BART improves tree-based predictions.
problem Axis-aligned decision rules in BART can be suboptimal.
method Developed an oblique version of BART using data-adaptive hyperplane partitions.
result Oblique BART outperformed axis-aligned BART and other tree methods on benchmarks.
Regression-via-Classification (RvC) is the process of converting a regression problem to a classification one. Current approaches for RvC use ad-hoc discretization strategies and are suboptimal. We propose a neural regression tree model for RvC. In this model, we employ a joint optimization framework where we learn opt…
We analyze the consistency of decision trees and random forests in regression.
problem Consistency of decision trees and random forests in regression.
method Elementary proofs following classical arguments for smoothing methods.
result Establish weak and almost sure convergence of honest trees and forest averages to the true regression function.
A novel tree algorithm improves time series forecasting accuracy.
problem Improving accuracy in non-linear time series forecasting.
method Developed a hierarchical TAR model as a regression tree that trains globally across series, introducing a forecasting-specific tree algorithm with cross-series learning.
result Significantly higher accuracy than state-of-the-art tree-based algorithms and benchmarks across four metrics.
GPU-accelerated BART speeds up Bayesian regression.
problem Long running time of BART makes it impractical for large datasets.
method GPU-enabled implementation of BART.
result BART is now 200x faster on GPUs.
Tree ensembles like RF and GBT can be seen as kernels, improving regression and classification performance.
problem Improving kernel methods for tree ensemble based models.
method Investigation of RF and GBT kernels in simulation and real data.
result RF and GBT kernels are competitive to their respective ensembles in higher dimensions, particularly with noisy features.
Latent factor models have achieved great success in personalized recommendations, but they are also notoriously difficult to explain. In this work, we integrate regression trees to guide the learning of latent factor models for recommendation, and use the learnt tree structure to explain the resulting latent factors. S…
Additive regression trees are flexible non-parametric models and popular off-the-shelf tools for real-world non-linear regression. In application domains, such as bioinformatics, where there is also demand for probabilistic predictions with measures of uncertainty, the Bayesian additive regression trees (BART) model, i…
Decision trees are flexible models that are well suited for many statistical regression problems. In a Bayesian framework for regression trees, Markov Chain Monte Carlo (MCMC) search algorithms are required to generate samples of tree models according to their posterior probabilities. The critical component of such an …
Paper introduces methods to handle missing data in probabilistic regression trees.
problem Handling missing data in probabilistic regression trees.
method Three approaches: uniform probability, partial observation, and dimension-reduced smoothing.
result Preserves interpretability while extending applicability to incomplete datasets.
Develops regression trees for estimating cumulative incidence curves in competing risks.
problem Estimating cumulative incidence functions in competing risks settings.
method Uses augmented estimators of the Brier score risk to build and prune regression trees.
result Demonstrates the utility of the proposed methods through simulation studies and real data.
We introduce a novel boosting algorithm called `KTBoost' which combines kernel boosting and tree boosting. In each boosting iteration, the algorithm adds either a regression tree or reproducing kernel Hilbert space (RKHS) regression function to the ensemble of base learners. Intuitively, the idea is that discontinuous …
BARK optimizes black-box functions using Bayesian Additive Regression Trees.
problem Bayesian optimization of complex, black-box functions with uncertainty quantification.
method BART Kernel using tree agreement for posterior over piecewise-constant functions, explored using MCMC.
result BARK obtains samples of Gaussian processes for function distributions, enabling acquisition functions for optimization.
Two new methods improve monotonic constraint enforcement in regression and classification trees.
problem Improving monotonic constraints in regression and classification trees.
method Proposed two new methods: one yields better results than LightGBM, the other yields even better results but is slower.
result The best method consistently beats the current implementation of LightGBM, achieving up to 1% loss reduction.
Scaling regression to large datasets is a common problem in many application areas. We propose a two step approach to scaling regression to large datasets. Using a regression tree (CART) to segment the large dataset constitutes the first step of this approach. The second step of this approach is to develop a suitable r…
We present a detailed analysis of the class of regression decision tree algorithms which employ a regulized piecewise-linear node-splitting criterion and have regularized linear models at the leaves. From a theoretic standpoint, based on Rademacher complexity framework, we present new high-probability upper bounds for …
Tree-based ensemble methods, as Random Forests and Gradient Boosted Trees, have been successfully used for regression in many applications and research studies. Furthermore, these methods have been extended in order to deal with uncertainty in the output variable, using for example a quantile loss in Random Forests (Me…
TRUST improves tree models' accuracy while maintaining interpretability.
problem Piecewise-constant regression trees lack in predictive accuracy compared to black-box models.
method Combines Random Forest accuracy with interpretability of shallow trees and sparsity of linear models, using LLMs for explanations.
result TRUST outperforms other interpretable models in predictive accuracy and matches Random Forest's accuracy.
Fairness, through its many forms and definitions, has become an important issue facing the machine learning community. In this work, we consider how to incorporate group fairness constraints in kernel regression methods, applicable to Gaussian processes, support vector machines, neural network regression and decision t…
Decision trees are consistent for regression and classification tasks even with many predictors.
problem Consistency of decision trees with many predictors.
method CART and C4.5 methodology, oracle inequality, sparsity constraints.
result Decision trees and random forests are consistent for various types of data.
This paper develops a novel stochastic tree ensemble method for nonlinear regression, which we refer to as XBART, short for Accelerated Bayesian Additive Regression Trees. By combining regularization and stochastic search strategies from Bayesian modeling with computationally efficient techniques from recursive partiti…
Undirected graphical models encode in a graph G the dependency structure of a random vector Y. In many applications, it is of interest to model Y given another random vector X as input. We refer to the problem of estimating the graph G(x) of Y conditioned on X=x as ``graph-valued regression.'' In this pap…
Subagging improves regression tree performance, especially with many splits.
problem Improving regression tree performance with subsample aggregating.
method Formalized bias and variance dependencies, compared subagging to single trees, and analyzed optimal tree sizes.
result Subagging improves tree performance, especially with many splits.
Covariance-Driven Regression Trees reduce overfitting in CART.
problem Overfitting in CART decision trees, especially with small sample sizes.
method Covariance-driven splitting criterion for regression trees (CovRT).
result CovRT achieves superior prediction accuracy compared to CART in simulations and real-world tasks.
Tree-based synthesis improves forecast accuracy in GDP and inflation.
problem Improving forecast accuracy in GDP and inflation.
method Developed a nonparametric synthesis function using regression trees.
result Tree-based synthesis leads to improved forecast accuracy.
We propose to use boosted regression trees as a way to compute human-interpretable solutions to reinforcement learning problems. Boosting combines several regression trees to improve their accuracy without significantly reducing their inherent interpretability. Prior work has focused independently on reinforcement lear…
Tree-based algorithm for functional data analysis reduces generalization error.
problem Classification and regression problems with functional data.
method Constrained convex optimization for weighted functional L2 space, multiple splitting rules, and weighted integral features. result Reduces generalization error while maintaining interpretability.
The paper studies statistical properties of CART regression trees.
problem Understanding the statistical properties of CART regression trees.
method The paper constructs a prior distribution on split points and solves a nonlinear optimization problem to bound the Pearson correlation between the optimal decision stump and response data.
result CART with cost-complexity pruning achieves an optimal complexity/goodness-of-fit tradeoff when the depth scales with the logarithm of the sample size.