Unified theory and debiasing framework for random oblique projections in high dimensions.
problem Systematic statistical bias in random oblique projections induced by sampling.
method Unified non-asymptotic theory and debiasing framework.
result Sharp bias--variance characterizations and improved approximation accuracy.
Decision forests, including Random Forests and Gradient Boosting Trees, have recently demonstrated state-of-the-art performance in a variety of machine learning settings. Decision forests are typically ensembles of axis-aligned decision trees; that is, trees that split only along feature dimensions. In contrast, many r…
Proposes PredVAR model for reduced-dimensional dynamics from noisy data.
problem Extracting low-dimensional dynamics from high-dimensional noisy data.
method Probabilistic reduced-dimensional vector autoregressive model with oblique projection.
result Iterative algorithm yields dynamic latent variables with rank-ordered predictability.
Efficient oblique RSF method improves prediction and interpretability.
problem Limited computational efficiency and difficulty in interpreting oblique RSF ensembles.
method Newton-Raphson scoring for computational efficiency and negation importance for variable importance estimation.
result The method reduces computational overhead by 450 times and improves prediction accuracy.
Develops an oblique projection technique to approximate a foliation for non-normal dynamics.
problem Modeling dynamics far from a primary Spectral Submanifold (SSM) in non-normal systems.
method Oblique projection technique based on experimental data.
result Approximates a stable invariant foliation for non-normal dynamics efficiently.
This paper optimizes high-dimensional oblique splits for decision trees, enhancing performance and computational efficiency.
problem Enhancing decision tree performance and computational efficiency in high-dimensional data.
method Established Sufficient Impurity Decrease (SID) convergence for s0-sparse oblique splits, proposing progressive trees for iterative refinement. result Demonstrated that SID function class expands with s0-sparsity, enabling capture of complex data-generating processes. Oblique BART improves tree-based predictions.
problem Axis-aligned decision rules in BART can be suboptimal.
method Developed an oblique version of BART using data-adaptive hyperplane partitions.
result Oblique BART outperformed axis-aligned BART and other tree methods on benchmarks.
FoLDTree improves oblique decision trees with ULDA, enhancing accuracy and feature selection.
problem Axis-orthogonal splits limit traditional decision trees' performance on oblique decision boundaries.
method Integrates ULDA into decision tree structure for efficient oblique splits, feature selection, and handling missing values.
result FoLDTree outperforms other methods in accuracy and feature selection, comparable to random forest.
Machine learning identifies chimera states in complex dynamical systems.
problem Chimera states are hard to identify due to their varied appearance and peculiar nature.
method Machine learning techniques, specifically random forest and oblique random forest with null space regularization.
result High accuracy in identifying chimera states across different dynamical models.
New tree and forest methods use oblique splits for better risk bounds.
problem Improving risk bounds for regression algorithms.
method Randomized decision trees and forests with oblique splits.
result Oblique splits lead to better risk bounds for multi-index models.
Enhanced ODT with Feature Concatenation boosts learning efficiency.
problem Insufficient learning efficiency of ODT due to linear projections not being transmitted to child nodes.
method Feature Concatenation ( exttt{FC-ODT}) to transmit linear projections along decision paths.
result Experiments show exttt{FC-ODT} outperforms state-of-the-art decision trees with a limited tree depth.
Both neural networks and decision trees are popular machine learning methods and are widely used to solve problems from diverse domains. These two classifiers are commonly used base classifiers in an ensemble framework. In this paper, we first present a new variant of oblique decision tree based on a linear classifier,…
New random forest variants achieve optimal performance in high dimensions.
problem Handling dependencies between features in high-dimensional data.
method Using oblique splits in random forests with general split directions.
result Achieved minimax optimal convergence rates in arbitrary dimension.
New algorithms improve policy evaluation in reinforcement learning.
problem Off-policy stability and on-policy efficiency issues in policy evaluation.
method Introduced novel algorithms using oblique projection method.
result Demonstrated both off-policy stability and on-policy efficiency.
The paper simplifies hedging and portfolio allocation in markets without a risk-free asset.
problem Optimal hedging and portfolio allocation in markets without a risk-free asset.
method Establishes equivalence between hedging with and without numeraire change, uses oblique projections.
result Explicit expressions for optimal strategies and efficient frontier computation.
We present a new way of constructing an ensemble classifier, named the Guided Random Forest (GRAF) in the sequel. GRAF extends the idea of building oblique decision trees with localized partitioning to obtain a global partitioning. We show that global partitioning bridges the gap between decision trees and boosting alg…
Decision trees are a popular technique in statistical data classification. They recursively partition the feature space into disjoint sub-regions until each sub-region becomes homogeneous with respect to a particular class. The basic Classification and Regression Tree (CART) algorithm partitions the feature space using…
We introduce canonical correlation forests (CCFs), a new decision tree ensemble method for classification and regression. Individual canonical correlation trees are binary decision trees with hyperplane splits based on local canonical correlation coefficients calculated during training. Unlike axis-aligned alternatives…
Sparse oblique decision tree improves security rules for renewable power systems.
problem Identifying secure operating conditions in power systems with high renewable energy.
method Sparse weighted oblique decision tree to learn and embed linear security rules.
result The method significantly increases secure states and reduces solution time.
Proposes oblique predictive clustering trees for faster, more efficient learning.
problem Learning time scales poorly with output space dimensionality and cannot exploit data sparsity.
method Design and implement oblique splits using linear combinations of features.
result Achieves performance on-par with state-of-the-art methods and is orders of magnitude faster.
NR retraction approximates geodesics on submanifolds efficiently.
problem Efficiently approximating geodesics on submanifolds for practical algorithms.
method Introducing Newton retraction (NR) as a class of retractions on submanifolds induced by a foliation of the ambient manifold.
result NR is more stable and computationally cheaper than oblique projection, with superlinear convergence regions.
Decision forests (Forests), in particular random forests and gradient boosting trees, have demonstrated state-of-the-art accuracy compared to other methods in many supervised learning scenarios. In particular, Forests dominate other methods in tabular data, that is, when the feature space is unstructured, so that the s…
Resource-efficient oblique trees reduce neural signal classification costs.
problem Implementing efficient neural signal classifiers on resource-constrained devices.
method Integrating model compression, probabilistic routing, and cost-aware learning.
result Significant reduction in model size and feature extraction cost compared to state-of-the-art models.
We show how neural models can be used to realize piece-wise constant functions such as decision trees. The proposed architecture, which we call locally constant networks, builds on ReLU networks that are piece-wise linear and hence their associated gradients with respect to the inputs are locally constant. We formally …
Single tree outperforms random forest in testing accuracy.
problem The challenge of improving single decision tree performance.
method Gradient-based entire tree optimization framework, scaled sigmoid approximation, numerical stability algorithm, subtree polish strategy.
result Optimized single tree outperforms classic random forest by 2.03% on average.
Generalizes Hoeffding's decomposition for dependent inputs under mild conditions.
problem Performing global sensitivity analysis on black-box models with dependent inputs.
method Proposes a novel framework based on probability theory, functional analysis, and combinatorics to handle dependencies.
result Any square-integrable, real-valued function of random elements with mild dependence assumptions can be uniquely additively decomposed.
Surface area and mean width of a cylinder (the convex hull of two parallel disks) in R^3 are computed. It is more difficult to obtain analogous results for a cone (the convex hull of a disk D and a point p). Oblique formulas for mean width, as well as those for mean curvature, are new. Let L denote the unique diameter …
Paper proposes a novel SVM method for creating survival trees.
problem Creating non-linear survival trees for right-censored data.
method L2-regularized dipole splitting criteria with kernel methods.
result Non-linear splits using polynomial and Gaussian kernels show similar predictive power but often smaller tree sizes.
This paper presents four different ways of looking at the well-known Least Squares Temporal Differences (LSTD) algorithm for computing the value function of a Markov Reward Process, each of them leading to different insights: the operator-theory approach via the Galerkin method, the statistical approach via instrumenta…
New method learns nonlinear projections for reduced-order modeling of complex dynamical systems.
problem Modeling transient dynamics near a manifold in nonlinear systems.
method Constrained autoencoder neural networks with invertible activation functions and biorthogonal weight matrices.
result Demonstrated effectiveness on a vortex shedding model, learning oblique fibers for fast dynamics.
This paper surveys various methods for dimensionality reduction and nearest neighbor search.
problem Efficiently reducing high-dimensional data to lower dimensions while preserving essential information.
method Linear and nonlinear random projections, including sparse random projections, random Fourier Features, and Random Kitchen Sinks.
result Various methods for dimensionality reduction and nearest neighbor search are explained and compared.
The paper studies how norms of random vectors are preserved by random projections.
problem Understanding how random matrix affects norms of random vectors.
method Proved the distribution of the norm of random vector is preserved by random projection.
result Random matrix preserves the distribution of the norm of random vectors with i.i.d. entries.
Study integrates reliability constraints into generation planning models.
problem Challenges in integrating reliability constraints with generation planning models.
method Leverages a weighted oblique decision tree (WODT) technique to embed reliability verification constraints.
result Demonstrates effectiveness in achieving reliable and optimal planning solutions.
Paper studies binary random projections with controllable sparsity patterns for computational and accuracy advantages.
problem Improving computational efficiency and accuracy in random projections.
method Proposes two sparse binary projection models with controllable sparsity patterns.
result Significant computational advantages and improved accuracies in empirical evaluations.
The study calculates the average genus of 2-bridge knots based on their crossing numbers.
problem Determining the average genus of 2-bridge knots with a given crossing number.
method Analytical approach focusing on the properties of 2-bridge knots.
result Obtained the oblique asymptote of the average genus as crossing numbers increase.
Free Random Projection enhances reinforcement learning by naturally incorporating hierarchical structure.
problem Improving reinforcement learning algorithms for better generalization and adaptability.
method Introduces Free Random Projection, a method that uses free probability theory to create random orthogonal matrices encoding hierarchical structure.
result Empirically shows consistent improvement in generalization over standard methods on multi-environment benchmarks.
Conventional decision trees have a number of favorable properties, including interpretability, a small computational footprint and the ability to learn from little training data. However, they lack a key quality that has helped fuel the deep learning revolution: that of being end-to-end trainable, and to learn from scr…
Random projections have been applied in many machine learning algorithms. However, whether margin is preserved after random projection is non-trivial and not well studied. In this paper we analyse margin distortion after random projection, and give the conditions of margin preservation for binary classification problem…
Tensorized random projections reduce high-dimensional tensor size efficiently.
problem Efficiently reducing the dimension of very high-dimensional tensors.
method Proposes two tensorized random projection maps using TT and CP decompositions.
result TT format offers superior performance in terms of required random projection size.
Random projections help in representing sparse graphs efficiently.
problem Efficiently representing sparse graphs of varying sizes and vertex sets.
method Random projection of adjacency matrices to retain graph functionality and properties.
result Random projections can accurately represent graphs of different sizes and vertex sets in the same space.
Study shows double descent curve in high-dimensional linear regression with random projections.
problem Understanding the generalization performance in high-dimensional settings with random projections.
method Fixed prediction problem, ridge regression estimator, minimum norm least-squares fit, random matrix theory, asymptotic equivalents.
result Exhibit a double descent curve for high-dimensional linear regression with random projections.
A new topology design improves zero-shot classification performance in contrastive learning.
problem Improving zero-shot classification performance in contrastive visual-textual alignment.
method Proposed an alternative topology design using multiple class tokens and an oblique manifold with negative inner product.
result Improves zero-shot classification performance by an average of 6.1%.
A new framework for dimension reduction using ensemble of random projections.
problem High-dimensional regression problems with limited data.
method Aggregating an ensemble of carefully chosen random projections, retaining based on empirical performance, and selecting singular vectors.
result The proposed method stabilizes error as the number of projection groups increases.
Study on volumes of random inscribed polytopes in projective geometries.
problem Estimating volumes of random inscribed polytopes in projective geometries.
method Central limit theorems and normal approximation for volumes and dual volumes of random inscribed polytopes.
result Established central limit theorems and normal approximation for volumes and dual volumes of random inscribed polytopes.
The paper examines how well node similarities are preserved by random projections in graph embeddings.
problem The preservation of node similarities under random projections in graph embeddings.
method Investigation of dot product and cosine similarity preservation by random projections over graph matrix rows.
result Random projections produce unreliable embeddings for dot product, especially for high-degree nodes.
A new ensemble method using random projections for kNN classification.
problem Improving kNN classification accuracy through ensemble methods.
method Random projection of bootstrap samples into lower dimensions, using extended neighbourhood rule for base learners.
result Enhanced classification accuracy compared to traditional kNN and other ensembles.
NSOTree combines neural networks and trees for better survival analysis interpretability.
problem Survival analysis event prediction with interpretability.
method Proposes NSOTree integrating neural networks and trees for better interpretability.
result NSOTree improves both performance and interpretability in survival analysis.
Random projections offer an appealing and flexible approach to a wide range of large-scale statistical problems. They are particularly useful in high-dimensional settings, where we have many covariates recorded for each observation. In classification problems there are two general techniques using random projections. T…