Paper develops a method to identify feature subspaces contributing to local data complexity.
problem Identifying feature subspaces that contribute to local data complexity.
method Develops an estimator of Local Intrinsic Dimension (LID) along axis projections to identify feature subspaces.
result Preliminary evidence suggests LID decomposition can indicate axis-aligned data subspaces supporting cluster formation.
GTBO uses group testing to optimize high-dimensional functions efficiently.
problem Optimizing expensive, high-dimensional functions with limited data.
method Group testing to identify active dimensions, then guide optimization.
result GTBO outperforms state-of-the-art methods on high-dimensional benchmarks.
BO method identifies sparse subspaces for efficient high-dimensional optimization.
problem Efficient optimization of high-dimensional black-box functions.
method Sparse Gaussian process surrogate models on axis-aligned subspaces with Hamiltonian Monte Carlo inference.
result SAASBO achieves excellent performance on synthetic and real-world problems.
GTBO uses group testing to optimize high-dimensional functions efficiently.
problem Challenges in optimizing high-dimensional, expensive functions due to the curse of dimensionality.
method GTBO combines testing and optimization phases to identify active variables and guide efficient optimization.
result GTBO outperforms state-of-the-art methods on high-dimensional optimization tasks.
We introduce a new method for sparse principal component analysis, based on the aggregation of eigenvector information from carefully-selected axis-aligned random projections of the sample covariance matrix. Unlike most alternative approaches, our algorithm is non-iterative, so is not vulnerable to a bad choice of init…
Two-dimensional embeddings remain the dominant approach to visualize high dimensional data. The choice of embeddings ranges from highly non-linear ones, which can capture complex relationships but are difficult to interpret quantitatively, to axis-aligned projections, which are easy to interpret but are limited to biva…
A new method for unsupervised disentanglement using axis-aligned cliffs.
problem Unsupervised disentanglement of latent factors under nonlinear maps.
method Encouraging axis-aligned discontinuities (cliffs) in the estimated density of factors.
result Cliff method outperforms baselines on disentanglement benchmarks.
Space partitioning methods such as random forests and the Mondrian process are powerful machine learning methods for multi-dimensional and relational data, and are based on recursively cutting a domain. The flexibility of these methods is often limited by the requirement that the cuts be axis aligned. The Ostomachion p…
Oblique BART improves tree-based predictions.
problem Axis-aligned decision rules in BART can be suboptimal.
method Developed an oblique version of BART using data-adaptive hyperplane partitions.
result Oblique BART outperformed axis-aligned BART and other tree methods on benchmarks.
New kernel interprets 3D anisotropic data with rotations and improved predictions.
problem Capturing rotated anisotropy in 3D spatial fields.
method Introduces a Lie-algebraic kernel with three principal length-scales and an explicit rotation.
result Posterior recovers rotated anisotropy and improves prediction over axis-aligned kernels.
Decision forests, including Random Forests and Gradient Boosting Trees, have recently demonstrated state-of-the-art performance in a variety of machine learning settings. Decision forests are typically ensembles of axis-aligned decision trees; that is, trees that split only along feature dimensions. In contrast, many r…
New tree and forest methods use oblique splits for better risk bounds.
problem Improving risk bounds for regression algorithms.
method Randomized decision trees and forests with oblique splits.
result Oblique splits lead to better risk bounds for multi-index models.
Tree regularization makes deep models interpretable by approximating them with simple decision trees.
problem Lack of interpretability in deep neural networks.
method Tree regularization to train deep models to resemble compact, axis-aligned decision trees.
result Tree regularized models are easier for humans to interpret without sacrificing accuracy.
New method learns representations for decision forests using input perturbation.
problem Decision forests struggle with raw structured data and lack effective representations.
method Approximate decision forest gradients through input perturbation.
result Effective representation learning for decision forests without structural changes.
We introduce canonical correlation forests (CCFs), a new decision tree ensemble method for classification and regression. Individual canonical correlation trees are binary decision trees with hyperplane splits based on local canonical correlation coefficients calculated during training. Unlike axis-aligned alternatives…
Bayesian nonparametric method partitions shapes using curves.
problem Capturing complex shapes in multi-dimensional data.
method Proposes a novel spline partitioning approach using curves.
result Demonstrates improved shape modeling compared to existing methods.
New random forest variants achieve optimal performance in high dimensions.
problem Handling dependencies between features in high-dimensional data.
method Using oblique splits in random forests with general split directions.
result Achieved minimax optimal convergence rates in arbitrary dimension.
This work uses stochastic geometry to improve STIT processes in machine learning.
problem Improving STIT processes for efficient and consistent machine learning applications.
method Utilizing tools from stochastic geometry to characterize kernels and obtain consistency results.
result Generalization of STIT processes and their kernels, leading to improved machine learning methods.
Sharp-SSL uses random projections to identify important variables for semi-supervised learning.
problem High-dimensional semi-supervised learning problems.
method Careful aggregation of low-dimensional results from many axis-aligned random projections.
result Sharp-SSL algorithm can recover signal coordinates with high probability.
Lean 4 formalizes Stokes' theorem for smooth singular cubes.
problem Formalizing Stokes' theorem for singular cubes in arbitrary dimensions.
method Using true differential-form pullback via Frechet derivative, bridging to mathlib4's extDeriv.
result d^2=0 for singular cubical chains, chain-level Stokes extended.
Decision trees and shallow neural networks have different geometric complexities, impacting their interpretability and accuracy.
problem The geometric simplicity of decision boundaries in decision trees conflicts with the approximation capabilities of shallow neural networks.
method Analysis of the Radon total variation (RTV) seminorm to compare geometric complexity of decision regions and neural network approximations.
result Smooth barrier scores can approximate decision regions with finite RTV, but their performance depends on the tube-mass condition near the decision boundary.
Decision Machines embeds decision trees into vector spaces for improved optimization.
problem Overfitting and difficulty in finding optimal decision tree structure.
method Embedding Boolean tests into a binary vector space and representing tree structure as matrices.
result Optimized decision trees with enhanced predictive power.
We introduce a new and improved characterization of the label complexity of disagreement-based active learning, in which the leading quantity is the version space compression set size. This quantity is defined as the size of the smallest subset of the training data that induces the same version space. We show various a…
The paper develops a new method to test if two multidimensional distributions are equivalent or significantly different.
problem Testing equivalence of multidimensional distributions with sub-linear sample complexity.
method Uses generalized A_k distance and Ramsey theory to develop a computationally efficient closeness tester.
result First sub-linear sample complexity closeness tester for multidimensional distributions.
Paper uses random projection to preserve subspace structure for efficient data analysis.
problem Efficiently analyzing data with low-dimensional structure.
method Compressed Subspace Learning (CSL) framework based on Johnson-Lindenstrauss property.
result Random projection preserves the UoS structure of data, enabling efficient analysis.
2D Total Variation Denoising (TVD) is a widely used technique for image denoising. It is also an important nonparametric regression method for estimating functions with heterogenous smoothness. Recent results have shown the TVD estimator to be nearly minimax rate optimal for the class of functions with bounded variatio…
Paper bounds subspace estimator error from noisy projections.
problem Estimating subspaces from noisy data.
method Derives perturbation bound on optimal subspace estimator.
result Fundamental result with implications in matrix completion and clustering.
PCA is one of the most widely used dimension reduction techniques. A related easier problem is "subspace learning" or "subspace estimation". Given relatively clean data, both are easily solved via singular value decomposition (SVD). The problem of subspace learning or PCA in the presence of outliers is called robust su…
Study Sp(n)-orbits in complex and Σ-complex subspaces of Hermitian quaternionic vector spaces.
problem Characterize Sp(n)-orbits in Grassmannians of complex and Σ-complex subspaces. method Decompose subspaces into 4-dimensional complex addends and 2-dimensional totally complex subspace. Use properties of isoclinic subspaces and principal angles.
result Determine full set of invariants for Sp(n)-orbits in GrR(2k,4n). In subspace clustering, a group of data points belonging to a union of subspaces are assigned membership to their respective subspaces. This paper presents a new approach dubbed Innovation Pursuit (iPursuit) to the problem of subspace clustering using a new geometrical idea whereby subspaces are identified based on the…
We consider the problem of detecting whether a tensor signal having many missing entities lies within a given low dimensional Kronecker-Structured (KS) subspace. This is a matched subspace detection problem. Tensor matched subspace detection problem is more challenging because of the intertwined signal dimensions. We s…
Paper shows affine constraint is unnecessary for high-dimensional data.
problem The necessity of an affine constraint in affine subspace clustering.
method Theoretical and empirical analysis of conditions for correctness of affine subspace clustering methods.
result Affine constraint has negligible effect on clustering performance for high-dimensional data.
A low-rank transformation learning framework for subspace clustering and classification is here proposed. Many high-dimensional data, such as face images and motion sequences, approximately lie in a union of low-dimensional subspaces. The corresponding subspace clustering problem has been extensively studied in the lit…
This paper investigates the generalization of Principal Component Analysis (PCA) to Riemannian manifolds. We first propose a new and general type of family of subspaces in manifolds that we call barycentric subspaces. They are implicitly defined as the locus of points which are weighted means of k+1 reference points.…
Flow Matching models help generative models stay within the subspace of real data.
problem How do generative models stay within the subspace of real data?
method Flow Matching models using a learned velocity field to transform a simple prior into a complex target distribution.
result Generated samples memorize real data points and represent the sample data subspace exactly.
In this letter, we consider two sets of observations defined as subspace signals embedded in noise and we wish to analyze the distance between these two subspaces. The latter entails evaluating the angles between the subspaces, an issue reminiscent of the well-known Procrustes problem. A Bayesian approach is investigat…
Give deterministic necessary and sufficient conditions to guarantee that if a subspace fits certain partially observed data from a union of subspaces, it is because such data really lies in a subspace. Furthermore, Give deterministic necessary and sufficient conditions to guarantee that if a subspace fits certain parti…
In this paper, we exhibit the tradeoffs between the (training) sample, computation and storage complexity for the problem of supervised classification using signal subspace estimation. Our main tool is the use of tensor subspaces, i.e. subspaces with a Kronecker structure, for embedding the data into lower dimensions. …
A new method for faster optimization in high dimensions.
problem Slow convergence in high-dimensional optimization problems.
method Subspace cubic regularized Newton method within Krylov subspace.
result Achieves a dimension-independent convergence rate of O(1/mk + 1/k^2).
We assume data sampled from a mixture of d-dimensional linear subspaces with spherically symmetric distributions within each subspace and an additional outlier component with spherically symmetric distribution within the ambient space (for simplicity we may assume that all distributions are uniform on their correspondi…
An elliptic theory is constructed for operators acting in subspaces defined via odd pseudodifferential projections. Subspaces of this type arise as Calderon subspaces for first order elliptic differential operators on manifolds with boundary, or as spectral subspaces for self-adjoint elliptic differential operators of …
WPCA improves subspace recovery robustness to outliers.
problem Improving subspace recovery in the presence of outliers.
method Winsorized PCA (WPCA) with theoretical analysis of accuracy and robustness.
result WPCA provides consistent subspace recovery from contaminated data.
New forms of symmetric shift-invariant subspaces found for harmonic maps.
problem Understanding harmonic maps into symmetric and k-symmetric spaces. method Imposing a symmetry condition on shift-invariant subspaces of a Hilbert space.
result Obtained new general forms for symmetric shift-invariant subspaces and extended solutions.
Transforms data into separable subspaces for clustering.
problem Data is not always separable into subspaces.
method Embeds subspace clustering techniques into transform learning.
result Improves upon state-of-the-art clustering techniques.
Stochastic Sparse Subspace Clustering improves subspace clustering by reducing over-segmentation through dropout.
problem Over-segmentation in subspace clustering.
method Introducing dropout regularization to enforce denser connections between points from the same subspace.
result Stochastic Sparse Subspace Clustering effectively handles large datasets and reduces over-segmentation.
Proposes a method to enforce nestedness in subspace learning methods.
problem Consistency between data representations when choosing different subspaces dimensions.
method Lifts Grassmannian optimization criteria to flag manifolds via nested projectors.
result Successfully addresses the nestedness issue in several machine learning methods.
This work presents GROUSE (Grassmanian Rank-One Update Subspace Estimation), an efficient online algorithm for tracking subspaces from highly incomplete observations. GROUSE requires only basic linear algebraic manipulations at each iteration, and each subspace update can be performed in linear time in the dimension of…
DCFSC uses a simple auto-encoder for subspace clustering without parameters.
problem Subspace clustering for large-scale high-dimensional datasets.
method Closed-form shallow auto-encoder for data-driven self-expressive layer, no parameters or optimization.
result Significant memory benefits over existing methods on large datasets.