A new method for faster optimization in high dimensions.
problem Slow convergence in high-dimensional optimization problems.
method Subspace cubic regularized Newton method within Krylov subspace.
result Achieves a dimension-independent convergence rate of O(1/mk + 1/k^2).
Optimal subspace embedding with near-optimal sparsity for high-dimensional data.
problem Efficiently preserving norms of vectors in high-dimensional subspaces.
method Near-optimal sparsity oblivious subspace embedding with decoupling argument and cumulant method.
result Achieved near-optimal sparsity of O~(1/ε) non-zeros per column. New method solves saddle-point problems faster than existing methods.
problem Large-scale saddle-point problems in optimization.
method Sequential subspace optimization with proximal regularization.
result Significantly better convergence compared to first-order methods.
This paper investigates the generalization of Principal Component Analysis (PCA) to Riemannian manifolds. We first propose a new and general type of family of subspaces in manifolds that we call barycentric subspaces. They are implicitly defined as the locus of points which are weighted means of k+1 reference points.…
AdaSub optimizes with second-order info in low-dims subspace.
problem Efficiently use second-order optimization methods with low computational cost.
method Adaptive subspace selection for second-order optimization.
result AdaSub outperforms other stochastic optimizers in time and iterations.
Paper bounds subspace estimator error from noisy projections.
problem Estimating subspaces from noisy data.
method Derives perturbation bound on optimal subspace estimator.
result Fundamental result with implications in matrix completion and clustering.
Proposes a method to enforce nestedness in subspace learning methods.
problem Consistency between data representations when choosing different subspaces dimensions.
method Lifts Grassmannian optimization criteria to flag manifolds via nested projectors.
result Successfully addresses the nestedness issue in several machine learning methods.
BO method identifies sparse subspaces for efficient high-dimensional optimization.
problem Efficient optimization of high-dimensional black-box functions.
method Sparse Gaussian process surrogate models on axis-aligned subspaces with Hamiltonian Monte Carlo inference.
result SAASBO achieves excellent performance on synthetic and real-world problems.
This letter presents a new spectral-clustering-based approach to the subspace clustering problem. Underpinning the proposed method is a convex program for optimal direction search, which for each data point d finds an optimal direction in the span of the data that has minimum projection on the other data points and non…
Fast robust subspace tracking in sparse data-dependent noise with near-optimal delay.
problem Robustly tracking time-varying subspaces in the presence of sparse outliers.
method Introduces a fast mini-batch robust ST solution under mild assumptions.
result Provably correct subspace tracking with near-optimal delay and same time complexity as simple PCA.
Extends subspace detour method to Gromov-Wasserstein problem.
problem Matching shapes using Gromov-Wasserstein distance.
method Project measures onto a subspace, then compute optimal transport plan.
result Connections with Knothe-Rosenblatt rearrangement.
Algorithm finds a subspace minimizing distances to inliers with outliers.
problem Finding a k-dimensional subspace minimizing distances to inliers with outliers. method Extends dimension reduction techniques and bi-criteria approximations based on sampling.
result Efficient algorithm for multiplicative (1+ε)-approximation of optimal solution. A new method for one-class classification using ellipsoidal encapsulation.
problem One-class classification for data optimization.
method Iterative transformation into an optimized subspace with regularization terms.
result Better results in one-class classification compared to existing methods.
We study sparse principal components analysis in high dimensions, where p (the number of variables) can be much larger than n (the number of observations), and analyze the problem of estimating the subspace spanned by the principal eigenvectors of the population covariance matrix. We introduce two complementary not…
POTD estimates SDR subspace using optimal transport for binary response.
problem Insufficient performance of existing SDR methods for categorical responses.
method Principal optimal transport direction (POTD) using optimal transport coupling.
result POTD exclusively estimates SDR subspace for error-free class labels.
Computing optimal transport (OT) between measures in high dimensions is doomed by the curse of dimensionality. A popular approach to avoid this curse is to project input measures on lower-dimensional subspaces (1D lines in the case of sliced Wasserstein distances), solve the OT problem between these reduced measures, a…
Stochastic Sparse Subspace Clustering improves subspace clustering by reducing over-segmentation through dropout.
problem Over-segmentation in subspace clustering.
method Introducing dropout regularization to enforce denser connections between points from the same subspace.
result Stochastic Sparse Subspace Clustering effectively handles large datasets and reduces over-segmentation.
A new method tackles bilevel optimization using Lanczos process for efficient hyper-gradient computation.
problem Efficiently solving large-scale bilevel optimization problems with gradient-based methods.
method Constructing low-dimensional approximate Krylov subspaces with the Lanczos process to approximate the Hessian inverse vector product.
result Demonstrates a O(ε−1) convergence rate and efficiency in synthetic and deep learning tasks. Paper extends multivariate rank tests for robust subspace detection.
problem Testing distributional similarity in multivariate data.
method Soft and subspace robust multivariate rank tests based on entropy regularized optimal transport.
result Trade-off between detection power and false alarm rate via projections.
LDAdam optimizes large models with low memory by adapting to lower-dimensional subspaces.
problem Training large models efficiently and accurately.
method Adaptive optimization in lower-dimensional subspaces with a new projection-aware update rule and error feedback mechanism.
result LDAdam achieves accurate and efficient training of language models.
Develops accelerated methods for optimization using low-dimensional projected-gradient information.
problem Optimization with low-dimensional projected-gradient information and Nesterov acceleration.
method Randomized-subspace Nesterov accelerated gradient methods for smooth convex and strongly convex optimization.
result Established accelerated oracle-complexity guarantees and unified basis for comparing sketch families.
Bayesian optimization guided by experimenter intuition and beliefs.
problem Finding optimal functions with experimenter's beliefs incorporated.
method Sequential Subspace Search using Gaussian Process.
result Algorithm converges in sub-linear time with finite effective dimension.
Optimal smooth subspaces approximate large data sets efficiently.
problem Approximating large data sets with invariant subspaces.
method Smooth functions under lattice translations or crystallographic groups, with optimal selection of Paley-Wiener space.
result Optimal lattice selection enhances approximation efficiency.
Study optimizes shared singular subspace estimation from noisy matrices.
problem Estimating shared singular subspaces across multiple noisy matrices.
method Low-rank matrix denoising framework with Stack-SVD and novel estimators.
result Stack-SVD achieves minimax rate-optimality for identical shared subspaces, and novel estimators for partial sharing.
Due to the lack of enough generalization in the state-space, common methods in Reinforcement Learning (RL) suffer from slow learning speed especially in the early learning trials. This paper introduces a model-based method in discrete state-spaces for increasing learning speed in terms of required experience (but not r…
RS-NSGD improves SGD convergence for heavy-tailed noise.
problem Nonconvex optimization with heavy-tailed noise.
method Integrates direction normalization into subspace updates.
result Achieves better oracle complexity than full-dimensional normalized SGD.
In this paper, we study the adversarial robustness of subspace learning problems. Different from the assumptions made in existing work on robust subspace learning where data samples are contaminated by gross sparse outliers or small dense noises, we consider a more powerful adversary who can first observe the data matr…
Subspace segmentation or subspace learning is a challenging and complicated task in machine learning. This paper builds a primary frame and solid theoretical bases for the minimal subspace segmentation (MSS) of finite samples. Existence and conditional uniqueness of MSS are discussed with conditions generally satisfied…
The problem of finding the sparsest vector (direction) in a low dimensional subspace can be considered as a homogeneous variant of the sparse recovery problem, which finds applications in robust subspace recovery, dictionary learning, sparse blind deconvolution, and many other problems in signal processing and machine …
Paper improves ℓ0-SSC for noisy data by proving SDP and proposing Noisy-DR-ℓ0-SSC.
problem Noisy data and less restrictive subspace affinity in sparse subspace clustering.
method Proposes Noisy-DR-ℓ0-SSC, which projects data onto a lower dimensional space and then applies noisy ℓ0-SSC. result Theoretical guarantee on the correctness of noisy ℓ0-SSC in terms of SDP on noisy data. Driven by a wide range of applications, many principal subspace estimation problems have been studied individually under different structural constraints. This paper presents a unified framework for the statistical analysis of a general structured principal subspace estimation problem which includes as special cases no…
This paper will serve as an introduction to the body of work on robust subspace recovery. Robust subspace recovery involves finding an underlying low-dimensional subspace in a dataset that is possibly corrupted with outliers. While this problem is easy to state, it has been difficult to develop optimal algorithms due t…
Kernel methods linked to feature subspaces and maximal correlation kernels.
problem Understanding kernel methods and their relationship to feature extraction.
method Established a correspondence between feature subspaces and kernels, introduced maximal correlation kernels, and demonstrated their optimality.
result Kernel SVM on maximal correlation kernel achieves minimum prediction error.
PCA adapted for curved spaces improves data analysis.
problem PCA's limitations in curved spaces.
method Space Form PCA (SFPCA) for Riemannian manifolds.
result SFPCA provides faster and more accurate subspaces estimation.
New MCMC algorithm reduces subset selection passes to 2 for optimal k-dimensional subspace approximation.
problem Subset selection for k-dimensional subspace approximation with ε-approximation. method MCMC sampling algorithm reducing passes to 2 for p=2 case, poly(k/ε) size subset. result Subset selection of nearly optimal size in 2 passes, (1+ε) approximation. Proposes an optimization framework for sparse robust subspace estimation.
problem Sparse robust one-dimensional subspace estimation.
method l1-norm regularization, linear relaxation, simple ratios, sorting techniques.
result Achieves global optimality for sparse robust subspace with polynomial time efficiency.
GTBO uses group testing to optimize high-dimensional functions efficiently.
problem Optimizing expensive, high-dimensional functions with limited data.
method Group testing to identify active dimensions, then guide optimization.
result GTBO outperforms state-of-the-art methods on high-dimensional benchmarks.
Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
problem Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
method Analyzing the behavior of memory-efficient optimizers like GaLore, which project gradients onto a rank-r subspace recomputed every T steps.
result Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
PED-ANOVA efficiently calculates HP importance in arbitrary subspaces.
problem Understanding the role of different hyperparameters in arbitrary subspaces.
method Derive a novel f-ANOVA formulation for arbitrary subspaces and use Pearson divergence (PED) for a closed-form calculation of HP importance.
result Demonstrates successful identification of important HPs in different subspaces.
New method fuses audio and magnetic data to identify underlying subspaces.
problem Identifying complex trends in multi-modality data.
method Robust Group Subspace Recovery (RoGSuRe) algorithm based on group sparsity and bi-sparsity pursuit.
result Competitive performance in clustering and classification of multi-modal data.
Extends active subspace analysis to infinite dimensions.
problem Dimension reduction in infinite dimensional functionals.
method Defines an operator for Hilbert space, extends Euclidean properties, proposes Monte Carlo procedure.
result Desirable properties extend to infinite dimensional setting.
In this work, we study the robust subspace tracking (RST) problem and obtain one of the first two provable guarantees for it. The goal of RST is to track sequentially arriving data vectors that lie in a slowly changing low-dimensional subspace, while being robust to corruption by additive sparse outliers. It can also b…
We describe ways to define and calculate L1-norm signal subspaces which are less sensitive to outlying data than L2-calculated subspaces. We focus on the computation of the L1 maximum-projection principal component of a data matrix containing N signal samples of dimension D and conclude that the general proble…
We propose Deep Closed-Form Subspace Clustering (DCFSC), a new embarrassingly simple model for subspace clustering with learning non-linear mapping. Compared with the previous deep subspace clustering (DSC) techniques, our DCFSC does not have any parameters at all for the self-expressive layer. Instead, DCFSC utilizes …
Paper proves IRLS converges to subspace from any start, with practical benefits.
problem Robust subspace estimation in machine learning.
method Iteratively Reweighted Least Squares (IRLS) with dynamic smoothing regularization.
result IRLS converges linearly to the underlying subspace from any initialization under deterministic conditions.
CobBO optimizes expensive functions in high dimensions by using a two-stage kernel approach.
problem Bayesian optimization struggles in high dimensions due to computational inefficiency.
method Coordinate backoff Bayesian Optimization with two-stage kernels.
result CobBO finds solutions comparable to or better than other methods in high dimensions.
Flow Matching models help generative models stay within the subspace of real data.
problem How do generative models stay within the subspace of real data?
method Flow Matching models using a learned velocity field to transform a simple prior into a complex target distribution.
result Generated samples memorize real data points and represent the sample data subspace exactly.
In subspace clustering, a group of data points belonging to a union of subspaces are assigned membership to their respective subspaces. This paper presents a new approach dubbed Innovation Pursuit (iPursuit) to the problem of subspace clustering using a new geometrical idea whereby subspaces are identified based on the…