Proposes Neural Random Subspace for deep learning.
problem Combining random subspace method with deep learning.
method Neural Random Subspace (NRS) integrating deep learning and random subspace.
result Achieves superior performance on 35 datasets and improves recognition tasks.
Improved neural network training in low-dimensional random bases.
problem Inefficient optimization in large-scale neural networks.
method Re-draw random subspace at each training step, apply independent projections to different network parts.
result Significantly better optimization performance and efficiency.
Random projections are able to perform dimension reduction efficiently for datasets with nonlinear low-dimensional structures. One well-known example is that random matrices embed sparse vectors into a low-dimensional subspace nearly isometrically, known as the restricted isometric property in compressed sensing. In th…
Poor approximators found in neural networks and random feature models.
problem Understanding why certain neural networks and models perform poorly in approximating functions.
method Established a scale separation of Kolmogorov width type and applied it to neural networks and random feature models.
result Reproducing kernel Hilbert spaces and two-layer neural networks are poor L2-approximators for certain functions. New algorithm learns principal subspace from random samples.
problem Learning principal subspace from small random submatrices.
method Stochastic gradient descent algorithm for neural networks.
result Algorithm can handle infinite rows and columns.
We develop embeddings for nonlinear subspaces preserving vector norms.
problem Preserving vector norms in nonlinear subspaces.
method Low-distortion embeddings for subspaces under nonlinear transformations.
result First low-distortion embeddings for a wide class of nonlinear functions.
Deep ensembles improve model accuracy and robustness, but their theoretical underpinnings are not fully understood.
problem Understanding why deep ensembles work well in practice despite theoretical limitations.
method Investigating the loss landscape of neural networks and exploring the diversity of functions in function space.
result Random initializations explore diverse modes in function space, while ensembles along an optimization trajectory cluster within a single mode.
A new method for anomaly detection using random subspaces and Gaussian mixture models.
problem Anomaly detection in high-dimensional data.
method Statistical estimation of probability density using random subspaces combined with geometric averaging.
result The method achieves competitive AUC scores and is interpretable.
RaSE ensemble framework improves sparse classification accuracy.
problem Sparse classification challenges in high-dimensional data.
method Random Subspace Ensemble (RaSE) framework with subspace selection via RIC.
result RaSE achieves low misclassification rates and accurate feature ranking.
RS-NSGD improves SGD convergence for heavy-tailed noise.
problem Nonconvex optimization with heavy-tailed noise.
method Integrates direction normalization into subspace updates.
result Achieves better oracle complexity than full-dimensional normalized SGD.
Spectral Method is a commonly used scheme to cluster data points lying close to Union of Subspaces by first constructing a Random Geometry Graph, called Subspace Clustering. This paper establishes a theory to analyze this method. Based on this theory, we demonstrate the efficiency of Subspace Clustering in fairly broad…
One technique to visualize the training of neural networks is to perform PCA on the parameters over the course of training and to project to the subspace spanned by the first few PCA components. In this paper we compare this technique to the PCA of a high dimensional random walk. We compute the eigenvalues and eigenvec…
Study on using random subspaces for ERM with various loss functions.
problem Improving learning accuracy with computational savings from random subspaces.
method Random subspaces of a hypothesis space, considering data-dependent subspaces.
result Unified analysis showing computational efficiency can be improved without performance loss.
RaSE screens variables via random subspaces, identifying joint effects.
problem Missing joint effects of predictors in ultra-high dimensional data.
method Random Subspace Ensemble (RaSE) framework combining subspace evaluation criteria.
result RaSE identifies signals with no marginal effect or high-order interactions.
Union of Subspaces (UoS) is a popular model to describe the underlying low-dimensional structure of data. The fine details of UoS structure can be described in terms of canonical angles (also known as principal angles) between subspaces, which is a well-known characterization for relative subspace positions. In this pa…
Develops accelerated methods for optimization using low-dimensional projected-gradient information.
problem Optimization with low-dimensional projected-gradient information and Nesterov acceleration.
method Randomized-subspace Nesterov accelerated gradient methods for smooth convex and strongly convex optimization.
result Established accelerated oracle-complexity guarantees and unified basis for comparing sketch families.
Study analyzes perturbations in singular subspaces under random noise.
problem Understanding singular vector and subspace changes in signal-plus-noise models.
method Generalized Davis-Kahan-Wedin theorem for any unitarily invariant norm, considering ℓ∞ and ℓ2,∞ bounds. result Fine-grained insights into singular vector and subspace perturbations, including ℓ∞ and ℓ2,∞ bounds. With increasing concerns about security, the need for highly secure physical biometrics-based authentication systems utilizing \emph{cancelable biometric} technologies is on the rise. Because the problem of cancelable template generation deals with the trade-off between template security and matching performance, many …
Fewer degrees of freedom can train deep networks, showing a sharp phase transition.
problem Training deep networks with fewer degrees of freedom than parameters.
method Examined success probability of hitting training loss sub-level sets within random subspaces.
result Threshold training dimension increases as desired final loss decreases.
KSS method converges and recovers correct clustering under certain conditions.
problem Subspace clustering for semi-randomly sampled data.
method Local convergence analysis and recovery guarantee for KSS method.
result KSS method converges superlinearly and finds correct clustering within loglog N iterations.
Optimal subspace embedding with near-optimal sparsity for high-dimensional data.
problem Efficiently preserving norms of vectors in high-dimensional subspaces.
method Near-optimal sparsity oblivious subspace embedding with decoupling argument and cumulant method.
result Achieved near-optimal sparsity of O~(1/ε) non-zeros per column. New faster, space-saving methods for subspace embeddings in tensors.
problem Efficiently embedding large tensors with fewer random bits.
method Modewise Johnson-Lindenstrauss embeddings for rank-r tensors. result Improved space complexity for tensor subspaces with fewer random bits.
New Krylov subspace methods speed up mixed-effects models with crossed random effects.
problem Slow computations for high-dimensional crossed random effects in mixed-effects models.
method Krylov subspace-based methods for generalized mixed-effects models with cross effects.
result Speedups by factors of up to 10,000 in computations for mixed-effects models.
This paper improves Koopman operator approximations by pruning subspaces in RKHS.
problem Improving predictive accuracy of Koopman operator approximations.
method Computes principal angles and vectors in RKHS to prune subspaces.
result Validated approach enhances Koopman operator approximations for large datasets.
Two types of nonidentifiability in latent position graphs identified and characterized.
problem Identifying and characterizing nonidentifiability in latent position random graph models.
method Defined and examined subspace nonidentifiability and model-based nonidentifiability, providing examples and characterizing limits.
result Characterized the limits of model-based nonidentifiability and obtained additional limiting results for specific graph models.
Although the convolutional neural networks (CNNs) have become popular for various image processing and computer vision task recently, it remains a challenging problem to reduce the storage cost of the parameters for resource-limited platforms. In the previous studies, tensor decomposition (TD) has achieved promising co…
Kernel methods obtain superb performance in terms of accuracy for various machine learning tasks since they can effectively extract nonlinear relations. However, their time complexity can be rather large especially for clustering tasks. In this paper we define a general class of kernels that can be easily approximated …
Subspace clustering refers to the problem of clustering unlabeled high-dimensional data points into a union of low-dimensional linear subspaces, assumed unknown. In practice one may have access to dimensionality-reduced observations of the data only, resulting, e.g., from "undersampling" due to complexity and speed con…
This paper explores and analyzes two randomized designs for robust Principal Component Analysis (PCA) employing low-dimensional data sketching. In one design, a data sketch is constructed using random column sampling followed by low dimensional embedding, while in the other, sketching is based on random column and row …
A new method for faster optimization in high dimensions.
problem Slow convergence in high-dimensional optimization problems.
method Subspace cubic regularized Newton method within Krylov subspace.
result Achieves a dimension-independent convergence rate of O(1/mk + 1/k^2).
New method explains GNNs using power iteration clustering.
problem Mysterious mechanism of message passing in GNNs.
method Subspace power iteration clustering (SPIC) models.
result Message passing in GNNs can be understood through power iteration.
Neural networks simplify SDR in regression tasks.
problem Sufficient dimension reduction in regression problems.
method Applying neural networks with rank regularization to estimate the central mean subspace.
result Neural networks effectively perform SDR, consistent with theoretical estimations.
A method for identifying joint and individual subspaces from multi-view data.
problem Unclear conditions for reliably identifying joint and individual subspaces from noisy, high-dimensional measurements.
method Rigorously quantifies conditions based on signal rank, principal angles, and noise levels. Characterizes spectrum perturbations of product of projection matrices.
result Estimates joint and individual subspaces more accurately than existing approaches in simulations and real-world applications.
Paper analyzes neural networks using active subspace for structural analysis and vulnerability, reducing model size and improving attacks.
problem Analyzing and reducing the complexity of neural networks.
method Active subspace method for measuring active neurons, network structure modification, and additive universal adversarial attack vector.
result ASNet achieves significant parameter and flops reduction, and improves universal adversarial attack performance.
Feature engineering plays an important role in the success of a machine learning model. Most of the effort in training a model goes into data preparation and choosing the right representation. In this paper, we propose a robust feature engineering method, Randomized Union of Locally Linear Subspaces (RULLS). We generat…
Paper improves ℓ0-SSC for noisy data by proving SDP and proposing Noisy-DR-ℓ0-SSC.
problem Noisy data and less restrictive subspace affinity in sparse subspace clustering.
method Proposes Noisy-DR-ℓ0-SSC, which projects data onto a lower dimensional space and then applies noisy ℓ0-SSC. result Theoretical guarantee on the correctness of noisy ℓ0-SSC in terms of SDP on noisy data. High-dimensional data often lie in low-dimensional subspaces corresponding to different classes they belong to. Finding sparse representations of data points in a dictionary built using the collection of data helps to uncover low-dimensional subspaces and address problems such as clustering, classification, subset sele…
This paper considers the problem of subspace clustering under noise. Specifically, we study the behavior of Sparse Subspace Clustering (SSC) when either adversarial or random noise is added to the unlabelled input data points, which are assumed to be in a union of low-dimensional subspaces. We show that a modified vers…
Develops a new feature theory for robust machine learning.
problem Creating robust machine learning features from training data.
method Stochastic tensor space feature theory with Karhunen-Loeve expansion and hierarchical subspaces.
result Dramatic increases in accuracy for predicting Alzheimer's disease stages.
Model explains surprising properties of neural network training landscapes.
problem Understanding the geometry of neural network loss landscapes.
method Developed a simple theoretical model of gradients and Hessians.
result Unified model accounts for 4 surprising properties of neural loss landscapes.
Paper introduces S-SSE for stable sparse subspace embedding.
problem Inefficient sparse random projection matrices with uneven non-zero distribution.
method Uses uniform sampling without replacement to create a stable sparse subspace embedded matrix (S-SSE).
result S-SSE maintains Euclidean distance better after dimension reduction.
New bounds improve neural network generalization through slicing.
problem Difficulty in evaluating mutual information in high dimensions for neural networks.
method Slicing the parameter space and using disintegrated mutual information and k-sliced mutual information.
result Slicing improves generalization and offers significant computational and statistical advantages.
We introduce and develop a novel approach to outlier detection based on adaptation of random subspace learning. Our proposed method handles both high-dimension low-sample size and traditional low-dimensional high-sample size datasets. Essentially, we avoid the computational bottleneck of techniques like minimum covaria…
SGD updates align with a low-rank subspace but do not lead to further loss reduction.
problem Understanding the training dynamics of deep neural networks, particularly the role of the dominant subspace.
method Exploring whether neural networks can be trained within the dominant subspace of the loss Hessian.
result SGD updates, when projected onto the dominant subspace, do not decrease the training loss further, suggesting spurious alignment.
BAxUS optimizes high-dimensional functions adaptively, avoiding performance degradation and failure.
problem State-of-the-art HDBO methods degrade or fail with increasing dimensions.
method BAxUS uses nested random subspaces to adaptively optimize high-dimensional functions.
result BAxUS outperforms state-of-the-art methods across various applications.
In this letter, we consider two sets of observations defined as subspace signals embedded in noise and we wish to analyze the distance between these two subspaces. The latter entails evaluating the angles between the subspaces, an issue reminiscent of the well-known Procrustes problem. A Bayesian approach is investigat…
The nowadays massive amounts of generated and communicated data present major challenges in their processing. While capable of successfully classifying nonlinearly separable objects in various settings, subspace clustering (SC) methods incur prohibitively high computational complexity when processing large-scale data. …
This work simplifies Bayesian inference for neural networks by identifying influential parameter directions.
problem High computational complexity in Bayesian inference for neural networks due to high-dimensional parameter space.
method Constructing an active subspace of influential parameter directions to reduce dimensionality.
result Effective and scalable Bayesian inference achieved via reduced active subspace.