A new method learns meaningful distances between samples using optimal transport.
problem Learning meaningful distances between samples in datasets without labeled data.
method Computes OT distances between samples and features using singular vectors of a function mapping ground metrics to OT distances.
result Wasserstein Singular Vectors provide a scalable solution for unsupervised ground metric learning.
Improved Frank-Wolfe algorithm solves convex trace-norm ball problems.
problem Optimizing convex functions over trace-norm balls.
method Rank-k variant of Frank-Wolfe algorithm using top-k singular-vector computation.
result Linear convergence rate for smooth and strongly convex objectives with rank-limited solutions.
We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine transform (allowing comparison between different layers and networks) and fast to compute (allowing more comparisons to be calculated than with p…
Study on overlaps of singular vectors in Gaussian matrix submatrices.
problem Analyzing overlaps of singular vectors in submatrices of Gaussian matrices.
method Utilizes dynamics of singular vectors and specific resolvents for Brownian trajectories.
result Explicit forms for limiting rescaled mean squared overlaps in the bulk of spectra.
Paper develops a bootstrap method for estimating sketched SVD errors.
problem Lack of tools for accurately estimating sketched SVD errors.
method Develops a fully data-driven bootstrap method for numerical error estimation.
result Allows users to adaptively predict extra work needed for desired error tolerance.
A new method reduces task interference in model merging.
problem Model merging overlooks structural information and is susceptible to task interference.
method Task Singular Vectors (TSV) and TSV-Compress for compression and interference reduction.
result TSV-Merge significantly outperforms existing methods.
Generically, the set of points along which two non-singular vector fields on the three-sphere are positively (resp. negatively) collinear form a link. We prove that the two vector fields are homotopic if and only if the linking number of those links is zero. We use this criterion to give a new proof of a result of Yano…
Study analyzes perturbations in singular subspaces under random noise.
problem Understanding singular vector and subspace changes in signal-plus-noise models.
method Generalized Davis-Kahan-Wedin theorem for any unitarily invariant norm, considering ℓ∞ and ℓ2,∞ bounds. result Fine-grained insights into singular vector and subspace perturbations, including ℓ∞ and ℓ2,∞ bounds. Spectral embedding based on the Singular Value Decomposition (SVD) is a widely used "preprocessing" step in many learning tasks, typically leading to dimensionality reduction by projecting onto a number of dominant singular vectors and rescaling the coordinate axes (by a predefined function of the singular value). Howe…
Proposes a regularization method for unsupervised domain adaptation that aligns predictions with target data's top singular vectors.
problem Domain adaptation challenges in high joint error scenarios.
method Regularizes classifier to align with unsupervised target data guided by label alignment property (LAP).
result The method improves performance in MNIST-USPS domain adaptation and cross-lingual sentiment analysis.
Paper studies tensor models using random matrix theory.
problem Analyzing asymmetric order-d spiked tensor models with Gaussian noise.
method Uses variational definition of singular vectors and values, constructs equivalent spiked symmetric block-wise random matrix from tensor contractions.
result Characterizes asymptotic singular values and alignments of singular vectors with true spike components.
Extends Serre-Swan theorem to all finitely generated modules over smooth functions.
problem Classical Serre-Swan theorem limitations.
method Introduces tepui fibrations and singular vector bundles.
result Realizes all finitely generated modules over smooth functions.
Improves community detection in directed networks with theoretical guarantees.
problem Degree heterogeneity affects community detection in directed networks.
method Introduced D-SCORE algorithm and established theoretical guarantees for Directed-DCBM.
result Established theoretical guarantees and provided improvements for D-SCORE.
We classify and explicitly describe homomorphisms of Verma modules for conformal Galilei algebras cgaℓ(d,C) with d=1 for any integer value ℓ∈N. The homomorphisms are uniquely determined by singular vectors as solutions of certain differential operators of flag type, and id…
The paper proposes a new model to analyze directed networks and accurately estimate community memberships.
problem Modeling and estimating community memberships in directed networks with heterogeneous degrees.
method Directed Degree Corrected Mixed Membership (DiDCMM) model and DiMSC algorithm.
result The proposed DiMSC algorithm is asymptotically consistent and provides error bounds for community membership vectors.
The paper studies phase transitions in random matrices and tensor unfolding for detecting signals.
problem Phase transitions in singular values and vectors of large random matrices.
method Analysis of singular values and vectors of long rectangular random matrices, and tensor unfolding algorithm for asymmetric rank-one spiked tensor models.
result An exact threshold for tensor unfolding to detect signals, independent of unfolding procedure.
The paper improves Kaczmarz algorithm with momentum for linear least squares.
problem Improving convergence of the Kaczmarz algorithm for linear least squares.
method Integrates geometrically smoothed momentum into the randomized Kaczmarz algorithm.
result Proves expected error reduction in singular vector directions.
The paper identifies redundant columns in matrices for feature selection and clustering.
problem Identifying redundant columns in matrices for feature selection and clustering.
method Proves that after re-ordering columns, a matrix can be block-diagonalized revealing linearly dependent columns.
result Identifies redundant columns in matrices, aiding in feature selection and clustering.
We study the convergence properties of the VR-PCA algorithm introduced by \cite{shamir2015stochastic} for fast computation of leading singular vectors. We prove several new results, including a formal analysis of a block version of the algorithm, and convergence from random initialization. We also make a few observatio…
SVD training reduces DNN rank and computation load without SVD per step.
problem High memory and computational load in deep neural networks.
method Explicitly achieves low-rank DNNs during training without SVD per step, using orthogonality regularization and sparsity-inducing regularizers.
result Significantly reduces DNN rank and computation load compared to existing methods.
Based on the Lie theoretical methods of algebraic Fourier transformation, we classify in the case of generic values of inducing parameters the scalar singular vectors corresponding to the diagonal branching rules for scalar generalized Verma modules in the case of orthogonal Lie algebra and its conformal parabolic suba…
Gradient descent in deep networks tends to find flat minima, which are nearly balanced.
problem Understanding the effect of gradient descent on the structure of minima in deep neural networks.
method Characterized flat minima in linear neural networks trained with a quadratic loss.
result Flat minima correspond to nearly balanced networks where the gain from input to intermediate representations is nearly constant.
We solve matrix denoising with both row and column correlations, setting limits and designing optimal methods.
problem Matrix denoising with doubly heteroscedastic noise (both row and column correlations).
method Established information-theoretic and algorithmic limits, designed a novel spectral estimator with optimality guarantees.
result The novel spectral estimator achieves positive correlation with the signal and Bayes-optimal error under one-sided heteroscedasticity.
Study one-sided matrix completion with two observations per row.
problem Recover right singular vectors of a low-rank matrix X with few observations. method Impute missing values of XTX and analyze recovery guarantees. result Provable recovery of XTX with Ω(r2dlogd) rows, outperforming standard methods. The paper analyzes how random perturbations affect RSVD and its applications.
problem Analyzing the impact of random perturbations on RSVD.
method Derives bounds for distances between exact and approximated singular vectors using RSVD.
result Established nearly-optimal convergence rates and asymptotic normality for RSVD in various inference problems.
Efficient SVD algorithm robust to outliers.
problem Outliers in data matrix affect SVD accuracy and speed.
method Spherically Normalized SVD (SpherSVD) algorithm.
result Significantly faster and more robust than existing methods.
Paper develops inference methods for low-rank tensors without debiasing.
problem Statistical inference for low-rank tensor models.
method Two-iteration alternating minimization for asymptotic distribution.
result Asymptotic distributions and confidence regions for singular subspaces.
Given a real matrix A with n columns, the problem is to approximate the Gram product AA^T by c << n weighted outer products of columns of A. Necessary and sufficient conditions for the exact computation of AA^T (in exact arithmetic) from c >= rank(A) columns depend on the right singular vector matrix of A. For a Monte-…
Random matrix ensembles yield uniform distributions on manifolds.
problem Understanding distributions of vectors in random matrix ensembles.
method Analyzing eigenvalues, singular values, and Autonne-Takagi vectors of various random matrix ensembles.
result Uniform distributions on specific manifolds for different types of random matrix ensembles.
The paper analyzes L2-regularized linear autoencoders and their loss landscapes.
problem Understanding the loss landscapes of L2-regularized linear autoencoders. method Smoothly parameterizing the critical manifold and relating minima to the MAP estimate of probabilistic PCA.
result Proves that L2-regularized LAEs learn principal directions as left singular vectors of the decoder. We prove a Poincare lemma for a set of r smooth functions on a 2n-dimensional smooth manifold satisfying a commutation relation determined by r singular vector fields associated to a Cartan subalgebra of sp(2r,R). This result has a natural interpretation in terms of the cohomology associated to the inf…
The contractive auto-encoder learns a representation of the input data that captures the local manifold structure around each data point, through the leading singular vectors of the Jacobian of the transformation from input to representation. The corresponding singular values specify how much local variation is plausib…
LoRA fine-tuning creates intruder dimensions that can cause forgetting, and a new law predicts when this happens.
problem Predicting when LoRA fine-tuning creates intruder dimensions that can cause catastrophic forgetting.
method Derived a per-layer critical update strength s∗ and an exact secular-equation characterization of the updated spectrum. result The law localizes the empirical threshold within a factor of two on 82% of layers and separates intruder-bearing from intruder-free layers at deployment.
New tensor formulation reveals gradient flow's bias in linear neural networks.
problem Understanding implicit bias in linear neural network training.
method Tensor formulation of neural networks, including fully-connected, diagonal, and convolutional networks.
result Gradient flow on linear tensor networks converges to solutions of specific optimization problems.
Study analyzes accuracy of tensor deflation in noisy conditions.
problem Analyzing accuracy of tensor deflation in noisy conditions.
method Asymptotic study of Hotelling-type tensor deflation in large tensor dimensions.
result Characterization of estimated singular values and singular vector alignments.
We generalize Turaev's definition of torsion invariants of pairs (M,ξ), where M is a 3-dimensional manifold and ξ is an Euler structure on M (a non-singular vector field up to homotopy relative to the boundary of M and local modifications in the interior of M). Namely, we allow M to have arbitrary boundar…
PLS-SVD struggles with missing data in multimodal datasets, showing a phase transition in performance.
problem Missing data in PLS-SVD for multimodal datasets.
method Replica-symmetric analysis of spiked rectangular random matrices with missing entries.
result PLS-SVD performance transitions from uninformative to informative singular vectors at a critical signal-to-noise threshold.
In this article parametric versions of Wilson's plug and Kuperberg's plug are discussed. We show that there is a weak homotopy equivalence induced by the inclusion between the space of non-singular vector fields tangent to a foliation and the subspace of those without closed orbits, as long as the leaves of the foliati…
Paper optimizes sparse feature selection for cancer detection using GSVP and SVM.
problem Sparse feature selection for cancer detection.
method Regularized GSVP with proximal gradient descent, feature selection via SVM.
result Near-perfect balanced accuracy with few selected features.
Ranky solves SVD for large sparse matrices in distributed systems.
problem Rank problem in large sparse matrices for SVD.
method Distributed approach to solve rank problem.
result Recovers SVD with negligible error for large sparse matrices.
In this paper the exact linear relation between the leading eigenvectors of the modularity matrix and the singular vectors of an uncentered data matrix is developed. Based on this analysis the concept of a modularity component is defined, and its properties are developed. It is shown that modularity component analysis …
Optimal estimation of low-rank matrices from contaminated data.
problem Reconstructing a low-rank matrix from a contaminated version of itself.
method Developed an asymptotically optimal algorithm to estimate the original matrix from the singular values of the contaminated matrix.
result Found an explicit signal-to-noise cutoff below which estimation fails.
We extend Turaev's definition of torsion invariants of 3-dimensional manifolds equipped with non-singular vector fields, by allowing (suitable) tangency circles to the boundary, and manifolds with non-zero Euler characteristic. We show that these invariants apply in particular to (the exterior of) Legendrian links in c…
This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models---including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation---which exploits a certain tensor structure in their low-order observable moments (typically…
Cluster Quilting clusters fragmented data sets for neuroscience and genomics.
problem Clustering fragmented data sets in neuroscience and genomics.
method Cluster Quilting method using patch ordering, patchwise SVD, sequential linear mapping, and k-means.
result Cluster Quilting discovers more accurate clusters than other methods.
The paper updates SVD of evolving matrices using projection techniques.
problem Updating the rank-k truncated SVD of evolving matrices.
method Projection viewpoint, building subspaces to approximate singular vectors.
result The proposed algorithm leads to higher accuracy, especially for large singular values.
The branching problem for a couple of non-compatible Lie algebras and their parabolic subalgebras applied to generalized Verma modules was recently discussed in \cite{ms}. In the present article, we employ the recently developed F-method, \cite{KOSS1}, \cite{KOSS2} to the couple of non-compatible Lie algebras $({\LieGt…
A new framework for dimension reduction using ensemble of random projections.
problem High-dimensional regression problems with limited data.
method Aggregating an ensemble of carefully chosen random projections, retaining based on empirical performance, and selecting singular vectors.
result The proposed method stabilizes error as the number of projection groups increases.