SCOPE estimator improves covariance and precision matrix estimation.
problem Estimating covariance and precision matrices accurately.
method Distributionally robust optimization with convex spectral divergence.
result SCOPE estimator reduces spectral bias and improves condition number.
High-dimensional inference for sparse spectral precision matrices
problem Inference on the spectral precision matrix at a fixed frequency
method Full likelihood-based inference using neighboring discrete Fourier transforms
result Simultaneous control of regularization, finite-sample truncation, and smoothing biases
The inverse covariance matrix provides considerable insight for understanding statistical models in the multivariate setting. In particular, when the distribution over variables is assumed to be multivariate normal, the sparsity pattern in the inverse covariance matrix, commonly referred to as the precision matrix, cor…
New method preserves spectral clustering performance under aggressive sparsification and quantization.
problem Maintaining spectral clustering performance with sparse and quantized data.
method Random matrix theory applied to eigenspectrum changes under sparsification and quantization.
result Spectral clustering performance is preserved even with aggressive sparsification and quantization.
Spectral clustering performance depends on eigenvector fluctuations, shown to be Gaussian.
problem Predicting the performance of spectral clustering.
method General spike random matrix model and rotational invariance of noise.
result Fluctuations of eigenvector entries are Gaussian in large-dimensional regime.
PAC-Bayesian matrix completion with a spectral scaled Student prior offers efficient inference.
problem Matrix completion with underlying low-rank structure.
method Spectral scaled Student prior and PAC-Bayesian bounds.
result Minimax-optimal oracle inequality for model misspecification and general sampling distribution.
Optimal spectral method found for inhomogeneous spiked Wigner model.
problem Structured noise in learning scenarios.
method Random matrix theory and spectral analysis.
result Optimal threshold for phase transition in block-structured Wigner model.
Many modern statistical applications ask for the estimation of a covariance (or precision) matrix in settings where the number of variables is larger than the number of observations. There exists a broad class of ridge-type estimators that employs regularization to cope with the subsequent singularity of the sample cov…
We consider a fundamental algorithmic question in spectral graph theory: Compute a spectral sparsifier of random-walk matrix-polynomial Lα(G)=D−∑r=1dαrD(D−1A)r where A is the adjacency matrix of a weighted, undirected graph, D is the diagonal matrix of weighted degrees, and α=(α1...αd) are nonn…
Study shows deterministic equivalent for neural network kernel convergence.
problem Understanding convergence of neural network kernels.
method Analyzes empirical spectral distribution of Conjugate Kernel, proving convergence to a deterministic limit.
result Obtains a deterministic equivalent for the Stieltjes transform and resolvent of the Conjugate Kernel.
Spectral methods improve parameter estimation in structured GLMs.
problem Parameter estimation in high-dimensional generalized linear models with structured data.
method Spectral methods using the principal eigenvector of a data-dependent matrix, with preprocessing for optimal performance.
result Precise asymptotic performance characterization and optimal preprocessing identified.
Study spectral estimators for multi-index models to recover low-dimensional signal subspaces.
problem Recovering low-dimensional signal subspaces in multi-index models.
method Spectral estimators for multi-index models.
result Precise asymptotic characterization of spectral methods' performance, revealing a phase transition for weak recovery.
We introduce RSE to measure robustness in estimation problems.
problem Estimating statistical models from observed data.
method Developed theory for spectral functions of measures to compute RSE.
result RSE reveals a reciprocal relationship with problem complexity.
Paper finds exact Hessian sharpness in deep matrix factorization.
problem Understanding the geometry of loss landscapes in deep matrix factorization.
method Presented the first exact expression for Hessian maximum eigenvalue.
result Spectral-norm balance is a sufficient condition for flatness in deep matrix factorization.
Spectral methods improve signal recovery in mixed GLMs with precise asymptotics.
problem Estimating multiple signals from unlabeled observations in mixed GLMs.
method Developed exact asymptotics for spectral methods in a proportional regime.
result Optimized spectral method combined with a linear estimator minimizes estimation error.
New method for triclustering with reduced arbitrariness.
problem Need for reduced arbitrariness in specifying cluster size.
method Spectral decomposition of tensor slices and intersection of clusters.
result Effective triclustering on synthetic and real-world data.
Develops precise expressions for random projections for better machine learning tasks.
problem Improving the accuracy of dimensionality reduction in machine learning tasks.
method Exploits recent developments in spectral analysis of random matrices to derive accurate expressions for random projection matrices.
result Provides precise expressions that reflect the practical performance of sketching methods, including Gaussian and Rademacher sketches.
The paper analyzes how quantization affects the Fisher Information Matrix's dominant eigenvalue.
problem The impact of quantization on the Fisher Information Matrix's dominant eigenvalue.
method The study examines spectral perturbation of the empirical Fisher Information Matrix under in-distribution input and quantized parameter perturbations.
result A bound on the eigenvalue under quantization noise, showing it strictly exceeds the unperturbed value at leading order.
Analysis of DPPs and k-DPPs via spectral decomposition reveals identifiable parameters and non-identifiability gaps.
problem Identifying parameters of DPPs and k-DPPs through spectral decomposition.
method Spectral decomposition of the covariance matrix, analysis of invariances, and counting arguments.
result Identifiability of parameters changes fundamentally for k-DPPs, with specific invariances and non-identifiability gaps.
Study uncovers scaling laws and spectral properties of shallow neural networks.
problem Understanding scaling laws and spectral properties of shallow neural networks.
method Leveraging connections with matrix compressed sensing and LASSO, derived a phase diagram for excess risk.
result Uncovered crossovers between scaling regimes and plateau behaviors, validated empirical observations.
Learning meaningful graphs from data plays important roles in many data mining and machine learning tasks, such as data representation and analysis, dimension reduction, data clustering, and visualization, etc. In this work, for the first time, we present a highly-scalable spectral approach (GRASPEL) for learning large…
We consider the exact recovery problem in the hypergraph stochastic block model (HSBM) with k blocks of equal size. More precisely, we consider a random d-uniform hypergraph H with n vertices partitioned into k clusters of size s=n/k. Hyperedges e are added independently with probability p if e is…
Bayesian parametric matrix models provide uncertainty quantification for spectral learning.
problem Uncertainty quantification in spectral learning for safety-critical applications.
method Bayesian parametric matrix models (B-PMMs) that extend PMMs to provide uncertainty estimates.
result B-PMMs achieve exceptional uncertainty calibration (ECE < 0.05) while maintaining favorable scaling.
Characterizes Hessian eigenspectra for realistic nonlinear models.
problem Understanding Hessian eigenspectra in realistic nonlinear models.
method Deterministic equivalent techniques from random matrix theory.
result Hessian can have qualitatively different spectral behaviors.
Generalizes abelianization for framed local systems over surfaces.
problem Understanding framed local systems over punctured surfaces for various groups.
method Analysis of spectral networks, triangulations, and matrix reinterpretation of path lifting rules.
result Parametrizations of moduli spaces of decorated and framed local systems.
Unified spectral clustering for sparse networks with heterogeneous degrees.
problem Efficiently detecting communities in sparse networks with varying degrees.
method Developed a parametrized regularized Laplacian matrix for spectral clustering.
result Improved parametrization accounts for network heterogeneity and community hardness.
New matrix ensembles better match deep neural network spectral densities.
problem Theoretical spectral density models for deep networks do not match empirical observations.
method Introduced new matrix ensemble classes to better fit observed spectral densities.
result Theoretical models for deep networks are significantly flawed.
This work analyzes self-attention matrices using random matrix theory.
problem Understanding the theoretical behavior of self-attention layers in neural networks.
method Asymptotic spectral analysis of the attention matrix, Gaussian equivalence, and linearization.
result The singular value distribution of the attention matrix is asymptotically characterized by a linear model.
Spectral clustering is one of the most widely used techniques for extracting the underlying global structure of a data set. Compressed sensing and matrix completion have emerged as prevailing methods for efficiently recovering sparse and partially observed signals respectively. We combine the distance preserving measur…
Improved singular value approximation for convolutional layers.
problem Improving accuracy of singular value approximation for linear convolutional layers.
method Developed a new spectral density matrix method for singular value approximation with improved accuracy and reduced computational complexity.
result Obtained moderate improvement in singular value distribution compared to circular approximation.
Muon optimizer simplifies matrix optimization with spectral orthogonalization.
problem Matrix optimization challenges, especially with large condition numbers.
method Simplified Muon optimizer using spectral orthogonalization of gradients.
result Simplified Muon converges linearly with independent scalar sequences, outperforming gradient descent and Adam.
In this dissertation we propose alternative analysis of distributed stochastic gradient descent (SGD) algorithms that rely on spectral properties of the data covariance. As a consequence we can relate questions pertaining to speedups and convergence rates for distributed SGD to the data distribution instead of the regu…
A new R package for high-dimensional regression and precision matrix estimation.
problem High-dimensional linear regression and precision matrix estimation challenges.
method flare package implements various regression methods and extensions for sparse precision matrix estimation.
result The flare package is efficient and scalable for large problems.
We consider the problem of estimating a consensus community structure by combining information from multiple layers of a multi-layer network using methods based on the spectral clustering or a low-rank matrix factorization. As a general theme, these "intermediate fusion" methods involve obtaining a low column rank matr…
The paper improves Bayesian precision matrix estimation for high-dimensional sparse data.
problem Estimating sparse precision matrices in high-dimensional settings.
method Tempered posterior with fully specified horseshoe prior.
result Concentration results and theoretical oracle inequality for posterior.
Develops an ℓ_p theory for PCA and spectral clustering.
problem Lack of precise characterizations of PCA scores for low-dimensional embedding.
method An ℓ_p perturbation theory for PCA in Hilbert spaces, analyzing eigenvectors and Gram matrix.
result Optimal recovery results for Gaussian mixture and stochastic block models.
Positive-curvature metrics on trees identified for specific configurations.
problem Classifying trees with positive-curvature discrete Einstein metrics.
method Spectral characterization and eigenvalue analysis of the Ricci matrix.
result Positive-curvature metrics found for specific tree configurations.
Develops quantum cluster algebra approach to solve tetrahedron equation.
problem Investigates a three-dimensional generalization of the Yang-Baxter equation.
method Quantum cluster algebra approach with realization of quantum Y-variables in terms of q-Weyl algebras.
result Obtains a solution with three spectral parameters and reproduces Sergeev's R matrix.
Study on linear regression with dependent covariates, proving universality and error characterization.
problem Linear regression with dependent covariates in high-dimensional settings.
method Analysis of ridge regression performance, Gaussian universality theorem, spectral properties of covariance matrices.
result Asymptotic performance of ridge regression is invariant under non-Gaussian covariates with preserved mean and covariance.
New method extrapolates spectral densities from smaller models to larger ones.
problem Limited practical computations for large machine learning models.
method Algebraic spectral curve theory for free decompression.
result Framework enables extrapolation of spectral densities with multiple or multi-modal bulks.
Dual regularized graph Laplacian improves spectral clustering for community detection.
problem Detecting clusters in networks with improved spectral clustering methods.
method Proposes dual regularized graph Laplacian for three spectral clustering approaches.
result Theoretical analysis shows DRSC and DRSLIM yield stable consistent community detection.
New analysis reveals masked self-supervised learning's effectiveness in extracting data structure.
problem Analyzing masked self-supervised learning in high-dimensional data.
method Developed precise high-dimensional analysis of masked modeling objectives.
result Identified phase transitions and structured regimes for masked self-supervised learning.
Spectral methods are popular in detecting global structures in the given data that can be represented as a matrix. However when the data matrix is sparse or noisy, classic spectral methods usually fail to work, due to localization of eigenvectors (or singular vectors) induced by the sparsity or noise. In this work, we …
New spectral methods improve matrix estimation in RL with low-rank structure.
problem Estimating matrices with low-rank structure in reinforcement learning.
method Spectral-based matrix estimation approaches.
result Spectral methods efficiently recover singular subspaces and minimize entry-wise error.
We give a polynomial-time algorithm for learning latent-state linear dynamical systems without system identification, and without assumptions on the spectral radius of the system's transition matrix. The algorithm extends the recently introduced technique of spectral filtering, previously applied only to systems with a…
Deep learning method clusters multi-view data matrices.
problem Clustering heterogeneous relational data matrices.
method Deep collective matrix tri-factorization (DCMTF).
result Discover latent clusters across input matrices and their associations.
The paper examines how gradient descent stabilizes low-rank matrix factorization in noisy conditions.
problem Stability of low-rank implicit regularization in perturbed deep matrix factorization.
method Derives spectral conditions for gradient descent to exhibit a low-rank phase in noiseless settings and analyzes perturbed dynamics.
result Gradient descent converges to a low-rank solution under perturbation, with explicit dependence on perturbation size.
Study compares different covariance estimation methods for portfolio allocation.
problem Comparing methods for estimating covariance and precision matrices in portfolio allocation.
method Gaussian Graphical Model (GGM), Shrinkage, Thresholding, Random Matrix Theory (RMT) methods.
result GGM methods outperform other methods in predictive ability for portfolio allocation.