New tools in nonlinear random matrices improve understanding of the Sum of Squares hierarchy.
problem Improving the Sum of Squares (SoS) hierarchy's performance on average-case problems.
method Developed new tools in nonlinear random matrices and applied them to analyze the SoS hierarchy.
result Subexponential-time SoS lower bounds for various problems, offering evidence for the low-degree likelihood ratio hypothesis.
We present a new paradigm for speeding up randomized computations of several frequently used functions in machine learning. In particular, our paradigm can be applied for improving computations of kernels based on random embeddings. Above that, the presented framework covers multivariate randomized functions. As a bypr…
Random feature matrices' singular values concentrate near their full expectation in high dimensions.
problem Characterizing the spectra of random feature matrices for regression problems.
method Analyzing two settings of input variables (random or well-separated) with conditions on dimension, complexity ratio, and sampling variance.
result The singular values of random feature matrices concentrate near their full expectation and near one with high probability.
The study sets limits on how well nonlinear models can generalize from training data.
problem Understanding the limits of generalization for nonlinear learning models.
method Deriving explicit generalization lower bounds for multi-layer neural networks and linear regression.
result Explicit bounds for general biased estimators in nonlinear networks, showing unacceptable performance for unbiased estimators.
Study on signal-plus-noise decomposition in nonlinear spiked random matrices.
problem Nonlinear spiked random matrix models with rank-one signal and noise.
method Signal-plus-noise decomposition and phase transition analysis.
result Identified precise phase transitions in signal components at critical thresholds.
The paper studies neural networks with wide layers and finds a deformed semicircle law.
problem Investigating spectral distributions of neural networks in the ultra-wide regime.
method Analyzes empirical kernel matrices, proves deformed semicircle law, provides nonlinear Hanson-Wright inequality.
result Emergence of a deformed semicircle law in the ultra-wide neural network regime.
A new method for efficient nonlinear process monitoring using random Bernoulli features.
problem High computational demands and real-time responsiveness in online monitoring systems.
method Random Bernoulli principal component analysis to capture nonlinear patterns efficiently.
result The proposed methods offer excellent scalability and reduced computational complexity.
This paper shows universality in spectrum behavior for random inner-product kernel matrices in polynomial regime.
problem Understanding spectrum behavior of random inner-product kernel matrices in polynomial regime.
method Analyzing matrices formed by a nonlinear function applied entrywise to a sample-covariance matrix, considering i.i.d. entries with all finite moments.
result The spectrum of random inner-product kernel matrices is universally described by the free convolution of the semicircular and Marčenko-Pastur distributions, with relative weights given by expanding the nonlinear function in the Hermite basis.
Study heavy-tailed weights' impact on neural network's spectral distribution.
problem Analyzing spectral distribution of conjugate kernel matrices with heavy-tailed weights.
method Computed limiting eigenvalue distribution through moments, considering heavy-tailed distributions and nonlinear activation functions.
result Heavy-tailed weights induce strong correlations, leading to fundamentally different spectral behavior.
Study reveals an equivalence principle for the spectrum of random inner-product kernel matrices in polynomial scaling.
problem Understanding the spectrum of random kernel matrices in polynomial scaling regimes.
method Investigates random matrices with nonlinear kernel functions applied to inner products of uniformly distributed vectors.
result The spectrum of the random kernel matrix is asymptotically equivalent to a simpler matrix model through free additive convolution.
Diagonal transformations preserve independence structures in non-Gaussian distributions.
problem Preserving independence structures in non-Gaussian distributions.
method Diagonal nonlinear transformations of multivariate normal variables.
result Independence structures are preserved in non-Gaussian distributions under diagonal transformations.
We consider the problem of rational decision making in the presence of nonlinear constraints. By using tools borrowed from spin glass and random matrix theory, we focus on the portfolio optimisation problem. We show that the number of ``optimal'' solutions is generically exponentially large: rationality is thus de fact…
Nonlinear RNNs' memory capacity varies widely, making it impractical.
problem The usefulness of memory capacity as a metric for linear RNNs is questioned.
method Analysis of random nonlinear RNNs with varying input scales.
result Memory capacity of nonlinear RNNs is arbitrary and impractical.
In order to pursue the issue of the relation between the financial cross-correlations and the conventional Random Matrix Theory we analyse several characteristics of the stock market correlation matrices like the distribution of eigenvalues, the cross-correlations among signs of the returns, the volatility cross-correl…
Better signal detection in undersampled data using joint and cross covariances.
problem Detecting shared signals in high-dimensional data with limited samples.
method Analysis of three covariance matrices: individual, cross, and joint.
result Joint and cross covariance matrices detect signals earlier than individual covariances.
Study eigenvalues and eigenvectors in neural networks, focusing on signal propagation.
problem Characterize signal eigenvalues and eigenvectors in neural networks.
method Characterizes signal eigenvalues and eigenvectors for a nonlinear spiked covariance model.
result Provides precise quantitative characterizations of signal eigenvalues and eigenvectors in neural networks.
Study on random matrices in deep neural networks with IID entries.
problem Distribution of singular values in product of random matrices for deep neural networks.
method Random matrix theory with a streamlined approach for non-Gaussian data.
result Generalization of macroscopic universality property to non-Gaussian data.
New method cleans cross-covariance matrices for better financial forecasting.
problem Asymptotically optimal cross-covariance cleaners fail in real-world, time-varying markets.
method Physics-informed neural network that learns from empirical singular values.
result Trained model outperforms analytical cleaners in out-of-sample cross-covariance prediction.
TRF uses ternary random features to improve ML performance without extra computation.
problem Improving ML performance with less computation and storage.
method Proposes Ternary Random Features (TRF) for random features compression.
result TRF asymptotically yields the same limiting kernel as original matrices, with improved efficiency.
The paper deals with distribution of singular values of product of random matrices arising in the analysis of deep neural networks. The matrices resemble the product analogs of the sample covariance matrices, however, an important difference is that the population covariance matrices, which are assumed to be non-random…
Many machine learning algorithms require precise estimates of covariance matrices. The sample covariance matrix performs poorly in high-dimensional settings, which has stimulated the development of alternative methods, the majority based on factor models and shrinkage. Recent work of Ledoit and Wolf has extended the sh…
Study of eigenvalues in nonlinear kernels for classification of separable data.
problem Understanding the applicability of linear equivalents in nonlinearly separable data classification.
method Analysis of conjugate kernels and their quadratic equivalents for a canonical nonlinearly separable dataset (XOR problem).
result Identification of regimes where nonlinear kernels deviate from linear equivalents, leading to label-aligned eigenspaces.
Algorithm recovers factors of rank-1 matrices from noisy measurements.
problem Estimating factors of a rank-1 matrix from nonlinearly transformed and noisy measurements.
method Alternating minimization with random initialization and analysis of empirical error recursion.
result Algorithm converges geometrically fast from random initialization, with sharp guarantees.
The paper proves local laws for non-separable sample covariance matrices.
problem Analyzing non-separable sample covariance matrices with dependent or nonlinearly transformed data.
method Tensor network framework for analyzing fluctuation averaging in the presence of higher-order cumulant structure.
result Optimal averaged local law and full anisotropic local law for non-separable sample covariance matrices.
Survey on strong convergence in random matrices and its applications.
problem Understanding convergence of random matrices to operators.
method Analysis of operator norms of noncommutative polynomials.
result New insights and applications in random graphs, geometry, and operator algebras.
New algorithms estimate Jacobian matrices for large-scale machine learning.
problem Efficiently computing search directions for large nonlinear least squares.
method Exploit low-rank structure in Hessian to estimate Jacobian matrices.
result Two algorithms perform well compared to state-of-the-art methods.
The paper proves a distribution claim for neural network Jacobians.
problem Distribution of singular values in deep neural networks.
method Free probability and random matrix theory techniques.
result Singular value distribution matches for specific cases.
Complex systems are typically represented by large ensembles of observations. Correlation matrices provide an efficient formal framework to extract information from such multivariate ensembles and identify in a quantifiable way patterns of activity that are reproducible with statistically significant frequency compared…
This paper develops a fast algorithm for solving nonlinear PDEs using sparse Cholesky factorization.
problem Efficiently solving nonlinear PDEs with Gaussian processes and kernel methods.
method Sparse Cholesky factorization for near-linear complexity.
result Near-linear complexity algorithm for working with kernel matrices of nonlinear PDEs.
We propose a scheme for recycling Gaussian random vectors into structured matrices to approximate various kernel functions in sublinear time via random embeddings. Our framework includes the Fastfood construction as a special case, but also extends to Circulant, Toeplitz and Hankel matrices, and the broader family of s…
New method solves constrained optimization problems efficiently.
problem Equality-constrained nonlinear, nonconvex optimization problems.
method Adaptive inexact Newton method with randomized iterative sketching.
result Global almost sure convergence and local linear/superlinear convergence.
Random projections help in representing sparse graphs efficiently.
problem Efficiently representing sparse graphs of varying sizes and vertex sets.
method Random projection of adjacency matrices to retain graph functionality and properties.
result Random projections can accurately represent graphs of different sizes and vertex sets in the same space.
Improved method for computing Fréchet means on SPD matrices.
problem Computing Fréchet means on the manifold of SPD matrices.
method Random matrix theory-based approach for estimating Fréchet means.
result Significantly outperforms state-of-the-art methods in experiments.
New invariants derived from random matrices for words in free groups.
problem Defining and understanding new topological invariants for words in free groups.
method Defining and analyzing invariants from w-random matrices and permutations. result Presented new topological, combinatorial, and algebraic invariants of words.
In this paper, we present a new framework to obtain tail inequalities for sums of random matrices. Compared with existing works, our tail inequalities have the following characteristics: 1) high feasibility--they can be used to study the tail behavior of various matrix functions, e.g., arbitrary matrix norms, the absol…
Paper develops new method for detecting latent structure in large symmetric data matrices.
problem Testing for latent structure in large symmetric data matrices.
method Introduces Wilcoxon--Wigner random matrices based on normalized rank statistics.
result Establishes asymptotic Gaussian fluctuations for leading eigenvalue and eigenvector of Wilcoxon--Wigner matrices.
Study extends bounds on sample covariance matrices with general dependence.
problem Quantitative bounds on sample covariance matrices with i.i.d. columns.
method Extends previous work on deterministic equivalent to rectangular random matrices with general dependence structure.
result Proves quantitative bounds involving dimensions and spectral parameter, including closer proximity to real positive semi-line.
The paper introduces a new method for tail bounds of random vectors and matrices.
problem Estimating norms of random vectors and matrices under moment assumptions.
method Variational tail bounds for norms of random vectors and matrices.
result Dimension-free concentration inequalities for various norms of random vectors and matrices.
Injectivity of ReLU networks is characterized for generative models and inverse problems.
problem Injectivity in ReLU networks for generative models and inverse problems.
method Layerwise analysis, worst-case Lipschitz constants, differential topology, random projections.
result Global injectivity of ReLU networks requires expansivity between 3.4 and 10.5 for Gaussian matrices.
Graph connection Laplacian (GCL) is a modern data analysis technique that is starting to be applied for the analysis of high dimensional and massive datasets. Motivated by this technique, we study matrices that are akin to the ones appearing in the null case of GCL, i.e the case where there is no structure in the datas…
New theory for eigenvectors of generalized Laplacian matrices, addressing dependency issues.
problem Dependency in random matrix theory hinders eigenvector analysis for latent embeddings.
method Introduces generalized Laplacian matrices and a new asymptotic theory framework.
result Established asymptotic normalities for spiked eigenvectors and eigenvalues.
Study evaluates thresholds for removing noise from DNN weights using random matrix theory.
problem Removing noise from deep neural network weights for better approximation.
method Model weights as signal + noise, use random matrix theory to estimate thresholds, evaluate using cosine similarity.
result Proposed threshold estimation method improves approximation quality.
Study provides guarantees on neural network generalization for solving Schrödinger equations.
problem Probability of centralizer being trivial for random matrices.
method Lower bounds on probability using random matrix theory and machine learning theory.
result Guarantees on transformer-based neural networks' in-context learning ability.
The study examines Fisher information matrices and neural tangent kernels for simple ReLU networks with random weights.
problem Understanding the relationship between Fisher information matrices and neural tangent kernels for 2-layer ReLU networks.
method Analyzes Fisher information matrices and neural tangent kernels for 2-layer ReLU networks with random hidden weights, focusing on spectral decomposition and eigenfunctions.
result Obtained an approximation formula for functions represented by 2-layer neural networks.
A new matrix concentration inequality for random products of matrices.
problem Understanding the behavior of random matrix products under bounded independent positive semidefinite matrices.
method Developed a non-asymptotic concentration inequality for the product of matrices.
result The inequality provides a bound on the deviation of the matrix product from its expected value.
Abstract: Nonlinear random walk with distributionally robust transition probabilities.
problem Modeling nonlinear random walks with robust transition probabilities.
method Scaling limit and nonlinear semigroup approach.
result Explicit computation of the generator and corresponding PDE.
This paper surveys various methods for dimensionality reduction and nearest neighbor search.
problem Efficiently reducing high-dimensional data to lower dimensions while preserving essential information.
method Linear and nonlinear random projections, including sparse random projections, random Fourier Features, and Random Kitchen Sinks.
result Various methods for dimensionality reduction and nearest neighbor search are explained and compared.
Paper analyzes and proves convergence of a new method for solving complex PDEs.
problem Solving high-dimensional nonlinear PDEs and PIDEs with random neural networks.
method Random deep splitting method using random neural networks.
result The method converges to the unique viscosity solution of nonlinear PDEs and PIDEs.