New ONMF model minimizes KL divergence for better sparse data modeling.
problem Clustering and data modeling with sparse vectors.
method Developed KL-ONMF algorithm based on alternating optimization.
result KL-ONMF outperforms Frobenius-norm ONMF for document classification and hyperspectral image unmixing.
Algorithm estimates covariance from noisy data efficiently.
problem Estimating covariance from a noisy set of points.
method Spectral techniques for list-decodable covariance estimation.
result Efficient algorithm with poly(1/α) sample and time complexity.
Study connects covariance cleaning theory to information theory for heavy-tailed distributions.
problem Optimizing covariance matrices for heavy-tailed distributions using information theory.
method Minimizing Frobenius norm and information loss between true and estimated covariance matrices.
result Asymptotic regime of large matrices minimizes information loss for Student's t distributions.
In this paper we extend DDVV-type inequalities involving the Frobenius norm of commutators from real symmetric and skew-symmetric matrices to Hermitian and skew-Hermitian matrices.
The paper investigates polynomial alternatives to softmax in transformer models.
problem The effectiveness of softmax attention in transformers is questioned.
method The authors explore polynomial activations as alternatives to softmax, focusing on their ability to regularize the attention matrix.
result Certain polynomials can serve as effective substitutes for softmax in transformer applications, achieving strong performance.
New method bounds singular values of convolutional kernels to stabilize gradients.
problem Stable gradients in convolutional neural networks.
method Frobenius norm regularization for convolutional kernels.
result Bounded singular values of transformation matrices.
New sample complexity bounds for linear predictors and neural networks, focusing on initialization.
problem Understanding sample complexity for vector-valued linear predictors and neural networks, especially under initialization-dependent conditions.
method Size-independent bounds on Frobenius norm distance from a fixed reference matrix, applying to vector-valued predictors and neural networks.
result Established new sample complexity bounds for feed-forward neural networks, resolving open questions and introducing a new learnable problem.
New pivoting strategy improves trace norm contraction in low-rank approximation.
problem Finding good low-rank approximations of symmetric, positive-definite matrices.
method Choosing rows with likelihood proportional to Aii2 for randomly pivoted partial Cholesky algorithm. result Same trace norm contraction result in Frobenius norm for improved pivoting strategy.
SGD converges globally to logistic loss minima for two-layer nets.
problem Global convergence of SGD for logistic loss on two-layer neural nets.
method Demonstrates existence of Frobenius norm regularized logistic loss functions as Villani functions, proving convergence and exponential rate.
result SGD converges globally to the global minima of appropriately regularized logistic empirical risk of depth 2 nets.
Tensor networks improve anomaly detection efficiency.
problem Anomaly detection in high-dimensional data.
method Tensor networks for learning a linear transformation over high-dimensional space, penalizing global tendency to normality.
result Tensor networks outperform deep and classical algorithms on various datasets.
GCNs' performance linked to feature, graph, and ground truth alignment.
problem Improving GCNs' classification performance.
method Subspace alignment measure (SAM) based on Frobenius norm of chordal distances.
result SAM quantifies the alignment between features, graph, and ground truth.
We explicitly compute the intrinsic volume of the set of real (and real symmetric) matrices of Frobenius norm one and given corank (the case of matrices with zero determinant as a special case). We give asymptotic formulas for our computations and we discuss several examples and applications.
Proposes momentum methods for Lie groups, improving on classical algorithms.
problem Optimization on nonlinear spaces, especially Lie groups.
method Generalizes Nesterov's Accelerated Gradient method to Lie groups.
result Demonstrates faster convergence for NAG-like methods on Lie groups.
New method for inferring time series graph from sparse-group log-sum penalty.
problem Inferring conditional independence graph from high-dimensional stationary multivariate Gaussian time series.
method Sparse-group log-sum penalty (LSP) and alternating direction method of multipliers (ADMM) for iterative optimization.
result Local convergence of inverse PSD estimators to the true value with rate of convergence.
Efficiently regularizes deep learning models using Jacobian nuclear norm.
problem Regularizing deep learning models to prevent overfitting and improve generalization.
method Proposes a denoising-style approximation to penalize the Jacobian nuclear norm without computing the Jacobian matrix.
result Demonstrates that penalizing the average squared Frobenius norm of Jg and Jh is equivalent to penalizing the Jacobian nuclear norm for function compositions. The problem of low-rank approximation with convex constraints, which appears in data analysis, system identification, model order reduction, low-order controller design and low-complexity modelling is considered. Given a matrix, the objective is to find a low-rank approximation that meets rank and convex constraints, w…
TTPUDR uses tensor-train decomposition for high-dimensional data analysis.
problem High-dimensional data analysis challenges.
method Tensor-train decomposition and manifold optimization.
result TTPUDR significantly outperforms past methods and state-of-the-art methods.
This paper deals with unsupervised clustering with feature selection. The problem is to estimate both labels and a sparse projection matrix of weights. To address this combinatorial non-convex problem maintaining a strict control on the sparsity of the matrix of weights, we propose an alternating minimization of the Fr…
Theory explains deep nonlinear networks' plateaus and transitions.
problem Understanding long plateaus and feature acquisition transitions in deep nonlinear networks.
method Derived an exact identity for Frobenius norms, classified activation functions, and reduced matrix flow to a scalar ODE.
result Escape time law τ⋆=Θ(ε−(r−2)) for deep nonlinear networks, where r is the number of bottleneck layers. Deep networks learn sparse hierarchical features without CoD.
problem Overparameterized deep networks struggle with the curse of dimensionality.
method Norm-constrained neural networks for sparse compositional functions.
result Deep networks can learn sparse hierarchical features efficiently.
This paper tackles fitting multilevel low rank matrices by addressing three problems.
problem Fitting a given matrix by an MLR matrix in the Frobenius norm.
method Factor fitting, rank allocation, and hierarchical partitioning.
result The proposed methods can fit a given matrix by an MLR matrix in the Frobenius norm.
Study on neural networks' sample complexity with one hidden layer.
problem Understanding how sample complexity is affected by network architecture and norm constraints.
method Norm-based uniform convergence bounds for scalar-valued one-hidden-layer networks, focusing on spectral and Frobenius norms.
result Spectral norm control is insufficient for uniform convergence guarantees, but Frobenius norm control is sufficient, with conditions.
Gradient flow on softmax attention minimizes nuclear norm of weight matrices.
problem Classification with separate key and query weight matrices.
method Gradient flow on exponential loss, separability assumption, reparameterization, approximate KKT conditions.
result Gradient flow implicitly minimizes nuclear norm of weight matrices, contrasting with Frobenius norm minimization.
Unified framework for coupled tensor completion improves recovery accuracy.
problem Improving recovery accuracy in coupled tensor completion.
method Unified framework using tensor ring (TR) decomposition with shared latent factors and novel optimization model.
result The proposed method achieves superior recovery accuracy on real-world data compared to state-of-the-art methods.
Power of network tests degrades when vertices are misaligned.
problem Power loss in network hypothesis testing due to vertex shuffling.
method Theoretical analysis and simulations of Frobenius norm differences in random dot product and stochastic block models.
result Shuffling vertices can significantly reduce the power of network tests.
Paper tests similarity between networks using a bootstrap method.
problem Determining if two networks are similar or proportional.
method Parametric bootstrap approach and Frobenius norm-based test.
result The method is versatile and consistent under various models.
This paper investigates Shampoo's heuristics and decouples preconditioner updates.
problem Improving Shampoo's heuristics for training neural networks.
method Decomposing preconditioner updates, correcting eigenvalues, and adapting eigenbasis computation frequency.
result Principled techniques to remove Shampoo's heuristics and improve training algorithms.
A new metric assesses latent variable models using data and model moments.
problem Difficulty in assessing the quality of unsupervised learning models.
method A moment-matching metric using matrix norms to compare data and model moments.
result The proposed metric is faster and has less variance than alternative methods.
We show that the objective function of conventional k-means clustering can be expressed as the Frobenius norm of the difference of a data matrix and a low rank approximation of that data matrix. In short, we show that k-means clustering is a matrix factorization problem. These notes are meant as a reference and intende…
New bounds show current methods overestimate system parameter errors.
problem Current bounds overestimate parameter errors in system identification.
method Utilized asymptotic normality and second-order decomposition.
result Obtained finite-sample bounds matching optimal rates up to constants.
Theoretical justification for deep networks' performance with regularization techniques.
problem Understanding the performance of deep networks trained with the square loss.
method Analysis of gradient flow and theoretical justification of regularization techniques.
result Convergence to solutions with smaller Frobenius norms leads to better classification error bounds.
The paper tackles matrix estimation from noisy data, focusing on low-rank matrices.
problem Estimating a low-rank matrix from noisy observations.
method The paper analyzes several estimators, including constrained nuclear-norm minimization, nuclear-norm regularized least squares, and a nonconvex constrained low-rank optimization problem.
result The estimators provide upper error bounds that depend on matrix rank, observed fraction, and matrix sums, and are minimax optimal.
New quantum state reconstruction method accelerates convergence.
problem Quantum state reconstruction for larger systems.
method Momentum-Inspired Factored Gradient Descent (MiFGD) combining compressed sensing, non-convex optimization, and acceleration.
result Converges to true density matrix at an accelerated linear rate, provably close to the true matrix.
Solves low-rank approximation problems in Hilbert spaces.
problem Low-rank approximation in Hilbert spaces.
method Closed-form solutions and error bounds for bounded linear operators.
result Generalization to bounded linear operators from finite dimensions.
New nonconvex regularizer speeds up low-rank matrix completion.
problem Low-rank matrix completion with good theoretical and empirical performance.
method Proposes a new nonconvex regularizer with adaptive shrinkage, scalable, and fast optimization.
result Proposed method achieves state-of-the-art recovery performance and is the fastest.
New algorithm learns PTFs with noisy data efficiently.
problem Learning low-degree PTFs with noisy data efficiently.
method Structural result and novel robust Chow vector estimation.
result PAC learns PTFs with nasty noise using efficient samples.
The paper surveys methods to approximate non-negative matrices using lower-dimensional factors.
problem Approximating high-dimensional non-negative matrices with lower-dimensional factors.
method Alternating minimization with surrogate functionals for Tikhonov functionals.
result Developed a general framework for adding penalty terms to surrogate functionals.
The paper studies the loss landscape of regularized deep matrix factorization, revealing unique and sharp minimizers.
problem Understanding the loss landscape and minimizers of regularized deep matrix factorization problems.
method Theoretical analysis of ℓ2-regularized deep matrix factorization/deep linear network training problems with squared-error loss. result The unique end-to-end minimizer exists for all target matrices except for a set of Lebesgue measure zero.
HMC with leapfrog integrator mixes faster than MALA under certain smoothness conditions.
problem Analyzing the mixing time of HMC and MALA for sampling from smooth distributions.
method Bounding gradient complexity and leveraging invariance of joint distribution.
result Metropolized HMC with more leapfrog steps outperforms MALA in total variation distance.
Robust covariance testing requires significantly more samples in contaminated data.
problem Testing the covariance matrix of a high-dimensional Gaussian in the presence of contamination.
method We study the problem in the Huber's contamination model, distinguishing between the identity matrix and matrices far from it in Frobenius norm.
result The sample complexity of covariance testing increases dramatically to Ω(d2) in the contaminated setting. Unified error analysis for low-rank approximation improves data assimilation performance.
problem Analyzing the error in low-rank approximation methods for data assimilation.
method Unified stochastic analysis framework for Frobenius norm error bounds on centered and non-standard Gaussian matrices.
result Unified bounds provide clearer interpretations and enable better practical choices for covariance matrices.
Novel network model estimates mixed-membership structure with covariate information.
problem Estimating latent mixed-membership structure in networks with covariate information.
method Proposes a novel network model that incorporates both community information and node covariate similarities.
result Achieves optimal estimation accuracy for similarity matrix and mixed-membership.
We identify linear dynamical systems under convex constraints with fewer samples.
problem Identifying linear dynamical systems with prior structural information.
method Constrained least squares estimator with error bounds dependent on convex set size.
result Linear dynamical systems can be reliably estimated with fewer samples than unconstrained settings.
SGD noise helps select flat minima by concentrating in sharp directions and being proportional to loss value.
problem Understanding the implicit regularization of SGD and selecting flat minima in over-parameterized models.
method Relating SGD's linear stability to the Frobenius norm of the Hessian and analyzing the alignment property of SGD noise.
result Flat minima are linearly stable for SGD, and their sharpness is bounded independently of model size and sample size.
New method for initializing low-rank neural networks improves performance.
problem Training low-rank neural networks efficiently and accurately.
method Inspired by function approximation, proposes a novel low-rank initialization framework.
result Demonstrates significant gap between spectral and low-rank initialization approaches.
We consider in this paper the problem of noisy 1-bit matrix completion under a general non-uniform sampling distribution using the max-norm as a convex relaxation for the rank. A max-norm constrained maximum likelihood estimate is introduced and studied. The rate of convergence for the estimate is obtained. Information…
We consider the problem of link prediction, based on partial observation of a large network, and on side information associated to its vertices. The generative model is formulated as a matrix logistic regression. The performance of the model is analysed in a high-dimensional regime under a structural assumption. The mi…
Improved sample complexity for ReLU networks with norm constraints.
problem Estimating sample complexity for ReLU networks under norm constraints.
method Refined Rademacher complexity analysis for function class.
result Often no explicit depth-dependence in sample complexity bound.