New ONMF model minimizes KL divergence for better sparse data modeling.
problem Clustering and data modeling with sparse vectors.
method Developed KL-ONMF algorithm based on alternating optimization.
result KL-ONMF outperforms Frobenius-norm ONMF for document classification and hyperspectral image unmixing.
New pivoting strategy improves trace norm contraction in low-rank approximation.
problem Finding good low-rank approximations of symmetric, positive-definite matrices.
method Choosing rows with likelihood proportional to Aii2 for randomly pivoted partial Cholesky algorithm. result Same trace norm contraction result in Frobenius norm for improved pivoting strategy.
Algorithm estimates covariance from noisy data efficiently.
problem Estimating covariance from a noisy set of points.
method Spectral techniques for list-decodable covariance estimation.
result Efficient algorithm with poly(1/α) sample and time complexity.
Efficiently regularizes deep learning models using Jacobian nuclear norm.
problem Regularizing deep learning models to prevent overfitting and improve generalization.
method Proposes a denoising-style approximation to penalize the Jacobian nuclear norm without computing the Jacobian matrix.
result Demonstrates that penalizing the average squared Frobenius norm of Jg and Jh is equivalent to penalizing the Jacobian nuclear norm for function compositions. Study on neural networks' sample complexity with one hidden layer.
problem Understanding how sample complexity is affected by network architecture and norm constraints.
method Norm-based uniform convergence bounds for scalar-valued one-hidden-layer networks, focusing on spectral and Frobenius norms.
result Spectral norm control is insufficient for uniform convergence guarantees, but Frobenius norm control is sufficient, with conditions.
Study connects covariance cleaning theory to information theory for heavy-tailed distributions.
problem Optimizing covariance matrices for heavy-tailed distributions using information theory.
method Minimizing Frobenius norm and information loss between true and estimated covariance matrices.
result Asymptotic regime of large matrices minimizes information loss for Student's t distributions.
In this paper we extend DDVV-type inequalities involving the Frobenius norm of commutators from real symmetric and skew-symmetric matrices to Hermitian and skew-Hermitian matrices.
The paper investigates polynomial alternatives to softmax in transformer models.
problem The effectiveness of softmax attention in transformers is questioned.
method The authors explore polynomial activations as alternatives to softmax, focusing on their ability to regularize the attention matrix.
result Certain polynomials can serve as effective substitutes for softmax in transformer applications, achieving strong performance.
The problem of low-rank approximation with convex constraints, which appears in data analysis, system identification, model order reduction, low-order controller design and low-complexity modelling is considered. Given a matrix, the objective is to find a low-rank approximation that meets rank and convex constraints, w…
Gradient flow on softmax attention minimizes nuclear norm of weight matrices.
problem Classification with separate key and query weight matrices.
method Gradient flow on exponential loss, separability assumption, reparameterization, approximate KKT conditions.
result Gradient flow implicitly minimizes nuclear norm of weight matrices, contrasting with Frobenius norm minimization.
New sample complexity bounds for linear predictors and neural networks, focusing on initialization.
problem Understanding sample complexity for vector-valued linear predictors and neural networks, especially under initialization-dependent conditions.
method Size-independent bounds on Frobenius norm distance from a fixed reference matrix, applying to vector-valued predictors and neural networks.
result Established new sample complexity bounds for feed-forward neural networks, resolving open questions and introducing a new learnable problem.
SGD converges globally to logistic loss minima for two-layer nets.
problem Global convergence of SGD for logistic loss on two-layer neural nets.
method Demonstrates existence of Frobenius norm regularized logistic loss functions as Villani functions, proving convergence and exponential rate.
result SGD converges globally to the global minima of appropriately regularized logistic empirical risk of depth 2 nets.
The matrix completion problem consists in reconstructing a matrix from a sample of entries, possibly observed with noise. A popular class of estimator, known as nuclear norm penalized estimators, are based on minimizing the sum of a data fitting term and a nuclear norm penalization. Here, we investigate the case where …
Deep networks learn sparse hierarchical features without CoD.
problem Overparameterized deep networks struggle with the curse of dimensionality.
method Norm-constrained neural networks for sparse compositional functions.
result Deep networks can learn sparse hierarchical features efficiently.
We consider in this paper the problem of noisy 1-bit matrix completion under a general non-uniform sampling distribution using the max-norm as a convex relaxation for the rank. A max-norm constrained maximum likelihood estimate is introduced and studied. The rate of convergence for the estimate is obtained. Information…
Improved sample complexity for ReLU networks with norm constraints.
problem Estimating sample complexity for ReLU networks under norm constraints.
method Refined Rademacher complexity analysis for function class.
result Often no explicit depth-dependence in sample complexity bound.
Tensor networks improve anomaly detection efficiency.
problem Anomaly detection in high-dimensional data.
method Tensor networks for learning a linear transformation over high-dimensional space, penalizing global tendency to normality.
result Tensor networks outperform deep and classical algorithms on various datasets.
A new metric assesses latent variable models using data and model moments.
problem Difficulty in assessing the quality of unsupervised learning models.
method A moment-matching metric using matrix norms to compare data and model moments.
result The proposed metric is faster and has less variance than alternative methods.
Theoretical justification for deep networks' performance with regularization techniques.
problem Understanding the performance of deep networks trained with the square loss.
method Analysis of gradient flow and theoretical justification of regularization techniques.
result Convergence to solutions with smaller Frobenius norms leads to better classification error bounds.
The spectral k-support norm enjoys good estimation properties in low rank matrix learning problems, empirically outperforming the trace norm. Its unit ball is the convex hull of rank k matrices with unit Frobenius norm. In this paper we generalize the norm to the spectral (k,p)-support norm, whose additional para…
We explicitly compute the intrinsic volume of the set of real (and real symmetric) matrices of Frobenius norm one and given corank (the case of matrices with zero determinant as a special case). We give asymptotic formulas for our computations and we discuss several examples and applications.
Proposes momentum methods for Lie groups, improving on classical algorithms.
problem Optimization on nonlinear spaces, especially Lie groups.
method Generalizes Nesterov's Accelerated Gradient method to Lie groups.
result Demonstrates faster convergence for NAG-like methods on Lie groups.
Advances robust principal component analysis with transformed ℓ1 regularization.
problem Recovering low-rank structures from noisy, partially observed data corrupted by sparse outliers.
method Proposes transformed ℓ1 (TL1) regularization to improve approximations of rank and ℓ0 functional.
result Achieves higher accuracy in estimating low-rank and sparse components compared to classical convex models, especially under non-uniform sampling schemes.
New nonconvex regularizer speeds up low-rank matrix completion.
problem Low-rank matrix completion with good theoretical and empirical performance.
method Proposes a new nonconvex regularizer with adaptive shrinkage, scalable, and fast optimization.
result Proposed method achieves state-of-the-art recovery performance and is the fastest.
The paper predicts survival functions using random survival trees and concordance maximization.
problem Predicting conditional survival functions in right-censored data.
method The approach combines regression strategies with random survival trees and maximizes concordance.
result The proposed weighted predictor outperforms the usual survival cobra in terms of concordance.
Matrix completion has been well studied under the uniform sampling model and the trace-norm regularized methods perform well both theoretically and numerically in such a setting. However, the uniform sampling model is unrealistic for a range of applications and the standard trace-norm relaxation can behave very poorly …
Paper analyzes singular subspace estimation in noisy matrix models.
problem Estimating low-rank signals in noisy matrix data.
method Asymptotic distributional theory, extreme value theory, saddle point approximation, random matrix theory.
result Plug-in test statistic based on two-to-infinity norm has higher power for detecting structured alternatives.
New bounds show current methods overestimate system parameter errors.
problem Current bounds overestimate parameter errors in system identification.
method Utilized asymptotic normality and second-order decomposition.
result Obtained finite-sample bounds matching optimal rates up to constants.
New method for inferring time series graph from sparse-group log-sum penalty.
problem Inferring conditional independence graph from high-dimensional stationary multivariate Gaussian time series.
method Sparse-group log-sum penalty (LSP) and alternating direction method of multipliers (ADMM) for iterative optimization.
result Local convergence of inverse PSD estimators to the true value with rate of convergence.
If (M,g) is a compact real analytic Riemannian manifold, we give a necessary and sufficient condition for there to be a sequence of quasimodes of order o(λ) saturating sup-norm estimates. In particular, it gives optimal conditions for existence of eigenfunctions satisfying maximal sup norm bounds. The condition is …
Extended Gauss-Markov theorem for linear estimation with bounded bias.
problem Linear estimation with bounded bias operator.
method Derive optimal estimator formulas for Nuclear and Spectral norms, analyze generalization error.
result Cross-validated Nuclear and Spectral regressors outperform Ridge regression in simulations.
In this work we construct an optimal linear shrinkage estimator for the covariance matrix in high dimensions. The recent results from the random matrix theory allow us to find the asymptotic deterministic equivalents of the optimal shrinkage intensities and estimate them consistently. The developed distribution-free es…
This work provides closed-form solutions and minimum achievable errors for a large class of low-rank approximation problems in Hilbert spaces. The proposed theorem generalizes to the case of bounded linear operators the previous results obtained in the finite dimensional case for the Frobenius norm. The theorem provide…
This paper deals with unsupervised clustering with feature selection. The problem is to estimate both labels and a sparse projection matrix of weights. To address this combinatorial non-convex problem maintaining a strict control on the sparsity of the matrix of weights, we propose an alternating minimization of the Fr…
Recently, metric learning and similarity learning have attracted a large amount of interest. Many models and optimisation algorithms have been proposed. However, there is relatively little work on the generalization analysis of such methods. In this paper, we derive novel generalization bounds of metric and similarity …
Efficiently learns matching rewards in two-sided markets with matrix completion.
problem Learning high-dimensional matching rewards in matching markets with limited data.
method Utilizes matrix completion with a novel approach to handle matching interference.
result Near-optimal guarantees for reward learning under matching interference.
The paper extends Frobenius-type theorems to non-smooth settings with Hölder estimates.
problem Extending Frobenius-type theorems to non-Lipschitz subbundles and vector fields.
method Develops a singular version of the Frobenius theorem for log-Lipschitz vector fields and proves Hölder estimates.
result Sharp regularity results for log-Lipschitz vector fields and their parameterizations.
Low-rank inducing unitarily invariant norms have been introduced to convexify problems with low-rank/sparsity constraint. They are the convex envelope of a unitary invariant norm and the indicator function of an upper bounding rank constraint. The most well-known member of this family is the so-called nuclear norm. To …
Theory explains deep nonlinear networks' plateaus and transitions.
problem Understanding long plateaus and feature acquisition transitions in deep nonlinear networks.
method Derived an exact identity for Frobenius norms, classified activation functions, and reduced matrix flow to a scalar ODE.
result Escape time law τ⋆=Θ(ε−(r−2)) for deep nonlinear networks, where r is the number of bottleneck layers. This paper tackles fitting multilevel low rank matrices by addressing three problems.
problem Fitting a given matrix by an MLR matrix in the Frobenius norm.
method Factor fitting, rank allocation, and hierarchical partitioning.
result The proposed methods can fit a given matrix by an MLR matrix in the Frobenius norm.
The Schatten-p quasi-norm (0<p<1) is usually used to replace the standard nuclear norm in order to approximate the rank function more accurately. However, existing Schatten-p quasi-norm minimization algorithms involve singular value decomposition (SVD) or eigenvalue decomposition (EVD) in each iteration, and thus may…
Paper estimates noise covariance in correlated multi-task linear models.
problem Estimating noise covariance in multi-task high-dimensional linear models with correlated noise.
method Uses multi-task elastic-net and lasso estimators to estimate noise covariance, correcting bias in squared residual matrix.
result Develops a novel estimator of noise covariance that converges at rate n−1/2, matching oracle estimator under suitable conditions. Power of network tests degrades when vertices are misaligned.
problem Power loss in network hypothesis testing due to vertex shuffling.
method Theoretical analysis and simulations of Frobenius norm differences in random dot product and stochastic block models.
result Shuffling vertices can significantly reduce the power of network tests.
This paper investigates Shampoo's heuristics and decouples preconditioner updates.
problem Improving Shampoo's heuristics for training neural networks.
method Decomposing preconditioner updates, correcting eigenvalues, and adapting eigenbasis computation frequency.
result Principled techniques to remove Shampoo's heuristics and improve training algorithms.
We show that the objective function of conventional k-means clustering can be expressed as the Frobenius norm of the difference of a data matrix and a low rank approximation of that data matrix. In short, we show that k-means clustering is a matrix factorization problem. These notes are meant as a reference and intende…
New quantum state reconstruction method accelerates convergence.
problem Quantum state reconstruction for larger systems.
method Momentum-Inspired Factored Gradient Descent (MiFGD) combining compressed sensing, non-convex optimization, and acceleration.
result Converges to true density matrix at an accelerated linear rate, provably close to the true matrix.
This paper proposes a new method for estimating sparse precision matrices in the high dimensional setting. It has been popular to study fast computation and adaptive procedures for this problem. We propose a novel approach, called Sparse Column-wise Inverse Operator, to address these two issues. We analyze an adaptive …
The heavy-tailed distributions of corrupted outliers and singular values of all channels in low-level vision have proven effective priors for many applications such as background modeling, photometric stereo and image alignment. And they can be well modeled by a hyper-Laplacian. However, the use of such distributions g…