Sharp global guarantees for noisy overparameterized low-rank recovery.
problem Understanding practical success of overparameterization in noisy conditions.
method Unified proof technique combining escape directions and counterexample inexistence.
result Near-second-order points achieve minimax-optimal recovery bounds.
Improved guarantees for nonconvex matrix factorization with rank overparameterization.
problem Minimizing nonconvex objective over low-rank matrices.
method Overparameterized Burer--Monteiro approach, leveraging smoothness and strong convexity.
result Local optimization globally converges to global optimum under certain rank conditions.
New approach uses compressible dynamics to train deep models efficiently.
problem Efficient training of deep overparameterized models with low-rank structures.
method Leveraging low-dimensional structures and compressible dynamics within model parameters.
result Improved training efficiency and reduced overfitting in language models.
Improved convergence for overparameterized low-rank matrix sensing.
problem Overparameterized low-rank matrix sensing with unknown rank and ill-conditioning.
method ScaledGD(λ) - preconditioned gradient descent method. result ScaledGD(λ) converges at a constant linear rate after a logarithmic number of iterations. Gradient descent with preconditioning finds global optima in overparameterized nonconvex factorization.
problem Finding global optima in nonconvex Burer-Monteiro factorization.
method Preconditioned gradient descent for overparameterized nonconvex function minimization.
result Gradient descent with preconditioning achieves linear convergence in the overparameterized case.
We reveal a model rank that predicts successful recovery of target functions at overparameterization.
problem Understanding the mysterious good generalization performance of overparameterized nonlinear models.
method Rank stratification and linear stability theory for general nonlinear models.
result Linearly stable functions are preferred by nonlinear training, and model rank predicts minimal training data size.
Paper studies asymmetric matrix sensing, proving gradient descent converges to low-rank solutions.
problem Reconstructing asymmetric low-rank matrices from linear measurements.
method Factorized gradient descent with coupling and regularization properties.
result Gradient descent from small random initialization converges to globally optimal and generalizing solutions.
Deep ReLU networks with extra parameters have mostly good loss landscapes.
problem Finding good local minima in the loss landscape of deep neural networks.
method Analyzing shallow and deep ReLU networks with extra parameters on a generic dataset.
result Most activation patterns correspond to regions with no bad local minima.
New approach shows why overparameterized neural nets generalize well.
problem Understanding why overparameterized neural nets generalize well in practice.
method An alternative notion of capacity for attention-based models based on the effective rank of attention matrices.
result Generalization bound matches empirical scaling laws observed in large language models.
Gradient descent with small random init mimics spectral methods for low-rank matrix recovery.
problem Reconstructing a low-rank matrix from few measurements.
method Gradient descent with small random initialization followed by a few iterations.
result Gradient descent from small random init converges to a well-generalizing solution.
KReTTaH uses tensor trains and Hadamard overparameterization for fast, interpretable multi-way data imputation.
problem Multi-way data imputation for high-dimensional functional MRI and dynamic graph recovery.
method Reformulates imputation as RKHS regression with TT-constrained coefficients and Hadamard overparameterization. Optimizes TT coefficients and kernel matrices on Riemannian manifolds.
result Consistently outperforms state-of-the-art methods in modeling accuracy.
KReTTaH uses tensor trains and Hadamard overparameterization for fast, interpretable multi-way data imputation.
problem Multi-way data imputation in high-dimensional spaces.
method Reformulates imputation as RKHS regression with TT-constrained coefficients, optimized on manifold frameworks.
result Consistently outperforms state-of-the-art methods in accuracy.
Flat minima lead to better generalization in low-rank matrix recovery models.
problem Understanding why flat minima generalize well in overparameterized models.
method Analysis of overparameterized matrix and bilinear sensing, robust PCA, covariance matrix estimation, and neural networks with quadratic activation functions.
result Flat minima, measured by the trace of the Hessian, exactly recover the ground truth in low-rank matrix recovery models under standard statistical assumptions.
Analyzes the structure and rank of neural network Hessians.
problem Understanding redundancy in overparameterized neural networks.
method Theoretical tools to analyze Hessian map range and rank deficiency.
result Exact formulas and tight upper bounds for Hessian rank of deep linear networks.
ScaledGD accelerates ill-conditioned low-rank estimation.
problem Slow convergence of gradient descent in ill-conditioned problems.
method Scaled gradient descent (ScaledGD) with preconditioning.
result Linear convergence rate independent of condition number.
Sharp asymptotics reveal how network width controls learnability in quadratic neural networks.
problem Understanding learnability in overparameterized quadratic neural networks.
method Mapping ERM to convex matrix sensing with nuclear norm penalization.
result Characterization of global minima and precise generalization thresholds.
PAC-Bayes bounds for Gibbs posteriors derived via singular learning theory.
problem Generalization bounds for overparameterized models with data-dependent priors.
method Explicit non-asymptotic PAC-Bayes bounds using singular learning theory.
result Explicit posterior-averaged risk bounds for overparameterized models.
Improved deep neural network generalization through noise resilience.
problem Understanding and predicting generalization error of deep neural networks.
method Noise resilience measures to predict generalization error.
result Secured 5th position in the PGDL competition at NeurIPS 2020.
Paper shows robustness of gradient descent in matrix sensing despite perturbations.
problem Understanding robustness of gradient descent in matrix sensing.
method Developed perturbed gradient flow to capture noise and improve robustness.
result Gradient descent is robust to perturbations in matrix sensing.
Overparameterization enhances SAM's effectiveness in minimizing sharpness.
problem Improving generalization in deep neural networks.
method Analysis of Sharpness-Aware Minimization (SAM) under varying degrees of overparameterization.
result Overparameterization significantly improves SAM's performance, particularly in noisy and sparse settings.
This work analyzes how bottleneck layers and skip connections affect linear denoising autoencoders' generalization.
problem Understanding the generalization of linear denoising autoencoders in overparameterized regimes.
method Analyzes two-layer linear denoising autoencoders with a bottleneck layer and skip connection, deriving test risk formulas.
result Bottleneck layers introduce an additional complexity measure, while skip connections can mitigate variance.
Study shows overparameterization helps in generalizing from smooth interpolants.
problem Understanding generalization in overparameterized linear models.
method Analysis of random Fourier series model with weighted trigonometric interpolation.
result Weighted trigonometric interpolation leads to lower generalization error in overparameterized scenarios.
Overparameterized models are more vulnerable to membership inference attacks.
problem Vulnerability of overparameterized models to membership inference attacks.
method Theoretical and empirical analysis of overparameterized linear and ridge-regularized linear regression models in the Gaussian data setting.
result Increased number of parameters and model complexity increase vulnerability to membership inference attacks.
Nonnegative low-rank matrix recovery can have spurious local minima.
problem Nonnegative low-rank matrix recovery problems can have spurious local minima.
method Investigated projected gradient methods for nonnegative low-rank recovery problems.
result Benign nonconvexity holds in the fully-observed case with RIP constant δ=0 but fails in the partially-observed case and higher-rank ground truths.
Overparameterized ensembles don't offer generalization benefits over single large models.
problem Theoretical limitations of ensembles in overparameterized settings.
method Using ensembles of random feature (RF) regressors, the paper clarifies how modern ensembles differ from underparameterized counterparts.
result Infinite ensembles of overparameterized RF regressors become pointwise equivalent to single infinite-width RF regressors, and finite width ensembles converge to single models with the same parameter budget.
This work analyzes how overparameterization aids GANs in reaching global saddle points.
problem Understanding the role of overparameterization in GANs for convergence to global saddle points.
method Theoretical and empirical analysis of overparameterized GANs with various architectures and datasets.
result GDA converges to a global saddle point in overparameterized GANs with certain assumptions.
Overparameterized models improve performance in sequential learning tasks.
problem Catastrophic forgetting in overparameterized neural networks.
method Two-task linear regression problem with random orthogonal transformations.
result Overparameterization mitigates catastrophic forgetting in sequential learning tasks.
Overparameterized models can worsen minority group errors even when overall test error improves.
problem Overparameterization exacerbates spurious correlations, harming minority groups.
method Simulations and experiments on image datasets, theoretical analysis of linear models.
result Subsampling the majority group can achieve low minority error in overparameterized models.
Estimates generalization gap for overparameterized models using Langevin approximation.
problem Estimating the difference between training and generalization performance in overparameterized models.
method Functional variance and Langevin approximation of functional variance.
result Demonstrates efficient estimation of generalization gaps for overparameterized models.
Meta learning works well with overparameterized models, a phenomenon called 'benign overfitting'.
problem Understanding why overparameterized models perform well in few-shot learning.
method Analyzed the generalization performance of gradient-based meta learning with an overparameterized meta linear regression model.
result Demonstrated that overparameterized meta learning can still generalize well, a phenomenon called 'benign overfitting'.
Weight normalization speeds up matrix sensing problems.
problem Matrix sensing with overparameterization.
method Generalized weight normalization with Riemannian optimization.
result WN achieves linear convergence, improving speed and complexity.
Overparameterized models generalize well despite fitting noisy data.
problem Understanding why overparameterized models generalize well despite fitting noisy data.
method Statistical signal processing perspective.
result Overparameterized models often outperform underparameterized models in test performance.
A new criterion selects models in overparameterized settings.
problem Model selection for overparameterized models with more parameters than data.
method Establishes Bayesian duality and introduces the Interpolating Information Criterion.
result The Interpolating Information Criterion selects models in overparameterized settings.
Improves overparameterized models' robustness to distribution shifts.
problem Accuracy drop on testing distributions different from training.
method Importance tempering to improve decision boundaries.
result State-of-the-art results on worst group classification tasks.
Natural gradient descent has proven effective at mitigating the effects of pathological curvature in neural network optimization, but little is known theoretically about its convergence properties, especially for \emph{nonlinear} networks. In this work, we analyze for the first time the speed of convergence of natural …
The paper shows how data and algorithm interactions affect overparameterized linear regression generalization.
problem Understanding generalization in overparameterized linear regression.
method Introducing data-algorithm compatibility and performing data-dependent trajectory analysis with gradient descent.
result Early stopping iterates lead to better generalization than last-iterate analysis, with weaker restrictions.
One of the most surprising and exciting discoveries in supervised learning was the benefit of overparameterization (i.e. training a very large model) to improving the optimization landscape of a problem, with minimal effect on statistical performance (i.e. generalization). In contrast, unsupervised settings have been u…
New algorithm tackles high-dimensional contextual bandits without sparsity.
problem High-dimensional linear contextual bandit problem with large feature space.
method Proposes explore-then-commit (EtC) and adaptive explore-then-commit (AEtC) algorithms.
result Derives optimal rate for ETC algorithm and shows adaptive AEtC achieves it.
This paper explains how overparameterization aids in meta-learning with few samples.
problem Building a generalizable model with few samples in meta-learning.
method Analyzes the optimal linear representation and sample complexity for meta-learning tasks.
result Overparameterization naturally answers fundamental meta-learning questions, reducing sample complexity.
Bayesian method improves predictions in overparameterized nonlinear regression.
problem Understanding overparameterization in nonlinear regression models.
method Bayesian framework with adaptive prior considering data spectral structure.
result Posterior contraction established for generalized linear and single-neuron models, demonstrating prediction consistency.
Overparameterized models generalize well in offline contextual bandits, but policy-based algorithms struggle.
problem The performance gap between value-based and policy-based algorithms in offline contextual bandits with overparameterized models.
method Analysis of action-stability in objectives and formal proofs of regret bounds.
result The performance gap is due to action-stability of objectives, with value-based objectives being stable and policy-based objectives unstable.
The paper examines VI for overparameterized BNNs, revealing a trade-off between likelihood and KL terms.
problem Critical issue in mean-field VI training for overparameterized BNNs.
method Theoretical and empirical study of overparameterized two-layer BNNs using VI.
result A trade-off between likelihood and KL terms in overparameterized regime, with KL scaling crucial.
Uniform bounds for neural networks' generalization error in overparameterized settings.
problem Generalization error in overparameterized neural networks.
method Neural Tangent kernel theory and Mercer decomposition of the NT kernel in spherical harmonics.
result Uniform generalization bounds for overparameterized neural networks in RKHS.
Improved TD learning with neural nets reduces sample complexity and overparameterization.
problem Temporal difference learning with neural networks in large state spaces.
method Projection-free and max-norm regularized Neural TD learning, with Lyapunov drift analysis.
result Max-norm regularization significantly improves TD learning's sample complexity and overparameterization.
This work explains how large neural networks generalize well despite overparameterization.
problem Understanding the generalization behavior of large neural networks.
method Theoretical analysis of approximation and generalization errors in regression and classification tasks.
result Deep overparameterized neural networks are statistically consistent across different tasks when regularization is applied.
Overparameterized MLR fits hyper-curves, improving model robustness.
problem Improper predictors degrade model generalizability.
method Parameterizing with a scalar and monomial basis, fitting hyper-curves.
result Hyper-curve approach yields robust predictions for noisy data.
Study robust recovery of low-rank matrices from corrupted measurements without rank prior.
problem Robust recovery of low-rank matrices from corrupted Gaussian measurements with unknown rank.
method Subgradient method with diminishing stepsizes for nonconvex nonsmooth problem.
result Subgradient method converges to exact low-rank solution at sublinear rate under RDPP condition.
New loss function restores importance weighting in overparameterized models.
problem Restoring importance weighting in overparameterized neural networks.
method Introduced polynomially-tailed losses to restore effects of importance weighting.
result Polynomially-tailed losses improve performance in correcting distribution shift.