This paper explores adaptive methods in over-parameterized linear regression.
problem Understanding why neural networks generalize well in over-parameterized settings.
method Characterizes two sub-classes of adaptive methods and their generalization performance.
result Adaptive methods in over-parameterized linear regression converge to the minimum norm solution.
Over-parameterized CNNs show U-shaped test risk with depth increase.
problem Understanding the impact of depth on test risk in over-parameterized CNNs.
method Empirical image classification experiments and linear regression framework.
result Test risk is U-shaped with increasing depth in over-parameterized CNNs.
The paper explores how over-parameterized linear regression models generalize without violating learning theory principles.
problem Understanding how over-parameterized linear regression models generalize without violating learning theory principles.
method The paper uses the predictive normalized maximum likelihood (pNML) learner to investigate the minimum norm solution of over-parameterized linear regression models.
result The model generalizes well when the test sample lies in a subspace spanned by eigenvectors associated with large eigenvalues of the training data.
The paper analyzes optimal implicit bias in linear regression for over-parameterized models.
problem Finding the best generalization performance in over-parameterized linear regression.
method Asymptotic analysis of generalization performance for convex functions/potentials.
result Optimal implicit bias that achieves the best generalization error under certain conditions.
New geometric interpretation explains over-parameterized models and adversarial perturbations.
problem Geometric understanding of over-parameterized regression and adversarial perturbations.
method Alternative geometric interpretation of regression in feature space.
result Adversarial perturbations are a natural feature of biased models due to underlying geometry.
Study shows how over-parameterized classifiers can still perform well on noisy data.
problem Understanding how maximum margin classifiers perform in over-parameterized settings with noisy data.
method Analyzes maximum margin classifiers on sub-Gaussian mixtures, providing risk bounds.
result Characterizes conditions for 'benign overfitting' in linear classification problems.
Analyzes generalization error in generalized linear models, explaining double descent phenomenon.
problem Understanding generalization of machine learning models in high dimensions.
method Develops a framework to characterize asymptotic generalization error for generalized linear models.
result Rigorously explains the double descent phenomenon in generalized linear models.
Study reveals benign overfitting in time series models with over-parameterization.
problem Analyzing over-parameterized linear models with dependent time-series data.
method Developed an estimator using interpolation and derived non-asymptotic risk bounds.
result Risk bound is influenced by the coherence of temporal covariance matrices at different time steps.
The study identifies spurious correlations in high-dimensional regression and quantifies their impact.
problem Spurious correlations in high-dimensional regression models.
method Statistical characterization of spurious correlations, quantifying their amount via ridge regularization.
result The value of regularization strength that minimizes test loss is in an interval where spurious correlations increase.
New methods solve tensor-on-tensor regression with unknown rank, revealing benefits of over-parameterization.
problem Connecting tensor responses to tensor covariates with unknown intrinsic rank.
method Riemannian gradient descent and Riemannian Gauss-Newton methods for tensor-on-tensor regression.
result Riemannian optimization methods converge linearly and quadratically to a statistically optimal estimate in rank over-parameterized settings.
The paper proposes a gradient-based method for multi-penalty Ridge regression.
problem Optimizing multiple regularization hyperparameters for linear regression.
method Gradient-based optimization through matrix differential calculus.
result The method outperforms traditional regularization techniques like LASSO and Ridge.
The paper studies the minimum ℓ₁-norm interpolator's risk behavior in over-parameterized settings.
problem Understanding the risk behavior of minimum ℓ₁-norm interpolators in high-dimensional settings.
method Exact characterization of the risk behavior through a system of two non-linear equations.
result Observation of a multi-descent phenomenon in the generalization risk of the minimum ℓ₁-norm interpolator.
New insights into optimization and generalization for linear models.
problem Understanding the implicit regularization of optimization methods for linear models.
method Investigating the norms minimized by interpolating solutions and using projections to move between solutions.
result Proving that for over-parameterized linear classification, projections onto the data-span enable the use of under-parameterized techniques.
Improves adaptivity in sequence models by over-parameterizing.
problem Adaptivity and generalization in sequence models.
method Over-parameterized gradient descent using eigenfunctions.
result Over-parameterization enhances model adaptivity and generalization.
MAML with over-parameterized DNNs converges globally at a linear rate.
problem Few-shot learning with limited data.
method Model-agnostic meta-learning (MAML) with over-parameterized deep neural networks (DNNs).
result MAML with over-parameterized DNNs converges globally at a linear rate.
Exact expressions for double descent and implicit regularization in over-parameterized models.
problem Understanding the generalization error of over-parameterized models like deep neural networks.
method Surrogate random design to replace standard i.i.d. design, leading to exact expressions for mean squared error and implicit regularization.
result Exact non-asymptotic expressions for double descent and implicit regularization in over-parameterized models.
Knoop enhances variable selection with over-parameterization and knockoffs.
problem Challenges of variable selection in high-dimensional datasets.
method Generates knockoff variables, integrates them into an over-parameterized model, and uses anomaly-based significance tests.
result Superior performance in variable selection compared to existing methods.
Study on ridge regression in convolutional models shows double descent error behavior.
problem Understanding generalization and estimation error in over-parameterized convolutional models.
method Analysis of ridge estimators for convolutional linear models, derivation of exact error formulae.
result Ridge estimators exhibit double descent error behavior in high-dimensional convolutional models.
Study optimizes linear regression analysis for high-dimensional settings.
problem Understanding high-dimensional linear regression with interpolation and regularization.
method Localized uniform convergence analysis of optimistic rates for linear regression.
result Recover guarantees for ridge and LASSO regression under random designs.
Deeper models have a more favorable optimization landscape, making them more robust to noise.
problem Characterizing the effect of depth on the optimization landscape of linear regression models.
method Robust and over-parameterized setting, simple sub-gradient method.
result A simple sub-gradient method converges to a balanced solution that is close to the ground truth and enjoys a flat local landscape.
Improved autoencoders show joint training benefits over weak training.
problem Improving unsupervised learning performance with over-parameterized networks.
method Analyzing gradient dynamics of two-layer autoencoders with ReLU activation, proving linear convergence in weakly-trained and jointly-trained regimes.
result Joint training leads to better global optima and requires less over-parameterization.
New insights into bias and variance in over-parameterized models.
problem Understanding bias and variance in over-parameterized models.
method Analytic expressions derived from statistical physics for two minimal models.
result Over-parameterized models can overfit even in noiseless conditions.
Linear models can overfit without harming OOD generalization under certain conditions.
problem Understanding how overparameterized linear models generalize to out-of-distribution data.
method Analyzing overparameterized linear models under covariate shift, providing guarantees for OOD generalization.
result Benign overfitting occurs in standard ridge regression under OOD conditions, with specific structural conditions on target covariance.
This work proposes a mathematical framework for loss landscapes and optimization in deep neural networks.
problem The effectiveness of gradient-based optimization in over-parameterized neural networks.
method A modern view and mathematical framework of loss landscapes and efficient optimization in over-parameterized machine learning models.
result Wide neural networks satisfy the PL∗ condition, explaining (S)GD convergence to a global minimum. PrecGD restores linear convergence in over-parameterized nonconvex matrix factorization.
problem Slow convergence of local search algorithms in over-parameterized nonconvex matrix factorization.
method Preconditioned Gradient Descent (PrecGD) with an inexpensive ℓ2 regularization. result PrecGD restores linear convergence rate even in the over-parameterized case.
The paper shows how the generalization curve can have multiple peaks, influenced by data and learning algorithm biases.
problem Understanding the generalization behavior of linear regression models under varying parameterizations.
method Analyzes generalization loss in linear regression models with varying parameterizations, both under- and over-parameterized.
result The generalization curve can have an arbitrary number of peaks, and their locations can be controlled.
Gradient descent trains neural networks to match kernel regression's sharp generalization rate.
problem Training over-parameterized neural networks for nonparametric regression.
method Gradient descent with early stopping on over-parameterized two-layer neural networks.
result Trained neural networks achieve sharp generalization rate of O(εn2). Paper proposes a method to learn linear regression models using multiple pre-trained models.
problem Learning a linear regression model with limited target data.
method Representation transfer learning method using multiple pre-trained models.
result The method achieves better sample complexity compared to baseline methods.
Neural networks trained with PGD achieve sharp regression rates in interpolation spaces.
problem Nonparametric regression using over-parameterized neural networks in interpolation spaces.
method Over-parameterized two-layer neural networks trained with Preconditioned Gradient Descent (PGD) and early stopping.
result Achieves a sharp regression rate of \(\cO(n^{-\frac{2αs'}{2αs'+1}})\) in interpolation spaces \(\bth{\cH_K}^{s'}\).
Paper explores why overfitted DNNs in adversarial training can generalize.
problem Understanding why overfitted DNNs in adversarial training can generalize despite poor robust generalization.
method An approximation viewpoint to analyze the robust overfitting of over-parameterized DNNs.
result Existence of infinitely many overfitted DNNs that achieve good robust generalization under certain conditions.
DEQs converge to optimal solutions with mild over-parameterization.
problem Training over-parameterized deep equilibrium models.
method Solves equilibrium point directly, uses gradient descent, and analyzes convergence via linear rate.
result Gradient descent converges to a globally optimal solution at a linear rate for quadratic loss.
Gradient Descent with Projection learns low-degree polynomials efficiently.
problem Learning low-degree spherical polynomials with neural networks.
method Over-parameterized two-layer neural network with Gradient Descent with Projection.
result Achieves nearly minimax optimal sample complexity and risk bound.
DebiNet uses over-parameterized neural networks to improve linear model performance and debiasing.
problem Improving linear model performance and debiasing in high-dimensional settings.
method Incorporates over-parameterized neural networks into semi-parametric models to estimate parameters consistently.
result DebiNet offers valid inference and accurate prediction by leveraging neural networks' universal approximation and linear model's interpretability.
Proposes a continuous, differentiable model from local adaptive models.
problem Inadequate continuity and differentiability in over-parameterized models.
method A global continuous and differentiable model constructed from weighted averages of locally learned models.
result Achieves faster statistical convergence and improved performance in various settings.
Bayesian interpretation explains double descent in deep learning models.
problem Understanding the risk function behavior of over-parameterized models.
method Bayesian model selection, Dickey-Savage ratio, ridge regression, global-local shrinkage.
result Double descent phenomenon explained through Bayesian interpretation.
We prove grokking in ridge regression, showing overfitting doesn't guarantee good generalization.
problem The onset of generalization long after overfitting in over-parameterized linear regression models.
method Proved end-to-end grokking results for learning over-parameterized linear regression models using gradient descent with weight decay.
result Generalization error eventually becomes arbitrarily small, but poor generalization persists long after overfitting.
Gradient descent recovers low-rank matrices from corrupted measurements with double over-parameterization.
problem Robust recovery of low-rank matrices from grossly corrupted measurements.
method Gradient descent with discrepant learning rates for double over-parameterized models.
result Gradient descent with discrepant learning rates provably recovers the underlying matrix without prior knowledge on rank or sparsity.
Supporting evidence for adaptive feature program across diverse models.
problem Analyzing feature learning in neural networks.
method Over-parameterized sequence models and feature error measure (FEM).
result FEM is decreasing during training of adaptive feature models.
New method improves deep learning models robustness to label noise.
problem Improving deep learning models' robustness to corrupted labels.
method Sparse over-parameterization and implicit regularization.
result State-of-the-art test accuracy against label noise on various datasets.
This work is substituted by the paper in arXiv:2011.14066. Stochastic gradient descent is the de facto algorithm for training deep neural networks (DNNs). Despite its popularity, it still requires fine tuning in order to achieve its best performance. This has led to the development of adaptive methods, that claim autom…
Deep neural networks can learn smooth functions without parameters.
problem Learning smooth functions from shallow ReLU neural networks.
method Using over-parameterized shallow ReLU neural networks with norm constraints.
result Least squares estimators based on shallow neural networks are minimax optimal.
This paper proves SGD converges to global minimum for over-parameterized ReLU networks.
problem Theoretical understanding of implicit neural networks is limited.
method Gradient flow analysis of ReLU activated implicit neural networks.
result Randomly initialized gradient descent converges to global minimum at a linear rate for square loss function in over-parameterized ReLU networks.
Gradient EM converges globally for over-parameterized Gaussian mixtures.
problem Global convergence of gradient EM for Gaussian mixtures with more than 2 components.
method Likelihood-based convergence analysis framework.
result Gradient EM converges globally with a sublinear rate of O(1/√t).
AdaLoss optimizes adaptive learning rates for efficient convergence in various models.
problem Efficiently optimizing adaptive learning rates for gradient descent methods.
method AdaLoss uses loss function information to dynamically adjust step sizes.
result AdaLoss achieves linear convergence in linear regression and robust global convergence in neural networks.
The paper analyzes how over-parameterization affects GD convergence in matrix sensing problems.
problem Matrix sensing problem with over-parameterized gradient descent.
method Analyzes symmetric and asymmetric parameterizations, provides lower bounds and convergence rates.
result Over-parameterization slows down GD convergence, but asymmetric parameterization can speed up convergence.
Proposes a new approach to generate sparse models from deep networks.
problem Training small networks can get stuck in local optima; over-parameterized models are preferred.
method Differential inclusion paths to generate a family of models from simple to complex.
result Algorithm converges to a critical point of empirical risks from any initializations.
New method to recover over-parameterized models corrupted during estimation.
problem Recovering statistical models corrupted after initial estimation.
method Robust estimation using over-parameterized models and redundancy.
result Stochastic gradient descent is well-suited for model repair, but sparsity is generally not repairable.
Optimal rates for shallow ReLU networks in nonparametric regression.
problem Approximating smooth and non-smooth functions with shallow ReLU networks.
method Analysis of shallow ReLUk neural networks, using variation norms and deep learning theory. result Optimal approximation rates for shallow ReLU networks in nonparametric regression.