Overparameterized models are more vulnerable to membership inference attacks.
problem Vulnerability of overparameterized models to membership inference attacks.
method Theoretical and empirical analysis of overparameterized linear and ridge-regularized linear regression models in the Gaussian data setting.
result Increased number of parameters and model complexity increase vulnerability to membership inference attacks.
Overparameterized MLR fits hyper-curves, improving model robustness.
problem Improper predictors degrade model generalizability.
method Parameterizing with a scalar and monomial basis, fitting hyper-curves.
result Hyper-curve approach yields robust predictions for noisy data.
Estimates generalization gap for overparameterized models using Langevin approximation.
problem Estimating the difference between training and generalization performance in overparameterized models.
method Functional variance and Langevin approximation of functional variance.
result Demonstrates efficient estimation of generalization gaps for overparameterized models.
Overparameterized models generalize well despite fitting noisy data.
problem Understanding why overparameterized models generalize well despite fitting noisy data.
method Statistical signal processing perspective.
result Overparameterized models often outperform underparameterized models in test performance.
The paper shows how data and algorithm interactions affect overparameterized linear regression generalization.
problem Understanding generalization in overparameterized linear regression.
method Introducing data-algorithm compatibility and performing data-dependent trajectory analysis with gradient descent.
result Early stopping iterates lead to better generalization than last-iterate analysis, with weaker restrictions.
Meta learning works well with overparameterized models, a phenomenon called 'benign overfitting'.
problem Understanding why overparameterized models perform well in few-shot learning.
method Analyzed the generalization performance of gradient-based meta learning with an overparameterized meta linear regression model.
result Demonstrated that overparameterized meta learning can still generalize well, a phenomenon called 'benign overfitting'.
Overparameterization helps prevent forgetting in sequential learning tasks.
problem Catastrophic forgetting in continual learning systems.
method Analytical study of gradient descent with linear regression model.
result Overparameterization can mitigate forgetting in a two-task setting.
Bayesian method improves predictions in overparameterized nonlinear regression.
problem Understanding overparameterization in nonlinear regression models.
method Bayesian framework with adaptive prior considering data spectral structure.
result Posterior contraction established for generalized linear and single-neuron models, demonstrating prediction consistency.
Gradient descent converges linearly for overparameterized linear networks.
problem Convergence of gradient descent for overparameterized neural networks.
method Local Polyak-Lojasiewicz and Descent Lemma for overparameterized linear models.
result Gradient descent achieves linear convergence for two-layer linear networks under relaxed assumptions.
Local SGD proves efficient in overparameterized linear regression.
problem Efficiently learning overparameterized linear models in distributed settings.
method Distributed SGD (DSGD) with overparameterized models.
result Excess risk of SGD is smaller than ridge regression in the same sample complexity.
Study shows DNNs can recover functions with fewer samples than model parameters at overparameterization.
problem Determining reliable function recovery in overparameterized deep neural networks.
method Introducing 'local linear recovery' (LLR) and proving upper bounds on sample sizes for recovery.
result Upper bounds on optimistic sample sizes for function recovery in overparameterized DNNs are achieved.
Gradient method converges locally linearly for overparameterized Gaussian mixtures.
problem Learning Gaussian mixtures under overparameterization.
method Gradient-based method alternating short descent steps and long Polyak steps.
result Gradient method converges locally linearly to minimizers.
Overparameterized linear model shows strong classification but weak regression, susceptible to adversarial perturbations.
problem Adversarial vulnerability of overparameterized linear models.
method Lifted Fourier feature map, analyzing overparameterized linear ensemble.
result Spatial localization leads to adversarial vulnerability in an intermediate classification regime.
Analyzes bias-variance in overparameterized linear models using random features.
problem Understanding bias-variance trade-off in overparameterized models.
method Zero-temperature cavity method and random matrix theory.
result Three phase transitions in the linear random features model.
Study shows overparameterization helps in generalizing from smooth interpolants.
problem Understanding generalization in overparameterized linear models.
method Analysis of random Fourier series model with weighted trigonometric interpolation.
result Weighted trigonometric interpolation leads to lower generalization error in overparameterized scenarios.
The paper analyzes how overparameterized models can generalize well in multiclass classification.
problem Generalization in multiclass classification with overparameterized models.
method Survival/contamination analysis framework adapted for multiclass classification.
result Multiclass classification can generalize well even with many classes, unlike regression tasks.
Task shift from classification to regression is possible in overparameterized linear models with limited additional data.
problem Transferability of latent knowledge from classification to regression in overparameterized linear models.
method Investigation of task shift in overparameterized linear regression, zero-shot and few-shot cases, with a focus on minimum-norm interpolation.
result Minimum-norm interpolators can transfer latent knowledge from classification to regression with limited additional data.
Weight normalization speeds up matrix sensing problems.
problem Matrix sensing with overparameterization.
method Generalized weight normalization with Riemannian optimization.
result WN achieves linear convergence, improving speed and complexity.
We reveal a model rank that predicts successful recovery of target functions at overparameterization.
problem Understanding the mysterious good generalization performance of overparameterized nonlinear models.
method Rank stratification and linear stability theory for general nonlinear models.
result Linearly stable functions are preferred by nonlinear training, and model rank predicts minimal training data size.
Overparameterized models improve performance in sequential learning tasks.
problem Catastrophic forgetting in overparameterized neural networks.
method Two-task linear regression problem with random orthogonal transformations.
result Overparameterization mitigates catastrophic forgetting in sequential learning tasks.
Efficiently compress overparameterized deep models by focusing on low-dimensional learning dynamics.
problem Overparameterized models increase computational and memory costs.
method Study of learning dynamics reveals updates occur within a low-dimensional subspace, leading to a compression algorithm.
result Compressed deep linear networks converge faster and yield smaller recovery errors.
Overparameterized models can worsen minority group errors even when overall test error improves.
problem Overparameterization exacerbates spurious correlations, harming minority groups.
method Simulations and experiments on image datasets, theoretical analysis of linear models.
result Subsampling the majority group can achieve low minority error in overparameterized models.
We analyze optimal weighted ridge regression in overparameterized linear models.
problem Optimal regularization in overparameterized linear regression models.
method Generalized ridge regression with weighted regularization.
result The optimal regularization parameter can be negative in overparameterized settings.
This paper explains how overparameterization aids in meta-learning with few samples.
problem Building a generalizable model with few samples in meta-learning.
method Analyzes the optimal linear representation and sample complexity for meta-learning tasks.
result Overparameterization naturally answers fundamental meta-learning questions, reducing sample complexity.
Study resolves conjecture on overparameterized linear models' generalization.
problem Asymptotic generalization of multiclass classification with overparameterized models.
method Gaussian covariates bi-level model, Hanson-Wright inequality variant.
result Min-norm interpolating classifier can be suboptimal compared to noninterpolating classifiers.
Last SGD iterate bounds for overparameterized linear regression.
problem Analyzing the last iterate risk bounds of SGD with decaying stepsize for overparameterized linear regression.
method Problem-dependent analysis of last iterate risk bounds of SGD with geometrically decaying stepsize.
result Proved nearly matching upper and lower bounds on the excess risk for last iterate SGD with geometrically decaying stepsize.
PCA-based dimensionality reduction improves robustness in overparameterized linear models.
problem Improving robustness in overparameterized linear models.
method PCA-based dimensionality reduction (PCA-OLS)
result PCA-OLS can achieve better generalization than ordinary least squares (OLS) in the overparameterized regime.
The paper analyzes how gradient descent implicitly regularizes solutions in overparameterized neural networks, revealing depth-dependent regularization effects.
problem Understanding implicit regularization in overparameterized linear neural networks for regression problems.
method Analyzing the approximation error between gradient flow limit points and ℓ1-minimization solutions, deriving tight upper and lower bounds. result The approximation error decreases linearly for D≥3 and at a slower rate for D=2, linked to null space property constants. This work analyzes how overparameterization aids GANs in reaching global saddle points.
problem Understanding the role of overparameterization in GANs for convergence to global saddle points.
method Theoretical and empirical analysis of overparameterized GANs with various architectures and datasets.
result GDA converges to a global saddle point in overparameterized GANs with certain assumptions.
ASGD outperforms SGD in overparameterized linear regression, especially in subspaces of small eigenvalues.
problem Generalization of ASGD for overparameterized linear regression.
method Established instance-dependent excess risk bound for ASGD in each eigen-subspace of the data covariance matrix.
result ASGD outperforms SGD in subspaces of small eigenvalues, exhibiting faster decay of bias error.
Optimal machine learning requires interpolating training data in high-dimensional linear regression.
problem Achieving optimal predictive risk in overparameterized linear regression models.
method Analyzing proportional asymptotics of random design and label noise variance.
result Optimal performance in linear regression requires fitting training data to higher accuracy than inherent noise.
A new tradeoff between regularization and sharpness improves model performance in overparameterized settings.
problem Improving model performance in overparameterized settings with minimum-norm interpolators.
method Proposes a regularization-sharpness tradeoff for overparameterized linear regression with an ℓ^p penalty.
result Empirical validation shows the tradeoff terms can distinguish performant linear interpolators.
Classification and regression tasks in overparameterized models show different generalization properties.
problem Comparing classification and regression in overparameterized models.
method Comparison of least-squares minimum-norm interpolation and hard-margin SVM using different loss functions.
result Interpolating solutions generalize well with 0-1 loss but not with square loss.
This work analyzes how bottleneck layers and skip connections affect linear denoising autoencoders' generalization.
problem Understanding the generalization of linear denoising autoencoders in overparameterized regimes.
method Analyzes two-layer linear denoising autoencoders with a bottleneck layer and skip connection, deriving test risk formulas.
result Bottleneck layers introduce an additional complexity measure, while skip connections can mitigate variance.
Removing spurious features can hurt model accuracy and disproportionately affect different groups.
problem Interference from spurious features in robust model performance across different groups.
method Characterization and analysis of spurious feature removal in noiseless overparameterized linear regression.
result Removal of spurious features can decrease accuracy and disproportionately affect different groups, even in balanced datasets.
Paper studies asymmetric matrix sensing, proving gradient descent converges to low-rank solutions.
problem Reconstructing asymmetric low-rank matrices from linear measurements.
method Factorized gradient descent with coupling and regularization properties.
result Gradient descent from small random initialization converges to globally optimal and generalizing solutions.
CPCR mitigates bias in PCR for overparameterized models.
problem Bias in Principal Component Regression (PCR) for overparameterized models.
method Calibrated Principal Component Regression (CPCR) learns a low-variance prior in the PC subspace and calibrates the model in the original feature space.
result CPCR outperforms standard PCR in overparameterized settings, improving prediction across multiple problems.
Overparameterization aids in model pruning, leading to improved test accuracy.
problem Improving lightweight model performance through pruning.
method Theoretical analysis and high-dimensional asymptotics of model pruning in overparameterized neural networks.
result Even with known informative features, training a large model and then pruning leads to better test accuracy.
Gradient descent with preconditioning finds global optima in overparameterized nonconvex factorization.
problem Finding global optima in nonconvex Burer-Monteiro factorization.
method Preconditioned gradient descent for overparameterized nonconvex function minimization.
result Gradient descent with preconditioning achieves linear convergence in the overparameterized case.
Overparameterization enhances SAM's effectiveness in minimizing sharpness.
problem Improving generalization in deep neural networks.
method Analysis of Sharpness-Aware Minimization (SAM) under varying degrees of overparameterization.
result Overparameterization significantly improves SAM's performance, particularly in noisy and sparse settings.
New loss function restores importance weighting in overparameterized models.
problem Restoring importance weighting in overparameterized neural networks.
method Introduced polynomially-tailed losses to restore effects of importance weighting.
result Polynomially-tailed losses improve performance in correcting distribution shift.
In this expository note we describe a surprising phenomenon in overparameterized linear regression, where the dimension exceeds the number of samples: there is a regime where the test risk of the estimator found by gradient descent increases with additional samples. In other words, more data actually hurts the estimato…
Lower bounds show OLS outperforms basis pursuit in overparameterized linear regression.
problem Excess risk of sparse interpolating procedures in overparameterized linear regression.
method Proved lower bounds on excess risk for OLS and basis pursuit.
result Excess risk of basis pursuit can converge at an exponentially slower rate than OLS.
Quantum models show improved performance in overparameterized regimes.
problem Overfitting in quantum machine learning models.
method Analytical demonstration and numerical experiments on quantum kernel methods.
result Quantum models can operate in the modern, overparameterized regime without overfitting.
A new complexity measure MDL-COMP for overparameterized models improves generalization performance.
problem Complexity measures based on Rissanen's MDL principle are not well-suited for overparameterized models.
method Developed a novel MDL-based complexity (MDL-COMP) for overparameterized models, defined via an optimality criterion over Ridge estimators.
result MDL-COMP scales linearly with d when d<n, but exponentially smaller for d>n; it upper bounds in-sample MSE. Gradient methods work well on overparameterized diagonal linear networks.
problem Understanding why gradient-based methods work well in overparameterized models.
method Study of Deep Diagonal Linear Networks with gradient flow analysis.
result Gradient flow on layer parameters induces a mirror-flow dynamic in the effective parameter space, leading to explicit convergence guarantees.
Unified analysis of parameter norms in overparameterized linear models, revealing scaling laws and thresholds.
problem Understanding the scaling of parameter norms in overparameterized linear models.
method Simple dual-ray analysis revealing competition between signal spike and bulk of null coordinates.
result Unified closed-form predictions for parameter norm scaling, including elbow and threshold laws.
Many statistical estimators for high-dimensional linear regression are M-estimators, formed through minimizing a data-dependent square loss function plus a regularizer. This work considers a new class of estimators implicitly defined through a discretized gradient dynamic system under overparameterization. We show that…