Double descent phenomenon explained in simple terms.
problem Understanding the surprising drop in test error in overparameterized models.
method Informal explanation using linear algebra and probability, visual intuition with polynomial regression, mathematical analysis with ordinary linear regression.
result Three factors create double descent: data undersampling, model size, and parameter count. Ablating any one of these factors prevents double descent.
This paper explains why double descent sometimes occurs weakly or not at all from an optimization perspective.
problem Understanding the role of optimization in the phenomenon of double descent.
method Investigates model-wise double descent from an optimization perspective, proposing a unified explanation for its occurrence.
result Model-wise double descent is observed if and only if the optimizer can find a sufficiently low-loss minimum.
Double descent risk in L2-regularized models explained and mitigated.
problem Risk of overparameterized models in machine learning.
method Analysis of L2-regularized models, two-layer neural networks, and CNNs.
result Double descent risk in L2-regularized models can be explained and mitigated by adjusting regularization strengths.
This paper explains double descent in linear neural networks, identifying new factors.
problem Understanding double descent in linear neural networks.
method Gradient flow derivation and necessary conditions for double descent.
result Singular values of input-output covariance matrix are important for double descent in two-layer models.
We analyze double descent in finite-width neural networks using influence functions.
problem Understanding double descent in finite-width neural networks.
method Using influence functions to derive population loss bounds and investigate loss function effects.
result Derived bounds exhibit double descent behavior at the interpolation threshold.
New insights into double descent phenomenon in neural networks.
problem Understanding the double descent behavior in deep learning models.
method Linear teacher-student setup and tools from statistical physics.
result Distinct features are learned at different scales, leading to epoch-wise double descent.
Study shows double descent curve in high-dimensional linear regression with random projections.
problem Understanding the generalization performance in high-dimensional settings with random projections.
method Fixed prediction problem, ridge regression estimator, minimum norm least-squares fit, random matrix theory, asymptotic equivalents.
result Exhibit a double descent curve for high-dimensional linear regression with random projections.
Optimal regularization can prevent the double descent phenomenon in learning models.
problem The double descent phenomenon in learning models, where test performance is non-monotonic in sample size and model size.
method Theoretical and empirical study of optimal ℓ2 regularization for linear regression models and neural networks. result Optimally-tuned ℓ2 regularization achieves monotonic test performance for certain models and mitigates the double descent phenomenon for more general models. Least squares regression shows unexpected double descent in under-parameterized models.
problem Understanding the generalization of under-parameterized models in regression.
method Analyzing the spectrum and eigenvectors of the sample covariance matrix.
result Least squares regression can exhibit a peak in generalization in the under-parameterized regime, contrary to previous explanations.
Double descent observed in tree-based models for genomic prediction.
problem Understanding the generalization behavior of tree-based models in machine learning.
method Systematic variation of model complexity in a genomic prediction task using whole-genome sequencing data.
result Double descent emerges only when complexity is scaled jointly across learner capacity and ensemble size.
Deep learning models can generalize well even when they fit training data perfectly.
problem Generalization in over-parameterized deep learning models.
method Combining empirical risk minimization with capacity control, exploring inductive biases and smooth empirical risk minimizers.
result Double descent phenomenon: test error can decrease after interpolation point.
This paper explains double descent using VC theory.
problem Understanding the generalization of overfitting neural networks.
method VC-theoretical analysis of double descent.
result Double descent can be explained by classical VC-generalization bounds.
Bayesian interpretation explains double descent in deep learning models.
problem Understanding the risk function behavior of over-parameterized models.
method Bayesian model selection, Dickey-Savage ratio, ridge regression, global-local shrinkage.
result Double descent phenomenon explained through Bayesian interpretation.
We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We u…
Study on double descent behavior in two-layer neural networks for binary classification.
problem Understanding the double descent phenomenon in model test error.
method Two-layer neural network with ReLU activation for binary classification. Quantified model size by sample-to-dimension ratio. Empirical risk minimization using Convex Gaussian Min Max Theorem.
result Observed and investigated the double descent behavior of model test error.
Improves early stopping in deep networks by adjusting stepsizes.
problem Epoch-wise double descent in deep networks.
method Analytical and empirical study of bias-variance tradeoffs in different network layers.
result Eliminating epoch-wise double descent through adjusting stepsizes of different layers improves early stopping performance.
Quantum models show improved performance in overparameterized regimes.
problem Overfitting in quantum machine learning models.
method Analytical demonstration and numerical experiments on quantum kernel methods.
result Quantum models can operate in the modern, overparameterized regime without overfitting.
Deep learning models can overfit noisy data without losing generalization.
problem Understanding the generalization of deep learning models in noisy data.
method Empirical investigation of epoch-wise double descent in fully connected neural networks trained on CIFAR-10 with 30% label noise.
result The model achieves strong re-generalization on test data after overfitting noisy training data, corresponding to a 'benign overfitting' state.
Paper connects neural networks to Gaussian processes for understanding double-descent.
problem Understanding the double-descent phenomenon in neural networks.
method Uses techniques from random matrix theory and Gaussian processes.
result Establishes a connection between NNGP and random matrix theory for neural networks.
Study shows double and triple descent in unsupervised autoencoders, improving performance in various tasks.
problem Exploring the phenomenon of double descent in unsupervised learning.
method Analytical demonstration and extensive experiments on synthetic and real datasets.
result Over-parameterized unsupervised autoencoders exhibit double and triple descent, enhancing performance in downstream tasks.
New findings challenge the traditional U-shaped curve of model complexity and error, revealing a second descent in error as model size increases.
problem The traditional U-shaped curve of model complexity and prediction error is incomplete, with recent work suggesting a second descent in error as model size increases.
method Careful consideration of multiple complexity axes and a nonparametric statistics perspective were used to interpret the observed double descent curves.
result The observed double descent curves in classical statistical machine learning methods fold back into traditional convex shapes, resolving tensions with statistical intuition.
Regression models can interpolate noisy data and still perform well, contrary to the bias-variance tradeoff.
problem Understanding why overparametrized models can generalize well despite the bias-variance tradeoff.
method Analysis of minimum norm solutions and ridge regression, focusing on the smallest singular value of the regression matrix.
result Testing error exhibits double descent behavior as model order increases, contrary to the classical bias-variance tradeoff.
Study on RF regression with SGD shows double descent phenomenon.
problem Understanding generalization in RF models trained with SGD.
method Precise non-asymptotic error bounds derived for RF regression under constant and polynomial-decay step-size SGD.
result RF regression generalizes well for interpolation learning and exhibits double descent behavior.
Hybrid regularization avoids double descent in random feature models.
problem Avoiding the double descent phenomenon in random feature models.
method Combines early stopping and weight decay, using GCV for hyperparameter selection.
result Hybrid method successfully avoids double descent and achieves comparable generalization.
GNNs generalize better on homophilic graphs than heterophilic ones.
problem Understanding the generalization error of GNNs on graph data.
method Analytical tools from statistical physics and random matrix theory.
result Risk is shaped by graph noise, feature noise, and training labels.
Double descent in portfolio optimization shows improved performance with complexity, then declines, due to overfitting.
problem Improving portfolio optimization performance with model complexity.
method Investigates the relationship between model complexity and out-of-sample performance in mean-variance portfolio optimization.
result Performance of low-dimensional models initially improves with complexity but declines due to overfitting. High-dimensional models show double ascent Sharpe ratio curve.
Double descent found in DRL, improving generalization with model capacity.
problem Generalization in over-parameterized DRL models.
method Actor-Critic framework, Policy Entropy metric.
result Policy Entropy significantly reduces as model capacity increases, indicating improved generalization.
Study shows how optimization affects deep neural network performance as model size increases.
problem Understanding the performance of deep neural networks as model size increases.
method Careful study of learning dynamics for least squares scenario, providing an excess risk bound.
result Excess risk bound depends on the smallest non-zero eigenvalue of the covariance matrix of input features, showing double descent behavior.
The paper explores how overfitting can lead to better predictions in high-dimensional data.
problem Understanding the behavior of linear models in high-dimensional settings with more predictors than observations.
method Analysis of ordinary least squares, penalized least squares, and spectral shrinkage estimates.
result The phenomenon of double descent, where model performance can improve with increasing model complexity.
New method estimates precision matrices without models, achieving dense, consistent, and model-free properties.
problem Lack of methods that are dense, consistent, and model-free for precision matrix estimation.
method General class of estimators that unify dense, consistent, and model-free properties within a nonasymptotic framework.
result Ridgeless regression exhibits the double descent phenomenon, establishing a precision matrix analogue to linear regression's double descent.
Changing kernel bandwidth during training improves kernel regression performance.
problem Improving kernel regression performance with varying model complexity.
method Investigated changing the bandwidth of a translational-invariant kernel during training for kernel regression using gradient descent.
result Kernel regression exhibits double descent behavior with decreasing model complexity (bandwidth).
Characterizes RFF regression in large n,p,N setting, providing precise learning phases and double descent curve.
problem Characterizes RFF regression in large n,p,N setting. method Characterizes the exact asymptotics of random Fourier feature (RFF) regression in the realistic setting of large n,p,N. result Characterizes two qualitatively different phases of learning and the corresponding double descent test error curve.
Double descent refers to the phase transition that is exhibited by the generalization error of unregularized learning models when varying the ratio between the number of parameters and the number of training samples. The recent success of highly over-parameterized machine learning models such as deep neural networks ha…
Study on ridge regression in convolutional models shows double descent error behavior.
problem Understanding generalization and estimation error in over-parameterized convolutional models.
method Analysis of ridge estimators for convolutional linear models, derivation of exact error formulae.
result Ridge estimators exhibit double descent error behavior in high-dimensional convolutional models.
The paper explores how model complexity affects OOD detection performance.
problem Ensuring reliability and safety of machine learning systems through OOD detection.
method Investigates the relationship between model capacity and OOD detection performance using empirical and theoretical analysis.
result The Double Descent phenomenon is observed in post-hoc OOD detection, indicating that overparameterization can enhance OOD detection.
Study reveals how model volume affects learning curves in machine learning.
problem Understanding the double descent risk phenomenon in machine learning.
method Investigates the role of model volume using MDL, Occam's Razor, and information geometry.
result Model volume can explain the double descent risk, suggesting better generalization with increased dimensionality.
Distillation works even with hard labels from overparameterized teacher, leading to better performance.
problem Improving model performance with hard labels from overparameterized teacher.
method Training a student model on a large held-out dataset labeled by a highly overparameterized teacher.
result Student model outperforms traditional approaches due to double descent phenomenon.
The paper analyzes how over-parameterization affects reinforcement learning performance.
problem Understanding the impact of over-parameterization in reinforcement learning.
method Theoretical analysis of Least-Square Temporal Difference (LSTD) algorithm with random features and asymptotic regime.
result Identification of a double descent phenomenon in reinforcement learning performance.
Recently, there have been significant interests in studying the so-called "double-descent" of the generalization error of linear regression models under the overparameterized and overfitting regime, with the hope that such analysis may provide the first step towards understanding why overparameterized deep neural netwo…
The paper analyzes high-dimensional kernel regression, showing different risk curves based on data and regularization.
problem Characterizing generalization properties of high-dimensional kernel ridge regression.
method Bias-variance decomposition of the expected excess risk, considering different regularization schemes and data eigen-profiles.
result The risk curve of kernel regression can be double-descent-like, bell-shaped, or monotonic, depending on n, d, and regularization level.
We analyze the condition number of random feature matrices and prove their well-conditioned nature.
problem Understanding the condition number of random feature matrices and its impact on generalization error.
method Established concentration bounds and derived risk bounds for regression problems using random feature matrices.
result The risk associated with random feature matrices exhibits the double descent phenomenon, improving even with noise.
New insights into neural network complexity reveal better generalization performance.
problem Mysterious generalization in deep models despite high parameter counts.
method Effective dimensionality as a measure of parameter space complexity.
result Double descent behavior in generalization as a function of parameters explained.
Monotonic improvement in uncertainty estimation with Gaussian processes as dimension increases.
problem Uncertainty quantification in machine learning models, especially with Gaussian processes, is challenging and poorly understood.
method Analyzing the behavior of marginal likelihood and cross-validation metrics as input dimension increases, and exploring the effects of cold posteriors.
result The marginal likelihood improves monotonically with input dimension, while cross-validation metrics exhibit double descent behavior.
Random Forests don't overfit, challenging the double-descent theory.
problem The overfitting behavior of Random Forests.
method Challenged the double-descent curve theory by analyzing the performance of Random Forests and proposing a new approach to diversity in ensembles.
result Random Forests do not exhibit a double-descent curve but rather a single descent, indicating they do not overfit in the classic sense.
New analysis reveals optimal regularization for ESNs, avoiding double descent.
problem Characterizing and optimizing Echo State Networks (ESNs) for precise bias-variance.
method Random matrix theory applied to ESNs in a teacher-student setting.
result ESNs achieve lower MSE with limited training samples and teacher memory.
The "double descent" risk curve was proposed to qualitatively describe the out-of-sample prediction accuracy of variably-parameterized machine learning models. This article provides a precise mathematical analysis for the shape of this curve in two simple data models with the least squares/least norm predictor. Specifi…
Theory and method for reducing prediction variance in noisy feature-subsampled ridge ensembles.
problem Reduction of prediction variance in noisy data with feature bagging.
method Developed analytical learning curves for noisy ridge ensembles, introduced heterogeneous feature ensembling.
result Subsampling shifts the double-descent peak, leading to improved performance over a single linear predictor.
Analyzes generalization error in generalized linear models, explaining double descent phenomenon.
problem Understanding generalization of machine learning models in high dimensions.
method Develops a framework to characterize asymptotic generalization error for generalized linear models.
result Rigorously explains the double descent phenomenon in generalized linear models.