Geometric analysis shows gradient descent in linear neural nets converges to global minima.
problem Analyzing convergence of gradient descent in linear neural networks.
method Geometric framework and invariance property of network structure.
result Gradient descent trajectories converge to global minima for linear neural nets.
New findings on how neural networks generalize with varying dimensions.
problem Understanding how neural networks generalize with different dimensions and noise levels.
method Study of 2-layer ReLU NNs in a classification setting, proving transitions in overfitting types.
result The type of overfitting transitions from tempered to benign as input dimension increases.
Deep heteroskedastic models overfit, showing a phase transition with regularization strength.
problem Overfitting in deep heteroskedastic regression models.
method Theoretical framework based on statistical field theory, empirical verification, and hyperparameter simplification.
result A phase transition in model behavior with varying regularization strength.
Deep networks can overfit benignly but still be vulnerable to adversarial attacks.
problem Adversarial vulnerability of deep neural networks trained with benign overfitting.
method Investigated causes of adversarial vulnerability, identified label noise as a key factor, and explored the impact of training procedures and representation learning.
result Adversarial robustness requires more complex decision boundaries than simple ones, suggesting the need for better representation learning.
Deep learning methods find near-optimal solutions without explicit regularization.
problem Theoretical challenges in understanding deep learning's success.
method Analysis of gradient methods, overparametrization, and implicit regularization.
result Gradient methods can find near-optimal solutions and exhibit excellent predictive accuracy without explicit regularization.
Calibrating a trading rule using a historical simulation (also called backtest) contributes to backtest overfitting, which in turn leads to underperformance. In this paper we propose a procedure for determining the optimal trading rule (OTR) without running alternative model configurations through a backtest engine. We…
Two-layer CNNs can overfit without harm under certain conditions.
problem Understanding when and how overfitting occurs in neural networks.
method Theoretical analysis of a two-layer CNN trained by gradient descent.
result A sharp phase transition between benign and harmful overfitting based on signal-to-noise ratio.
New algorithms reduce overfitting in multiclass classification.
problem Excessive reuse of test datasets in machine learning leads to overfitting, especially in multiclass classification.
method Developed computationally efficient algorithms to reduce overfitting bias in multi-class classification.
result Achieved overfitting bias of Θ(√(k/(mn)), k/n), matching known upper bounds.
Attention models can overfit without harming test performance.
problem Understanding benign overfitting in single-head attention models.
method Analyzing conditions for benign overfitting in a single-head softmax attention model.
result A single-head attention model can overfit without harming test performance under certain conditions.
The paper tackles backtest overfitting in cryptocurrency trading using deep reinforcement learning.
problem Backtest overfitting in deep reinforcement learning for cryptocurrency trading.
method Formulated hypothesis test for overfitting detection, trained agents, estimated overfitting probability, and rejected overfitted agents.
result Less overfitted deep reinforcement learning agents outperformed more overfitted agents and market benchmarks.
ResNet models overfit benignly on Cifar10 but not on ImageNet due to label noise.
problem Understanding why benign overfitting fails in real-world classification tasks with label noise.
method Theoretical analysis of benign overfitting under a mild overparameterization setup.
result Benign overfitting can fail in the presence of label noise, unlike in heavy overparameterization settings.
We prove overfitting in minimal and random NNs, tempering the effect.
problem Overfitting in minimal and random neural networks.
method Analyzing binary weight fitting to noisy data, proving overfitting is tempered.
result The overfitting of minimal and random neural networks is tempered.
Detects overfitting in models trained on test sets.
problem Challenges in verifying overfitting without independent test sets.
method Uses adversarial examples and unbiased error estimates to test for overfitting.
result Correctly identifies overfitting to the training set but not to the test set.
The paper examines how spike strengths and alignments affect overfitting in linear regression models.
problem The impact of spike strengths and alignments on overfitting in linear regression models.
method Characterization of generalization error through exact expressions and analysis of spike strengths, aspect ratio, and target alignment.
result Increasing spike strength can lead to catastrophic overfitting before benign overfitting, especially in well-specified aligned problems.
New insights into why neural networks can overfit without interpolating data.
problem Understanding why neural networks can overfit without interpolating data in fixed dimensions.
method Analyzing the smoothness of estimators and their derivatives.
result Benign overfitting is possible with estimators that have large enough derivatives, not just in high dimensions but also in fixed dimensions.
Framework for understanding overfitting and underfitting using information theory.
problem Understanding and preventing overfitting and underfitting in machine learning.
method Information-theoretic framework measuring algorithm capacity and dataset information transfer.
result Upper-bounding algorithm capacity and establishing its relationship to machine learning quantities.
Physics analogies explain machine learning overfitting control.
problem Understanding and controlling overfitting in machine learning.
method Analogies from physics and biology applied to algorithmic stability and GAN models.
result Physics formulas provide insights into reducing overfitting in machine learning.
New study finds many neural networks are not benignly overfitting.
problem Understanding the behavior of overfitting in neural networks.
method Exploring kernel ridge regression and deep neural networks to identify overfitting behaviors.
result Many interpolating methods, including neural networks, exhibit tempered overfitting rather than benign or catastrophic.
The paper proposes a method to discount backtest PnLs due to in-sample overfitting.
problem In-sample overfitting in backtest-based investment strategies.
method A simple framework to model and quantify in-sample PnL overfitting.
result Computes the appropriate discount factor for PnLs of in-sample investment strategies.
Two-layer ReLU networks can overfit without harm, study finds.
problem Understanding when and how two-layer ReLU networks can overfit without harming generalization.
method Established algorithm-dependent risk bounds for two-layer ReLU convolutional neural networks with label-flipping noise.
result Gradient descent-trained ReLU networks can achieve near-zero training loss and Bayes optimal test risk.
Study on GANs under and overfitting using discriminator unseen data.
problem Understanding and mitigating under and overfitting in GANs.
method Using discriminators unseen by the generator to measure generalization.
result Model capacity of the discriminator affects generator quality; large discriminator capacities do not prevent overfitting.
Overfitting occurs when RL agents correlate rewards with spurious observation features.
problem Overfitting in reinforcement learning due to correlation with spurious observation features.
method Developed a framework to analyze and design synthetic benchmarks from modified observation spaces.
result Agents can overfit to different observation spaces even if the MDP dynamics are fixed.
Study on benign overfitting in leaky ReLUs with moderate input dimensions.
problem Understanding when overfitting is beneficial in neural networks.
method Two-layer leaky ReLU networks trained with hinge loss, considering signal-to-noise ratio.
result Characterization of conditions for benign overfitting based on signal-to-noise ratio.
New method targets sparsity to prevent overfitting in deep nets.
problem Overfitting in deep neural networks with small datasets.
method Targeted sparsity regularization to visualize and counteract overfitting.
result Significant increase in image classification performance without overfitting.
Adversarial training can lead to overfitting without compromising robustness.
problem Explaining benign overfitting in adversarially robust linear classification.
method Theoretical analysis and numerical experiments on adversarial training.
result Adversarially trained linear classifiers can achieve near-optimal risks despite overfitting noisy data.
Paper finds conditions for benign overfitting in neural networks.
problem Benign overfitting in leaky ReLU two-layer neural networks.
method Established directional convergence and classification error bounds.
result Benign overfitting occurs with high probability on mixture data.
New insights into when benign overfitting occurs in linear and classification tasks.
problem Understanding when benign overfitting happens in linear and classification models.
method Analysis of a generic data model and comparison of predictors (minimum-norm interpolating and max-margin).
result The minimum-norm interpolating predictor is biased towards an inconsistent solution, preventing benign overfitting in linear regression.
Empirical study shows GANs overfit and drop modes when training is deterministic.
problem Understanding overfitting and mode drop in GAN training.
method Empirical analysis of GAN training with and without stochasticity.
result GANs overfit and drop modes when training is deterministic.
Paper explains adversarial training's robust overfitting through a minimax game perspective.
problem Adversarial training suffers from robust overfitting after learning rate decay.
method Viewing adversarial training as a dynamic minimax game, analyzing how LR decay breaks balance and leads to overfitting.
result ReBalanced Adversarial Training (ReBAT) alleviates robust overfitting without sacrificing robustness.
Tests for overfitting in machine learning models.
problem Overfitting in high complexity models.
method Hypothesis test using concentration bounds.
result Valid test for identifying overfitting.
Study the cost of overfitting in noisy KRR models.
problem Cost of overfitting in noisy kernel ridge regression.
method An agnostic view of overfitting cost as a function of sample size for any target function, using Gaussian universality ansatz and task eigenstructure.
result Characterization of benign, tempered, and catastrophic overfitting.
New bounds for KRR condition number reveal overfitting phenomena.
problem Characterizing overfitting in KRR with varying kernel spectral decay.
method Derived new bounds for kernel matrices, enhanced test error bounds, and identified feature independence role.
result Identified tempered and catastrophic overfitting phenomena.
A new exploration method for bandit and reinforcement learning.
problem Uncertainty quantification in exploration.
method Residual Overfit Method of Exploration (ROME).
result ROME drives exploration towards actions with overfitting.
Diffusion models don't overfit, contrary to expectations.
problem Understanding generalization in diffusion models.
method Fundamental impossibility results and analysis of score matching.
result Diffusion models exhibit classical U-shaped loss curve, not double descent.
Study shows that ridgeless Gaussian kernel regression overfits even with varying bandwidth or dimensionality.
problem Analyzing overfitting in Gaussian kernel ridgeless regression with varying bandwidth or dimensionality.
method Examined the behavior of minimum norm interpolating solutions for fixed and increasing dimensions under varying bandwidth and sample size.
result Ridgeless solutions are never consistent and can be worse than null predictor with large enough noise, even with varying bandwidth or dimensionality.
Paper tackles temporal overfitting in wind power curve modeling.
problem Temporal overfitting in wind power curve modeling.
method Proposes a Gaussian process-based method to partition and model time-invariant and time-varying components.
result Significant improvement in predicting responses for different time periods.
Meta learning works well with overparameterized models, a phenomenon called 'benign overfitting'.
problem Understanding why overparameterized models perform well in few-shot learning.
method Analyzed the generalization performance of gradient-based meta learning with an overparameterized meta linear regression model.
result Demonstrated that overparameterized meta learning can still generalize well, a phenomenon called 'benign overfitting'.
A new decision tree method avoids overfitting without hyperparameters.
problem Overfitting in decision trees.
method Hyperparameter-free approach that constructs smaller, shallower trees.
result Avoids overfitting without compromising accuracy or training time.
Minimum Description Length prevents overfitting in noisy data.
problem Learning from noisy data with overfitting risk.
method Minimum Description Length learning rule with tempered guarantees.
result Tempered agnostic finite sample learning guarantees and asymptotic behavior characterization.
New quantum kernels avoid overfitting by combining local and global components.
problem Exponential concentration in quantum kernels leads to overfitting.
method Local-global quantum kernels combining small subsystem and full-system measurements.
result Demonstrated benign overfitting in local-global quantum kernels.
Paper tackles catastrophic overfitting in single-step adversarial training.
problem Catastrophic overfitting leads to sudden drop in robust accuracy.
method Proposes a method to prevent overfitting by using all adversarial examples.
result Demonstrates prevention of catastrophic overfitting and improves robustness.
Covariance-Driven Regression Trees reduce overfitting in CART.
problem Overfitting in CART decision trees, especially with small sample sizes.
method Covariance-driven splitting criterion for regression trees (CovRT).
result CovRT achieves superior prediction accuracy compared to CART in simulations and real-world tasks.
Mitigates overfitting in UU classification from two unlabeled datasets.
problem Overfitting in the UU classification method.
method Wrapping negative empirical risk terms with correction functions and proving consistency.
result Successfully mitigates overfitting and improves classification accuracy.
Study analyzes overfitting in neural nets under class imbalance, proposing new loss functions.
problem Overfitting and class imbalance in neural nets, particularly in image segmentation.
method Analyzes logits distribution, derives asymmetric loss functions and regularizers.
result Proposed loss functions improve segmentation performance in brain tumor core.
Recent years have witnessed significant progresses in deep Reinforcement Learning (RL). Empowered with large scale neural networks, carefully designed architectures, novel training algorithms and massively parallel computing devices, researchers are able to attack many challenging RL problems. However, in machine learn…
Linear models can overfit without harming OOD generalization under certain conditions.
problem Understanding how overparameterized linear models generalize to out-of-distribution data.
method Analyzing overparameterized linear models under covariate shift, providing guarantees for OOD generalization.
result Benign overfitting occurs in standard ridge regression under OOD conditions, with specific structural conditions on target covariance.
ARO overfits by making constraints dependent on uncertainty, leading to brittleness.
problem ARO's adaptive policies become brittle when realizations fall outside the uncertainty set.
method Assigning constraint-specific uncertainty set sizes with probabilistic guarantees.
result Regularization through specific uncertainty set sizes ensures stability and flexibility.
One major challenge in training Deep Neural Networks is preventing overfitting. Many techniques such as data augmentation and novel regularizers such as Dropout have been proposed to prevent overfitting without requiring a massive amount of training data. In this work, we propose a new regularizer called DeCov which le…