Study on how initialization scale affects neural network training regimes.
problem Understanding the transition between kernel and rich regimes in overparametrized models.
method Analysis of simple depth-D models and empirical testing on complex models.
result Scale of initialization controls transition between kernel and rich regimes.
It has been observed \citep{zhang2016understanding} that deep neural networks can memorize: they achieve 100\% accuracy on training data. Recent theoretical results explained such behavior in highly overparametrized regimes, where the number of neurons in each layer is larger than the number of training samples. In thi…
Deep neural networks can interpolate any dataset in the overparametrized regime.
problem Interpolating any dataset with deep neural networks in the overparametrized regime.
method Proving universal approximations and interpolating any dataset with deep neural networks, considering specific conditions on activation functions.
result Interpolation of any dataset is possible in the overparametrized regime with deep neural networks.
Data splitting enhances model performance in overparametrized ridgeless regression.
problem Computational inefficiency in training models with large datasets.
method Data splitting as a regularization technique in overparametrized ridgeless regression.
result Data splitting improves statistical performance and computational complexity.
Overparametrized models are vulnerable to adversarial perturbations, affecting robust generalization.
problem Understanding how overparametrization impacts robustness in adversarial training.
method Analyzing random features regression models with a precise asymptotic formula.
result High overparametrization can hurt robust generalization in adversarially trained models.
A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus training with gradient descent has the effect of finding the minimum RKHS norm solution. This stands in contrast to other studies which demonstr…
Deep networks become equivalent to linear models in large data regimes.
problem Understanding the behavior of deep neural networks in large data regimes.
method Information-theoretic analysis of fully-trained neural networks in proportional scaling regime.
result Proves deep Gaussian equivalence principle, showing deep networks can be simplified to linear models.
New neural network class reduces VC dimension, leading to better generalization.
problem VC theory struggles with explaining small generalization errors in overparametrized neural networks.
method Developed hyperplane arrangement neural networks (HANNs) and used sample compression analysis.
result HANNs can have significantly smaller VC dimension than the number of weights, yet remain highly expressive.
Improved generative models using overparametrized shallow neural networks.
problem Improving generative models for data with hidden low-dimensional structure.
method Using energy-based models with overparametrized shallow neural networks as approximators.
result Models trained in the 'active' regime outperform those in the 'lazy' or kernel regime, leading to better adaptivity to hidden structure.
The study examines how posterior drift affects forecasting accuracy in overparametrized models, particularly in financial markets.
problem Impact of posterior drift on out-of-sample forecasting accuracy in overparametrized models.
method Investigation of posterior drift and its effect on model performance in financial markets.
result Overparametrized models can be sensitive to sub-periods and bandwidth parameters, leading to inconsistent returns.
Stochastic gradient method converges as fast as deterministic for overparametrized models.
problem Convergence rate of stochastic gradient methods in overparametrized models.
method Proposes a regularity condition enabling fast convergence of SGD.
result Stochastic gradient method achieves the same convergence rate as deterministic gradient method.
Overparametrization improves QNN trainability by reducing spurious local minima.
problem Understanding how overparametrization affects the loss landscape of QNNs.
method Rigorous analysis of overparametrization in QNNs with periodic structure.
result Overparametrization corresponds to a computational phase transition improving QNN trainability.
Neural networks can interpolate random data but still generalize well, studied in the NT regime.
problem Understanding how neural networks interpolate random labels and generalize well in the overparametrized regime.
method Characterization of the eigenstructure of the empirical NT kernel and generalization error of NT ridge regression.
result The generalization error is well approximated by polynomial ridge regression with an increased regularization parameter.
Dual training method for EBMs with overparametrized neural networks.
problem Training EBMs with non-convex energies is challenging.
method Derive variational principles and dual GDA algorithm for feature-learning regime.
result Dual GDA algorithm performs best with similar time scales for features and particles.
This work investigates square loss in overparametrized neural networks, revealing its advantages in robustness and calibration.
problem Theoretical understanding of square loss in overparametrized neural networks.
method Systematic investigation of square loss in the NTK regime for both separable and non-separable classes.
result Square loss shows fast convergence rates and robustness guarantees for overparametrized neural networks.
MSGD outperforms SGD in overparametrized settings with faster convergence rates.
problem Optimization of non-convex functions with momentum.
method Momentum Stochastic Gradient Descent (MSGD) with rigorous analysis.
result MSGD converges exponentially faster than SGD in overparametrized settings.
Deep networks generalize well even when they fit training data perfectly, thanks to overparametrization.
problem Understanding generalization in overparametrized deep networks.
method Random features regression, asymptotic analysis, ensemble averaging.
result Bias remains constant beyond the interpolation threshold, while variance components decay with overparametrization.
New theory explains how overparametrized neural networks generalize well without bias-variance trade-off.
problem Overparametrized neural networks generalize well despite classical bias-variance trade-off.
method Nonasymptotic generalization theory for two-layer neural networks with ReLU activation, incorporating scaled variation regularization.
result Prediction bounds for all network widths reproduce the double descent phenomenon, and overparametrized models are nearly minimax optimal.
Gradient descent in tensor factorization favors low-rank solutions.
problem Tackling implicit regularization in tensor factorization problems.
method Gradient descent with small random initialization for overparametrized tensor factorization.
result Gradient descent leads to implicit regularization towards low tubal rank solutions.
Study how data structure impacts classification performance in overparametrized models.
problem Understanding the role of data structure in overparametrized models.
method Analytical and statistical physics methods applied to structured data.
result Data structure significantly affects the performance of overparametrized models, particularly for logistic loss.
Deep learning methods find near-optimal solutions without explicit regularization.
problem Theoretical challenges in understanding deep learning's success.
method Analysis of gradient methods, overparametrization, and implicit regularization.
result Gradient methods can find near-optimal solutions and exhibit excellent predictive accuracy without explicit regularization.
Paper presents a new training method for overparametrized neural networks that reduces time per iteration.
problem Scalability issue in training overparametrized neural networks.
method Uses a new view of neural networks as binary search trees, modifying a small subset of nodes per iteration.
result Reduces amortized time per iteration to m1−αnd+n3 from previous mnd+n3. A recent line of work has shown that an overparametrized neural network can perfectly fit the training data, an otherwise often intractable nonconvex optimization problem. For (fully-connected) shallow networks, in the best case scenario, the existing theory requires quadratic over-parametrization as a function of the …
Downsampling can improve generalization in ridgeless linear regression, especially with optimal sketching size.
problem Improving generalization in ridgeless linear regression with limited data.
method Investigating the effects of downsampling on the sketched ridgeless least square estimator in the proportional regime.
result Optimal sketching size minimizes out-of-sample prediction risks and stabilizes risk curves.
The paper analyzes the implicit bias of SGD near loss manifold and provides new insights.
problem Understanding the implicit bias of SGD near loss manifolds in overparametrized models.
method Adapting ideas from Katzenberger (1991) to analyze SGD dynamics using a stochastic differential equation (SDE).
result SGD with label noise locally decreases the sharpness of loss, leading to a global analysis of implicit bias.
Study shows overparametrization can shift and bend loss landscapes, affecting signal recovery.
problem Understanding how overparametrization affects loss landscapes in neural networks.
method Field theory analysis of Hessian spectrum at initialization.
result Overparametrization can shift the BBP transition point, potentially reaching weak-recovery threshold.
New method interpolates training data and is consistent for various data distributions.
problem Establishing generalization guarantees for ensemble methods in the interpolating regime.
method Developed manifold-Hilbert kernel for Riemannian manifolds and used it in ensemble classification.
result Consistent ensemble classification method for broad data distributions.
Noise can affect the overparametrization of QNNs, enabling new directions but also suppressing sensitivity.
problem The overparametrization of QNNs in the presence of noise.
method Analyzing the Quantum Fisher Information Matrix (QFIM) to understand how noise affects the rank of QFIM.
result Noise can turn previously-zero eigenvalues of the QFIM to non-zero, enabling exploration of new directions.
Zero loss is achievable in overparametrized DL networks under specific conditions.
problem Achieving zero loss in overparametrized deep learning networks.
method Determine sufficient conditions for zero loss attainability and present an explicit construction of zero loss minimizers.
result Explicit minimizers for zero loss in overparametrized DL networks are constructed without gradient descent.
This work explores how overparametrization and priors affect Bayesian neural network posteriors.
problem Symmetries, non-identifiabilities, and weight-space priors fragment and inflate BNN posteriors.
method We study the interplay between overparametrization and priors in BNN posteriors, deriving key phenomena and validating through experiments.
result Overparametrization induces structured, prior-aligned weight posterior distributions.
Overparametrized neural networks can generalize well with proper regularization.
problem Generalization guarantee for noisy data in overparametrized neural networks.
method Nonparametric analysis of ℓ2-regularized GD trajectories. result Achieving minimax optimal rate of L2 estimation error with ℓ2 regularization. Deep ReLU networks generalize well with few parameters.
problem Generalization of overparametrized deep neural networks.
method Explicit bounds on test error independent of overparametrization and VC dimension.
result Generalization error is independent of network architecture and overparametrization.
New neural approach for estimating SEMs with provable convergence.
problem Estimating structural parameters in SEMs.
method Formulated as a min-max game with neural networks, learned using stochastic gradient descent.
result Global convergence in overparametrized regime, improving state-of-the-art.
Overparametrized neural networks retain significant epistemic uncertainty even with sufficient data.
problem Epistemic uncertainty in overparametrized neural networks persists despite model identifiability.
method Analysis of non-identifiability and characterization of residual uncertainty in one-hidden-layer ReLU networks.
result Substantial parameter uncertainty remains even when the underlying function is fully identified.
Study investigates overparametrization in survival models, revealing complex loss behavior.
problem Understanding overparametrization in survival models through interpolation.
method Defined interpolation and finite-norm interpolation, rigorously analyzed four survival models.
result Overparametrization can lead to improved performance in survival models, contrary to classical learning theory.
Deep networks without non-linearities are equivalent to shallow ones.
problem Training deep orthogonal linear networks with no non-linearity.
method Riemannian gradient descent and gradient descent on factorization.
result Training deep overparametrized networks is equivalent to shallow ones.
The paper challenges the belief that more inner iterations at test time improve performance in implicit deep learning.
problem The performance improvement of implicit deep learning models with increased inner iterations at test time.
method Theoretical analysis of a simple setting, validation on implicit deep learning problems.
result Overparametrization plays a key role; increasing the number of iterations at test time does not improve performance for overparametrized networks.
Study reveals hidden null components in overparametrized neural networks.
problem Hidden null components in overparametrized neural networks.
method Structure theorem of null space for neural networks using ridgelet transforms.
result Null components can be uniquely written as linear combinations of ridgelet transforms.
Quantum machine learning faces 'laziness' and 'barren plateaus', but noise can mitigate the latter.
problem Quantum machine learning's loss function landscape issues.
method Theoretical analysis of quantum variational circuits, neural tangent kernels, and noise effects.
result Noise can mitigate barren plateaus in quantum machine learning.
SGD with machine learning noise converges to global minimum exponentially fast.
problem Optimizing machine learning models with stochastic gradient descent.
method Analysis of SGD with machine learning noise, focusing on energy landscapes and gradient noise.
result SGD converges to the global minimum exponentially fast under certain conditions.
Sine activation functions enable two-layer neural networks to learn modular addition more efficiently.
problem Learning modular addition with two-layer neural networks.
method Introduced and analyzed sine activation functions, providing theoretical and empirical evidence.
result Sine activation functions allow for constant-width network realizations of modular addition, whereas ReLU networks require linear width scaling.
Gradient descent recovers principal components of overparametrized asymmetric matrices without explicit regularization.
problem Asymmetric matrix factorization under overparametrization with minimal rank assumptions.
method Vanilla gradient descent with small random initialization and proper early stopping.
result Gradient descent produces the best low-rank approximation without explicit regularization.
Study shows how large neural networks avoid overfitting through decoupling of feature learning and complexity growth.
problem Understanding inductive bias and generalization in large neural networks.
method Dynamical mean field theory applied to large two-layer networks.
result Training dynamics of large networks exhibit a separation of timescales, decoupling feature learning and overfitting.
Two-layer neural networks need more neurons to be robust.
problem Understanding the robustness of two-layer neural networks and the role of overparametrization.
method Investigation of the tradeoffs between network size and robustness, using Lipschitz constant as a measure.
result A conjecture that robustness requires overparametrization, with precise bounds for different cases.
Dropout training improves neural networks' performance.
problem Improving neural network convergence and generalization.
method Two-layer neural networks with ReLU activations, overparametrization, and positive margin assumption.
result Dropout training achieves ε-suboptimality in test error in O(1/ε) iterations.
Noise injection before gradient steps helps in regularization for neural networks.
problem Improving generalization in overparametrized neural networks.
method Injecting small noise perturbations before computing gradient steps, especially in layer-wise fashion.
result Small noise perturbations can explicitly regularize neural networks without variance explosion.
Bayesian free energy remains bounded for deep ReLU networks in overparametrized cases.
problem Understanding the generalization performance of deep ReLU neural networks.
method Analyzes Bayesian free energy in overparametrized deep ReLU neural networks.
result Bayesian free energy is bounded even in overparametrized deep ReLU networks.
New ridge regression bounds for high-dimensional data without proportional growth.
problem Moving beyond proportional asymptotics in high-dimensional statistics.
method Revisits ridge regression on i.i.d. data, allowing high-dimensional or infinite-dimensional feature vectors.
result Establishes non-asymptotic bounds approximating bias and variance of ridge regression.