The paper explains spikes in training loss as catapults, improving feature learning and generalization.
problem Understanding and improving the training process of neural networks.
method Analysis of training loss spikes in SGD, empirical evidence of catapults, and demonstration of AGOP alignment.
result Catapults in training loss promote feature learning and better generalization.
Quadratic models explain neural network behavior during training.
problem Understanding neural network dynamics during training with large learning rates.
method Developed and tested Neural Quadratic Models.
result Neural Quadratic Models exhibit the 'catapult phase' similar to neural networks.
Gradient descent with large momentum finds flatter minima.
problem Understanding the effects of momentum in gradient descent.
method Empirical and theoretical analysis of gradient descent with large momentum.
result Large momentum leads to flatter minima than gradient descent.
Catapult phase in neural nets shows exponential loss growth before quick decrease.
problem Understanding phase transitions in neural networks during training.
method Analyzing weight norm and loss behavior for super-critical learning rates.
result Proven existence of catapult phase in quadratic models and two-layer nets.
Study shows deep linear networks can converge to flatter minima at large learning rates.
problem Understanding the implicit bias of deep linear networks at large learning rates.
method Characterization of deep linear networks for binary classification using logistic loss in the large learning rate regime.
result Gradient descent iterates converge to a flatter minimum in the catapult phase for certain data separation conditions.
Large learning rates lead to various implicit biases in nonconvex optimization.
problem Understanding the conditions under which large learning rates yield edge of stability, balancing, and catapult phenomena.
method Developed a global convergence theory for nonconvex functions without globally Lipschitz continuous gradient, focusing on functions with good regularity.
result These implicit biases are more likely to occur in functions with good regularity, and large learning rates favor flatter regions.
Gradient descent dynamics in quadratic regression models are analyzed, revealing five phases: monotonic, catapult, periodic, chaotic, and divergent.
problem Analyzing the dynamics of gradient descent in quadratic regression models.
method Fine-grained bifurcation analysis of gradient descent dynamics using a cubic map parameterized by the step-size.
result Gradient descent dynamics in quadratic regression models exhibit five distinct phases: monotonic, catapult, periodic, chaotic, and divergent.
Breiman discusses two statistical cultures, advocating for more research on 'before' and 'after' the black box.
problem Statistical modeling lacks exploration of processes before and after the 'black box'.
method Analyzes Breiman's visual metaphor of two statistical cultures.
result Promotes the importance of studying the 'before' and 'after' of data transformations.
The paper shows how warming up the learning rate improves deep learning performance.
problem Improving deep learning performance through better handling of larger learning rates.
method Systematic experiments with SGD and Adam showing the benefits of warmup and different regimes of operation.
result Properly choosing ηextinit can eliminate the need for warmup and improve performance. New model explains deep learning performance at large learning rates.
problem Understanding deep learning performance at different learning rates.
method Developed neural networks with solvable training dynamics.
result Large learning rates lead to convergence to flatter minima.
The article first describes characteristics of major infrastructure projects. Second, it documents a much neglected topic in economics: that ex ante estimates of costs and benefits are often very different from actual ex post costs and benefits. For large infrastructure projects the consequence is cost overruns, benefi…
New insights into how large learning rates affect transformer training dynamics.
problem Understanding how large learning rates impact the training of transformer models.
method Analyzing a simplified linear transformer model with a two-factor product map.
result Large learning rates can lead to various training outcomes including cycles, chaos, or divergence.