Large initial learning rate helps neural nets generalize better.
problem Understanding why large initial learning rates lead to better neural net generalization.
method Developed a proof for a two-layer network and demonstrated with experiments on CIFAR-10.
result Proved that a two-layer network trained with a large initial learning rate and annealing generalizes better than one trained with a small learning rate.
New dynamics for SGD in small learning rate regime.
problem Improving stochastic gradient descent in small learning rate regime.
method Introducing stochastic modified flows and distribution dependent stochastic modified flows.
result Captures fluctuating dynamics of SGD in small learning rate - infinite width scaling regime.
TANGO optimizes models with small learning rates, converging to natural gradient.
problem Optimizing models with small learning rates to converge to natural gradient.
method TANGO, a simple algorithm that converges to natural gradient descent.
result TANGO achieves natural gradient descent with small learning rates.
Improved image classifier performance with faster training using adaptive learning rates.
problem Lower accuracy and training time in small datasets.
method Dynamic learning rate tuning for different model architectures.
result Two-fold to ten-fold speedup in accuracy over state-of-the-art methods.
SGD with large learning rates can achieve better test accuracy than expected.
problem SGD with large learning rates often outperforms expected convergence bounds.
method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.
result Explicitly adding an implicit regularizer to the loss improves test accuracy.
Learning rate annealing helps even in convex problems, improving generalization.
problem Improving generalization in neural networks, especially convex problems.
method Learning rate annealing schedule (large initial, then small learning rate).
result Gradient descent can reach minima with better generalization using learning rate annealing.
Large learning rates cause oscillations in NN weights that improve generalization.
problem Improving generalization of neural networks trained with large learning rates.
method Theoretical analysis and feature-noise data generation model.
result Oscillating SGD with large learning rates benefits NN generalization by effectively learning weak features.
Random learning rate improves neural network training without extra cost.
problem Optimizing neural networks with stochastic gradient descent.
method Uniformly-distributed random learning rate in SGD.
result Random learning rate yields better regularization without extra computational cost.
Transfer learning improves chaotic dynamics predictions with less data.
problem Efficiently predicting chaotic dynamics with limited data.
method Transfer learning for nonlinear dynamics, optimizing transfer rate and leveraging small-scale turbulence universality.
result Significantly more accurate inference of chaotic dynamics achieved.
Large learning rates prevent memorization in denoising score matching.
problem Memorization of training data in diffusion-based generative models.
method Investigating the role of large learning rates in the small-noise regime, proving that they prevent convergence to the empirical optimal score.
result Large learning rates prevent memorization by making it impossible for the learned score to be arbitrarily close to the empirical optimal score.
New model explains deep learning performance at large learning rates.
problem Understanding deep learning performance at different learning rates.
method Developed neural networks with solvable training dynamics.
result Large learning rates lead to convergence to flatter minima.
Improves accuracy in image classification tasks by optimizing learning rates.
problem Learning from small amounts of labeled data in deep learning.
method Optimizes learning rates for neural network layers based on dataset parameters.
result Improvements in accuracy of 127% on ImageNet22k and Oxford Flowers datasets.
New algorithm achieves small-loss bounds in online learning with improved rates.
problem Achieving strong stability in online learning algorithms.
method Introduces ρ-separation to enforce strong stability, unifying previous approaches. result Oracle-efficient algorithm achieves small-loss bounds with improved rates.
Study shows deep linear networks can converge to flatter minima at large learning rates.
problem Understanding the implicit bias of deep linear networks at large learning rates.
method Characterization of deep linear networks for binary classification using logistic loss in the large learning rate regime.
result Gradient descent iterates converge to a flatter minimum in the catapult phase for certain data separation conditions.
Stochastic Gradient Descent shows directional bias with moderate learning rates, impacting optimization outcomes.
problem Understanding the bias of SGD with moderate learning rates in practical scenarios.
method Analyzing SGD and GD on an overparameterized linear regression problem.
result SGD converges along large eigenvalue directions, GD along small ones, affecting early stopping outcomes.
Paper develops error rates for physics-informed learning, comparing it to data-driven methods.
problem Understanding the trade-off between soft penalties and hard constraints in PISL.
method Develops complexity-dependent error rates using the small-ball method.
result Physics-informed estimators have comparable error rates to hard constrained methods, differing only by constants.
In this paper, we study the problem of sparse multiple kernel learning (MKL), where the goal is to efficiently learn a combination of a fixed small number of kernels from a large pool that could lead to a kernel classifier with a small prediction error. We develop an efficient algorithm based on the greedy coordinate d…
TLRF improves timely COVID-19 outbreak detection with small sample size counties.
problem Balancing accuracy and speed in estimating COVID-19 case growth rates.
method Transfer Learning Random Forest (TLRF) framework for growth rate estimation.
result TLRF outperforms existing methods in predicting case growth rates and timely outbreak detection.
Large learning rates improve generalization, but optimal ranges are narrower than previously thought.
problem Optimizing learning rates for neural network training.
method Detailed exploration of learning rate ranges in a simplified setup, validating findings in a practical setting.
result Optimal learning rate ranges are significantly narrower than previously assumed.
Short-horizon bias causes meta-optimization to favor small learning rates.
problem Short-horizon bias in meta-optimization leads to suboptimal learning rates.
method Analyzes a noisy quadratic cost function and runs meta-optimization experiments on benchmark datasets.
result Meta-optimization chooses too small a learning rate, even with a long time horizon.
A new topology improves decentralized learning efficiency and accuracy.
problem Finding efficient decentralized learning topologies with fast consensus and low maximum degree.
method Proposed the Base-(k+1) Graph topology for decentralized learning. result The Base-(k+1) Graph enables faster convergence and better communication efficiency than the exponential graph. We investigate the learning rate of multiple kernel leaning (MKL) with elastic-net regularization, which consists of an ℓ1-regularizer for inducing the sparsity and an ℓ2-regularizer for controlling the smoothness. We focus on a sparse setting where the total number of kernels is large but the number of non…
Small batch training improves deep neural network performance and stability.
problem Improving deep neural network performance and stability with limited computational resources.
method Experimental comparison of test performance for different mini-batch sizes, focusing on learning rate scaling and training duration.
result Best performance achieved for mini-batch sizes between 2 and 32, contrasting recent work advocating larger batch sizes.
In a previous paper the authors defined the growth rate of the tunnel number of knots, an invariant that measures that asymptotic behavior of the tunnel number under connected sum. In this paper we calculate the growth rate of the tunnel number of m-small knots in terms of their bridge indices.
SGD dynamics influence neural network training speed and generalization.
problem Understanding SGD dynamics in relation to neural network loss.
method Empirical investigation and analysis of SGD dynamics in relation to the sharpest directions of the loss.
result Reducing learning rate along the sharpest directions improves training speed and generalization.
Custom loss functions improve accuracy of wildfire rate of spread forecasts.
problem Improving accuracy of wildfire rate of spread forecasts.
method Examined custom loss functions in machine learning models of fuel moisture content.
result Custom loss functions improved accuracy of ROS forecasts by a small amount.
We investigate the learning rate of multiple kernel learning (MKL) with ℓ1 and elastic-net regularizations. The elastic-net regularization is a composition of an ℓ1-regularizer for inducing the sparsity and an ℓ2-regularizer for controlling the smoothness. We focus on a sparse setting where the total …
Large learning rates lead to optimal generalization if chosen carefully.
problem Understanding the optimal range of large learning rates for neural network training.
method Empirical study focusing on two questions: optimal initial LR range and differences between models trained with different LRs.
result Optimal initial learning rates slightly above the convergence threshold lead to optimal results after fine-tuning with a small LR or weight averaging.
Introduces gradient decay in Softmax for better generalization.
problem Improving generalization performance in neural networks.
method Gradient decay hyperparameter in Softmax for varying gradient rates based on probability.
result Gradient decay rate affects generalization performance and can be tuned for better optimization.
Open manifolds with nonnegative Ricci curvature have virtually abelian fundamental groups if they escape from bounded balls at a small rate.
problem Understanding the fundamental groups of open manifolds with nonnegative Ricci curvature.
method Analyzing the escape rate of minimal geodesic loops and relating it to the fundamental group's properties.
result If an open manifold has a small escape rate, its fundamental group is virtually abelian.
A new model reduces rating transition matrix estimation errors for small portfolios.
problem Estimating rating transition matrices for small portfolios leads to unreliable and unstable predictions.
method A sparse structural model with three parameters that assumes an autoregressive mean-reverting ability-to-pay process.
result The model produces well-behaved transition probabilities, reducing statistical degrees of freedom and improving reliability.
New framework assesses regularization norms in ill-posed problems, revealing L2 instability and proposing adaptive fractional RKHS solutions.
problem Comparative analysis of regularization norms in ill-posed problems.
method Small noise analysis framework for Tikhonov and RKHS regularizations.
result Optimal convergence rates achieved with adaptive fractional RKHS, but hyper-parameters decay too fast.
Improves model accuracy in medical imaging with small datasets using transfer learning.
problem Challenges in training neural networks on small medical imaging datasets.
method Comparison of current techniques, proposing one cycle training, discriminative learning rates, gradual freezing, and parameter modification.
result Transfer learning is crucial for small datasets, especially when images from the same part of the body are available.
Develops methods to measure and set function-space learning rates in neural networks.
problem Measuring and optimizing changes in neural network output functions.
method Efficient methods to measure and set function-space learning rates, requiring minimal computational overhead.
result Demonstrates FLeRM (Function-space Learning Rate Matching) for hyperparameter transfer across model scales.
Catapult phase in neural nets shows exponential loss growth before quick decrease.
problem Understanding phase transitions in neural networks during training.
method Analyzing weight norm and loss behavior for super-critical learning rates.
result Proven existence of catapult phase in quadratic models and two-layer nets.
Bayesian evidence explains why SGD finds good minima.
problem How to predict generalization in machine learning models.
method Bayesian evidence, stochastic gradient descent, small mini-batches.
result Optimum batch size maximizes test set accuracy.
For binary classification we establish learning rates up to the order of n−1 for support vector machines (SVMs) with hinge loss and Gaussian RBF kernels. These rates are in terms of two assumptions on the considered distributions: Tsybakov's noise assumption to establish a small estimation error, and a new geometr…
Maximal initial learning rate for deep ReLU networks identified.
problem Finding the optimal initial learning rate for deep neural networks.
method Simple approach to estimate maximal initial learning rate η∗, analyzing its behavior in constant-width fully-connected ReLU networks. result Maximal initial learning rate η∗ is well predicted as a power of depth × width, with specific conditions for network width and input layer training. The paper analyzes how GANs converge using dual metric flows.
problem Understanding the convergence dynamics of GANs.
method Investigates the convergence of GANs using dual metric flows, formal definitions, and proving convergence.
result GAN learning dynamics converge to a limit when learning rate is small.
Adam optimizer fails to stay close to optimal point under certain conditions.
problem Adam optimizer's tendency to deviate from the optimal point in training neural networks.
method Analyzed Adam's behavior in convex regions and proposed a new algorithm to correct this.
result Adam optimizer cannot stay close to the optimal point when effective learning rate exceeds a certain bound.
Deep learning method improves risk assessment for small loan portfolios.
problem Measuring name concentration risk in small loan portfolios.
method Deep learning approach using Monte Carlo simulations with importance sampling.
result New method outperforms existing analytical methods for small portfolios.
Unintended effects from scaling neural network outputs with adaptive learning rates.
problem Adaptive learning rate optimization's behavior is altered by output scaling, leading to misinterpretation.
method Presented a modified optimization algorithm to mitigate unintended effects.
result Adaptive learning rate's effectiveness is significantly impacted by output scaling, especially for small scaling factors.
ScheduleFree+ improves large language model training without schedules or learning rates.
problem Scaling up Schedule-Free Learning to large language models.
method Learning-rate-free and schedule-free method for training large language models.
result ScheduleFree+ outperforms SOTA schedules by 31% at 1000 tokens per parameter.
New learning rates derived for Tikhonov-regularized problems without kernel assumptions.
problem Learning rates for Tikhonov-regularized learning problems.
method Minimax adaptive rates derived using Fourier isocapacitary condition and interpolation theory.
result Derivation of minimax adaptive rates without requiring kernel assumptions.
Two distinct phases of deep learning training improve model generalization.
problem Optimization and generalization in deep learning models.
method Isolating two phases of training: large-step and small-step, and tailoring training algorithms to each.
result Training algorithms optimized for each phase significantly simplify learning rate schedules.
ALRC clips learning rates to stabilize neural network training.
problem Stability issues in training with small batch sizes and high learning rates.
method Adaptive learning rate clipping (ALRC) to limit losses.
result ALRC decreases errors in unstable training conditions.
Study shows short rate can explode to infinity in HJM model, impacting Eurodollar futures.
problem Exploding short rate in HJM model affecting Eurodollar futures.
method Small-noise deterministic limit analysis.
result Explicit explosion criteria derived for short rate under mild assumptions.
Derives effective continuous dynamics for adaptive SGD methods.
problem Analyzing noise in adaptive SGD methods.
method Stochastic modified equations framework and Malladi's scaling rules.
result Sampling-induced noise in SGD limits to independent Brownian motions.