arXiv research

Stochastic gradient descent with a large initial learning rate is widely used for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is anne…

New dynamics for SGD in small learning rate regime.

SGD with large learning rates can achieve better test accuracy than expected.

Learning rate annealing helps even in convex problems, improving generalization.

Large learning rates cause oscillations in NN weights that improve generalization.

Random learning rate improves neural network training without extra cost.

Transfer learning improves chaotic dynamics predictions with less data.

Large learning rates prevent memorization in denoising score matching.

New model explains deep learning performance at large learning rates.

New algorithm achieves small-loss bounds in online learning with improved rates.

Study shows deep linear networks can converge to flatter minima at large learning rates.

Stochastic Gradient Descent shows directional bias with moderate learning rates, impacting optimization outcomes.

Paper develops error rates for physics-informed learning, comparing it to data-driven methods.

Learning from small amounts of labeled data is a challenge in the area of deep learning. This is currently addressed by Transfer Learning where one learns the small data set as a transfer task from a larger source dataset. Transfer Learning can deliver higher accuracy if the hyperparameters and source dataset are chose…

In this paper, we study the problem of sparse multiple kernel learning (MKL), where the goal is to efficiently learn a combination of a fixed small number of kernels from a large pool that could lead to a kernel classifier with a small prediction error. We develop an efficient algorithm based on the greedy coordinate d…

Our paper introduces an efficient combination of established techniques to improve classifier performance, in terms of accuracy and training time. We achieve two-fold to ten-fold speedup in nearing state of the art accuracy, over different model architectures, by dynamically tuning the learning rate. We find it especia…

We introduce a simple algorithm, True Asymptotic Natural Gradient Optimization (TANGO), that converges to a true natural gradient descent in the limit of small learning rates, without explicit Fisher matrix estimation. For quadratic models the algorithm is also an instance of averaged stochastic gradient, where the par…

TLRF improves timely COVID-19 outbreak detection with small sample size counties.

Large learning rates improve generalization, but optimal ranges are narrower than previously thought.

A new topology improves decentralized learning efficiency and accuracy.

We investigate the learning rate of multiple kernel leaning (MKL) with elastic-net regularization, which consists of an $\ell_1$-regularizer for inducing the sparsity and an $\ell_2$-regularizer for controlling the smoothness. We focus on a sparse setting where the total number of kernels is large but the number of non…

In a previous paper the authors defined the growth rate of the tunnel number of knots, an invariant that measures that asymptotic behavior of the tunnel number under connected sum. In this paper we calculate the growth rate of the tunnel number of m-small knots in terms of their bridge indices.

Custom loss functions improve accuracy of wildfire rate of spread forecasts.

We investigate the learning rate of multiple kernel learning (MKL) with $\ell_1$ and elastic-net regularizations. The elastic-net regularization is a composition of an $\ell_1$-regularizer for inducing the sparsity and an $\ell_2$-regularizer for controlling the smoothness. We focus on a sparse setting where the total …

Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly und…

Large learning rates lead to optimal generalization if chosen carefully.

Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footp…

Introduces gradient decay in Softmax for better generalization.

Develops methods to measure and set function-space learning rates in neural networks.

Open manifolds with nonnegative Ricci curvature have virtually abelian fundamental groups if they escape from bounded balls at a small rate.

New framework assesses regularization norms in ill-posed problems, revealing L2 instability and proposing adaptive fractional RKHS solutions.

Catapult phase in neural nets shows exponential loss growth before quick decrease.

We derive the fast convergence rates of a deep neural network (DNN) classifier with the rectified linear unit (ReLU) activation function learned using the hinge loss. We consider three cases for a true model: (1) a smooth decision boundary, (2) smooth conditional class probability, and (3) the margin condition (i.e., t…

Maximal initial learning rate for deep ReLU networks identified.

For binary classification we establish learning rates up to the order of $n^{-1}$ for support vector machines (SVMs) with hinge loss and Gaussian RBF kernels. These rates are in terms of two assumptions on the considered distributions: Tsybakov's noise assumption to establish a small estimation error, and a new geometr…

Unintended effects from scaling neural network outputs with adaptive learning rates.

Deep learning method improves risk assessment for small loan portfolios.

The paper analyzes how GANs converge using dual metric flows.

ScheduleFree+ improves large language model training without schedules or learning rates.

New learning rates derived for Tikhonov-regularized problems without kernel assumptions.

Derives effective continuous dynamics for adaptive SGD methods.

New insights into network generalization show learning rate affects both norm and sharpness.

The growing use of Machine Learning has produced significant advances in many fields. For image-based tasks, however, the use of deep learning remains challenging in small datasets. In this article, we review, evaluate and compare the current state-of-the-art techniques in training neural networks to elucidate which te…

Collaborative filtering is a useful technique for exploiting the preference patterns of a group of users to predict the utility of items for the active user. In general, the performance of collaborative filtering depends on the number of rated examples given by the active user. The more the number of rated examples giv…

We consider two questions at the heart of machine learning; how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? Our work responds to Zhang et al. (2016), who showed deep neural networks can easily memorize randomly labeled training …

Lung cancer continues to be a major healthcare challenge with high morbidity and mortality rates among both men and women worldwide. The majority of lung cancer cases are of non-small cell lung cancer type. With the advent of targeted cancer therapy, it is imperative not only to properly diagnose but also sub-classify …

We show that in dimension 4 and above, the lifespan of Ricci flows depends on the relative smallness of the Ricci curvature compared to the Riemann curvature on the initial manifold. We can generalize this lifespan estimate to the local Ricci flow, using which we prove the short-time existence of Ricci flow solutions o…

New learning rate approach reveals phase transitions in SGD performance.