arXiv research

Gradient descent finds global minima in deep neural networks of practical sizes.

The paper analyzes the expected size of conformal prediction sets.

Adaptive SGD learns optimal batch size for strong convex functions.

New analysis reveals batch size effects on stochastic conditional gradient methods.

The study provides theoretical foundations for using smaller instances to predict algorithm performance on larger ones.

One of the major issues in stochastic gradient descent (SGD) methods is how to choose an appropriate step size while running the algorithm. Since the traditional line search technique does not apply for stochastic optimization algorithms, the common practice in SGD is either to use a diminishing step size, or to tune a…

Adaptive batch size schedules improve language model training efficiency and generalization.

This paper compares Transformers and RNNs in various tasks, showing size differences.

Significant differences in the evolution of firm size distribution for various industries in the United States have been revealed and documented. For theoretical considerations, this finding puts major constraints on the modelling of firm growth. For practical purposes, the observed differences create a solid basis for…

Two novel search strategies reduce complexity for target localization with size-dependent noise.

The practical performance of online stochastic gradient descent algorithms is highly dependent on the chosen step size, which must be tediously hand-tuned in many applications. The same is true for more advanced variants of stochastic gradients, such as SAGA, SVRG, or AdaGrad. Here we propose to adapt the step size by …

When using active learning, smaller batch sizes are typically more efficient from a learning efficiency perspective. However, in practice due to speed and human annotator considerations, the use of larger batch sizes is necessary. While past work has shown that larger batch sizes decrease learning efficiency from a lea…

SP-NGD improves deep learning models' generalization with large mini-batch sizes.

New step-size methods improve SHB convergence for stochastic optimization.

Study assesses environmental management accounting practices in Bangladesh.

Study evaluates three position sizing methods for put-writing on S&P 500 Index options.

Implicit Q-learning and SARSA adjust step-sizes automatically, improving stability and performance.

Step decay schedules improve convergence in non-convex optimization.

Extends hyperparameter transfer across model sizes and modules, improving training speed.

Improved online prediction with guaranteed coverage.

This paper optimizes step sizes for faster convergence in sparse coding.

AutoStep MCMC adapts step size locally for better sampling efficiency.

New neural network models learn symmetric functions of varying input sizes.

The paper improves A/B testing for non-Gaussian data, ensuring reliable results with large sample sizes.

Deep forests enhance expressiveness exponentially with depth, not width or tree size.

New scheme adapts batch size for faster variance-reduced algorithms.

Structured pruning method reduces RNN sizes and speeds up inference.

Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …

New method reduces training time for deep hedging networks.

New analysis shows optimal embedding learning rate depends on vocabulary size, not just model width.

Muon optimizes training efficiency by improving data retention at large batch sizes.

Study non-monotonic loss functions in CRC, achieving valid risk control with large calibration samples.

This work studies scaling laws for low-precision training in high-dimensional linear regression.

Although stochastic gradient descent (SGD) method and its variants (e.g., stochastic momentum methods, AdaGrad) are the choice of algorithms for solving non-convex problems (especially deep learning), there still remain big gaps between the theory and the practice with many questions unresolved. For example, there is s…

Large batch sizes reduce gradient variance in DP-SGD, improving privacy.

New method improves uncertainty quantification for large batch sizes and misspecified models.

Unified scaling laws reveal how model size and training time impact neural network performance.

Simple private estimators for mean and covariance outperform existing methods.

Adaptive step-size improves optimization in complex geometries.

Faced with massive data, is it possible to trade off (statistical) risk, and (computational) space and time? This challenge lies at the heart of large-scale machine learning. Using k-means clustering as a prototypical unsupervised learning problem, we show how we can strategically summarize the data (control space) in …

The ability to learn from a small number of examples has been a difficult problem in machine learning since its inception. While methods have succeeded with large amounts of training data, research has been underway in how to accomplish similar performance with fewer examples, known as one-shot or more generally few-sh…

Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.

The paper analyzes and validates two step size schedules for SGD: exponential and cosine, proving their adaptivity and performance.

We consider large-scale studies in which it is of interest to test a very large number of hypotheses, and then to estimate the effect sizes corresponding to the rejected hypotheses. For instance, this setting arises in the analysis of gene expression or DNA sequencing data. However, naive estimates of the effect sizes …

Recently it has been shown that the step sizes of a family of variance reduced gradient methods called the JacSketch methods depend on the expected smoothness constant. In particular, if this expected smoothness constant could be calculated a priori, then one could safely set much larger step sizes which would result i…

The paper improves theoretical bounds on deep neural networks' convergence.

We highlight a pitfall when applying stochastic variational inference to general Bayesian networks. For global random variables approximated by an exponential family distribution, natural gradient steps, commonly starting from a unit length step size, are averaged to convergence. This useful insight into the scaling of…

Increasing the mini-batch size for stochastic gradient descent offers significant opportunities to reduce wall-clock training time, but there are a variety of theoretical and systems challenges that impede the widespread success of this technique. We investigate these issues, with an emphasis on time to convergence and…