arXiv research

This paper introduces a novel online inference method for high-dimensional GLMs.

Adaptive collection of data is commonplace in applications throughout science and engineering. From the point of view of statistical inference however, adaptive data collection induces memory and correlation in the samples, and poses significant challenge. We consider the high-dimensional linear regression, where the s…

ADML combines debiased learning with data-driven model selection for efficient inference.

Proposes online debiasing estimators for adaptive linear regression.

Paper proposes a debiased estimator for adaptive linear regression.

The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function or the error of the obtained solution, we investigate the problem of statistica…

This paper explains why Adam generalizes worse than SGD by analyzing its components.

Easyllp simplifies LLP, achieving low task loss at individual instance level.

Derives effective continuous dynamics for adaptive SGD methods.

Untuned SGD converges but with an exponential dependence on smoothness, adaptive methods prevent this.

Adaptive step-size method improves compressed SGD performance in machine learning.

This paper analyzes convergence of DP-SGD with adaptive quantile clipping.

New insights into SGD and SGD-M in high dimensions.

Adaptive distributed SGD reduces delay in slow workers.

Machine learning improves measuring climate adaptation impacts.

Introduces a new stochastic optimization method for deep learning.

To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers. However, the speedup brought by multiple workers is limited by the communication overhead. Two approaches, namely pipelining and gradient spar…

Paper analyzes high probability convergence of adaptive SGD with momentum.

Adaptive quantization improves SGD accuracy in data-parallel settings.

AdaGrad-Norm achieves optimal convergence rates for non-convex objectives without tuning.

New method neutralizes gender bias in word embeddings without losing semantic information.

From a simplified analysis of adaptive methods, we derive AvaGrad, a new optimizer which outperforms SGD on vision tasks when its adaptability is properly tuned. We observe that the power of our method is partially explained by a decoupling of learning rate and adaptability, greatly simplifying hyperparameter search. I…

Private adaptive methods improve on traditional SGD for convex optimization.

New adaptive SGD algorithms for federated learning over physical channels.

Adaptive SGD learns optimal batch size for strong convex functions.

IntSGD compresses SGD gradients without floats, converging as SGD.

Adapts SGD to noise and problem specifics for faster convergence.

New method trains deep networks robustly without adaptive methods.

Proposes a new adaptive learning rate for SGD.

AdaSGD combines SGD and Adam benefits, eliminating the need for transition.

Fine-tunes LLMs to correct bias in predictions.

ARFF reduces spectral bias in SGD-trained neural networks.

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drasti…

Improved SGD methods converge faster for nonconvex optimization.

Develops a parameter-free SGD algorithm with optimal convergence rate.

Improves distributed SGD convergence speed with reduced computation load.

Paper offers anytime-valid inference for causal parameters using DML.

End-to-end analysis of SGD for STL with adaptive sub-sampling.

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the gene…

Develops methods to identify and estimate causal effects with instrumental variables.

Adaptive-SGD method optimizes machine learning training with dynamic batch and step sizes.

Stochastic Gradient Descent (SGD) is one of the most widely used techniques for online optimization in machine learning. In this work, we accelerate SGD by adaptively learning how to sample the most useful training examples at each time step. First, we show that SGD can be used to learn the best possible sampling distr…

Unified analysis for decentralized SGD across various topologies and updates.

Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We prop…

This research explains why SGD generalizes better than ADAM in deep learning.

This paper analyzes SGD with biased gradients for deep learning models.

Privacy preserving machine learning algorithms are crucial for learning models over user data to protect sensitive information. Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed. At each step, these algorithms modify the gradie…

AdaScale SGD adapts learning rates for large-batch training efficiently.