arXiv research

New method improves online covariance estimation for SGD.

We propose accelerated randomized coordinate descent algorithms for stochastic optimization and online learning. Our algorithms have significantly less per-iteration complexity than the known accelerated gradient algorithms. The proposed algorithms for online learning have better regret performance than the known rando…

In this work we introduce a conditional accelerated lazy stochastic gradient descent algorithm with optimal number of calls to a stochastic first-order oracle and convergence rate $O\left(\frac{1}{\varepsilon^2}\right)$ improving over the projection-free, Online Frank-Wolfe based stochastic gradient descent of Hazan an…

An online decision-making algorithm using stochastic gradient descent for big data.

New method for online statistical inference in contextual bandits using SGD.

Algorithm improves online canonical correlation analysis.

Paper proposes an online estimator for covariance matrix of SGD iterates.

New method for online inference using SGD with random scaling.

New method improves online nonparametric estimators with minimal extra computation.

Paper studies PSGD for constrained optimization problems and its statistical properties.

The practical performance of online stochastic gradient descent algorithms is highly dependent on the chosen step size, which must be tediously hand-tuned in many applications. The same is true for more advanced variants of stochastic gradients, such as SAGA, SVRG, or AdaGrad. Here we propose to adapt the step size by …

Robust multivariate linear regression methods for online and offline use.

New method optimizes SDE models using continuous-time gradient descent.

We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allo…

Noise causes learning plateaus in neural networks.

NeCPD improves online tensor decomposition using SGD with Hessian analysis and NAG.

Cheap methods improve uncertainty in SGD solutions.

We study online convex optimization under stochastic sub-gradient observation faults, where we introduce adaptive algorithms with minimax optimal regret guarantees. We specifically study scenarios where our sub-gradient observations can be noisy or even completely missing in a stochastic manner. To this end, we propose…

Study Markov chain gradient descent in Hilbert spaces for quadratic loss.

Improves online learning algorithms for functional models with capacity assumptions.

New algorithms reduce regret in online convex optimization with heavy-tailed gradients.

A distributed system identification method for LTI systems using reverse experience replay.

Optimal algorithms for online convex optimization with random order.

Paper introduces P-FGD for online quantile regression models.

We propose a framework to perform streaming covariance selection. Our approach employs regularization constraints where a time-varying sparsity parameter is iteratively estimated via stochastic gradient descent. This allows for the regularization parameter to be efficiently learnt in an online manner. The proposed fram…

Stochastic gradient descent on l1 loss converges to true parameter in online robust regression.

A method for efficient statistical inference from online algorithms.

New algorithm optimizes nonlinear SDEs online with convergence guarantees.

We study convergence properties of Stochastic Gradient Descent (SGD) for convex objectives without assumptions on smoothness or strict convexity. We consider the question of establishing that with high probability the objective evaluated at the candidate minimizer returned by SGD is close to the minimal value of the ob…

New CLT for SGD in high-dimensional regression provides online inference.

Proves local convergence of various online and recurrent optimization algorithms.

Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.

We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov …

This paper proposes a family of online second order methods for possibly non-convex stochastic optimizations based on the theory of preconditioned stochastic gradient descent (PSGD), which can be regarded as an enhance stochastic Newton method with the ability to handle gradient noise and non-convexity simultaneously. …

We prove that the evolution of weight vectors in online gradient descent can encode arbitrary polynomial-space computations, even in very simple learning settings. Our results imply that, under weak complexity-theoretic assumptions, it is impossible to reason efficiently about the fine-grained behavior of online gradie…

Study shows Stochastic Mirror Descent optimizes convex problems with infinite noise variance.

Paper develops methods for statistical inference with SGD in nonconvex optimization.

Stochastic gradient descent in continuous time (SGDCT) provides a computationally efficient method for the statistical learning of continuous-time models, which are widely used in science, engineering, and finance. The SGDCT algorithm follows a (noisy) descent direction along a continuous stream of data. SGDCT performs…

Gradient equilibrium improves online learning performance without requiring sublinear regret.

Online SGD achieves consistent estimation in high-dimensional non-convex inference tasks.

The paper analyzes SGD in high-dimensional networks, revealing new scaling limits.

This work shows neural networks can solve non-convex constraints problems.

In recent literature, a general two step procedure has been formulated for solving the problem of phase retrieval. First, a spectral technique is used to obtain a constant-error initial estimate, following which, the estimate is refined to arbitrary precision by first-order optimization of a non-convex loss function. N…

New insights into SGD and SGD-M in high dimensions.

Paper shows pre-training and transfer learning reduce sample complexity for neural networks.

New bounds for M-SGD show its error distribution is nearly Gaussian.

Stochastic gradient descent (SGD) is commonly used for optimization in large-scale machine learning problems. Langford et al. (2009) introduce a sparse online learning method to induce sparsity via truncated gradient. With high-dimensional sparse data, however, the method suffers from slow convergence and high variance…

The paper provides statistical guarantees for SGD and ASGD in high-dimensional settings.