arXiv research

New methods reduce constraint violations to certainty in stochastic optimization.

A new method speeds up quantum state estimation.

First-order method solves stochastic bilevel optimization with linear constraints.

New methods solve optimization problems with heavy-tailed noise, improving upon existing complexity bounds.

Unified approach for first-order methods with Markovian noise in stochastic optimization and variational inequalities.

Optimized method tackles convex optimization with heavy-tailed noise.

Novel BSG method for efficient stochastic optimization.

Two classes of methods have been proposed for escaping from saddle points with one using the second-order information carried by the Hessian and the other adding the noise into the first-order information. The existing analysis for algorithms using noise in the first-order information is quite involved and hides the es…

In this paper, we propose a new technique named \textit{Stochastic Path-Integrated Differential EstimatoR} (SPIDER), which can be used to track many deterministic quantities of interest with significantly reduced computational cost. We apply SPIDER to two tasks, namely the stochastic first-order and zeroth-order method…

Method solves complex optimization problems with high probability bounds.

First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity. Second-order methods, while able to provide faster convergence, have been much less explored due to the high cost of computing the second-order information. In this paper we …

First order discretizations of Langevin diffusion can achieve better generalization error with additional smoothness assumptions.

SVRN accelerates Newton methods by reducing variance and improving performance.

A framework for decentralized optimization using first-order methods.

TRSVR combines SVRG with trust-region for faster optimization.

We study distributed optimization algorithms for minimizing the average of convex functions. The applications include empirical risk minimization problems in statistical machine learning where the datasets are large and have to be stored on different machines. We design a distributed stochastic variance reduced gradien…

Logistic regression is one of the most popular methods in binary classification, wherein estimation of model parameters is carried out by solving the maximum likelihood (ML) optimization problem, and the ML estimator is defined to be the optimal solution of this problem. It is well known that the ML estimator exists wh…

Paper improves stochastic bilevel optimization methods for highly-smooth problems.

Stochastic convex optimization problems with expectation constraints (SOECs) are encountered in statistics and machine learning, business, and engineering. In data-rich environments, the SOEC objective and constraints contain expectations defined with respect to large datasets. Therefore, efficient algorithms for solvi…

New methods solve complex optimization problems without strong convexity assumptions.

New lower bounds for bilevel optimization with first-order oracles.

This is the first in a series of papers in which we study an efficient approximation scheme for solving the Hamilton-Jacobi-Bellman equation for multi-dimensional problems in stochastic control theory. The method is a combination of a WKB style asymptotic expansion of the value function, which reduces the second order …

Develops first-order methods for average-reward MDPs with strong guarantees.

Study max- and min-stability under first-order stochastic dominance, finding new functional characterizations.

The paper studies the First Order BSPDEs (Backward Stochastic Partial Differential Equations) suggested earlier for a case of multidimensional state domain with a boundary. These equations represent analogs of Hamilton-Jacobi-Bellman equations and allow to construct the value function for stochastic optimal control pro…

Unified framework for analyzing batch updating methods with noisy gradients.

Improved first-order algorithm for entropy regularized OT with faster convergence.

SGD's performance improves with critical batch size, minimizing SFO complexity.

New adaptive step-size method for convex optimization without tuning.

We propose novel first-order stochastic approximation algorithms for canonical correlation analysis (CCA). Algorithms presented are instances of inexact matrix stochastic gradient (MSG) and inexact matrix exponentiated gradient (MEG), and achieve $ε$-suboptimality in the population objective in $\operatorname{poly}(\fr…

Unified analysis of first-order methods for smooth games using IQCs.

New algorithms optimize without knowing problem parameters.

Geodesic convexity generalizes the notion of (vector space) convexity to nonlinear metric spaces. But unlike convex optimization, geodesically convex (g-convex) optimization is much less developed. In this paper we contribute to the understanding of g-convex optimization by developing iteration complexity analysis for …

The paper calculates option prices using Mellin transform for stochastic volatility models.

A new method helps escape saddle points in non-convex optimization.

Two new methods solve nonsmooth optimization on Riemannian Stiefel manifold.

We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-…

Paper develops a TR-SSQP method for noisy optimization with heavy-tailed noise.

We analyze stochastic gradient algorithms for optimizing nonconvex problems. In particular, our goal is to find local minima (second-order stationary points) instead of just finding first-order stationary points which may be some bad unstable saddle points. We show that a simple perturbed version of stochastic recursiv…

A new method reduces the complexity of decentralized optimization.

Paper proposes an algorithm to solve complex minimax problems efficiently.

Consider the stochastic composition optimization problem where the objective is a composition of two expected-value functions. We propose a new stochastic first-order method, namely the accelerated stochastic compositional proximal gradient (ASC-PG) method, which updates based on queries to the sampling oracle using tw…

We propose a fast proximal Newton-type algorithm for minimizing regularized finite sums that returns an $ε$-suboptimal point in $\tilde{\mathcal{O}}(d(n + \sqrt{κd})\log(\frac{1}ε))$ FLOPS, where $n$ is number of samples, $d$ is feature dimension, and $κ$ is the condition number. As long as $n > d$, the proposed method…

Proposes a new method for optimizing large-scale models using Nyström approximation of the Hessian.

In this paper we present a new method to compute the first-order approximation of the price of derivatives on futures in the context of multiscale stochastic volatility of Fouque \textit{et al.} (2011, CUP). It provides an alternative method to the singular perturbation technique presented in Hikspoors and Jaimungal (2…

This paper studies distributed estimation and inference for a general statistical problem with a convex loss that could be non-differentiable. For the purpose of efficient computation, we restrict ourselves to stochastic first-order optimization, which enjoys low per-iteration complexity. To motivate the proposed metho…

New method reduces communication costs in distributed nonconvex optimization.

We propose a reduction for non-convex optimization that can (1) turn an stationary-point finding algorithm into an local-minimum finding one, and (2) replace the Hessian-vector product computations with only gradient computations. It works both in the stochastic and the deterministic settings, without hurting the algor…