arXiv research

Strongly convex bodies can be approximated by smooth ones.

We consider the problem of minimizing the sum of an average function of a large number of smooth convex components and a general, possibly non-differentiable, convex function. Although many methods have been proposed to solve this problem with the assumption that the sum is strongly convex, few methods support the non-…

Study smooths Finsler structures on Lie groups, proving extremal convergence.

This work accelerates gradient descent with anytime convergence guarantees.

We propose an optimization method for minimizing the finite sums of smooth convex functions. Our method incorporates an accelerated gradient descent (AGD) and a stochastic variance reduction gradient (SVRG) in a mini-batch setting. Unlike SVRG, our method can be directly applied to non-strongly and strongly convex prob…

New methods accelerate gradient descent for convex and strongly convex functions.

New lower bounds for gradient methods in strongly convex finite-sum optimization.

Improved dynamic regret analysis for strongly convex and smooth functions.

Paper solves minimax optimization gap with near-optimal algorithms.

We give a necessary complex geometric condition for a bounded smooth convex domain in Cn, endowed with the Kobayashi distance, to be Gromov hyperbolic. More precisely, we prove that if a smooth bounded convex domain contains an analytic disk in its boundary, then the domain is not Gromov hyperbolic for the Kobayashi di…

We study dual-based algorithms for distributed convex optimization problems over networks, where the objective is to minimize a sum $\sum_{i=1}^{m}f_i(z)$ of functions over in a network. We provide complexity bounds for four different cases, namely: each function $f_i$ is strongly convex and smooth, each function is ei…

Study optimizes zero-order strongly convex function minimization with higher order smoothness.

Epoch-GDA achieves optimal convergence rate for SCSC min-max problems.

Two new algorithms optimize decentralized convex optimization with reduced communication rounds.

Let $U\subseteq\mathbb{R}^{n}$ be open and convex. We show that every (not necessarily Lipschitz or strongly) convex function $f:U\to\mathbb{R}$ can be approximated by real analytic convex functions, uniformly on all of $U$. In doing so we provide a technique which transfers results on uniform approximation on bounded …

Paper closes convergence gap for SGD without replacement.

We develop and analyze an asynchronous algorithm for distributed convex optimization when the objective writes a sum of smooth functions, local to each worker, and a non-smooth function. Unlike many existing methods, our distributed algorithm is adjustable to various levels of communication cost, delays, machines compu…

We consider the convex-concave saddle point problem $\min_{x}\max_{y} f(x)+y^\top A x-g(y)$ where $f$ is smooth and convex and $g$ is smooth and strongly convex. We prove that if the coupling matrix $A$ has full column rank, the vanilla primal-dual gradient method can achieve linear convergence even if $f$ is not stron…

In large-scale distributed learning, security issues have become increasingly important. Particularly in a decentralized environment, some computing units may behave abnormally, or even exhibit Byzantine failures -- arbitrary and potentially adversarial behavior. In this paper, we develop distributed learning algorithm…

New methods optimize functions on hyperbolic and spherical spaces, matching Euclidean rates up to logarithmic factors.

Boosting is a popular way to derive powerful learners from simpler hypothesis classes. Following previous work (Mason et al., 1999; Friedman, 2000) on general boosting frameworks, we analyze gradient-based descent algorithms for boosting with respect to any convex objective and introduce a new measure of weak learner p…

In this paper, we study the optimal convergence rate for distributed convex optimization problems in networks. We model the communication restrictions imposed by the network as a set of affine constraints and provide optimal complexity bounds for four different setups, namely: the function $F(\xb) \triangleq \sum_{i=1}…

New study shows acceleration in hyperbolic spaces is impossible for strongly geodesically convex functions.

Paper analyzes complexity of solving nonconvex-strongly-concave problems.

New research shows many batch selection methods for training work just as well as full batch training.

Improved convergence for Polyak steps with momentum in smooth convex optimization.

In this paper, we propose a simple variant of the original SVRG, called variance reduced stochastic gradient descent (VR-SGD). Unlike the choices of snapshot and starting points in SVRG and its proximal variant, Prox-SVRG, the two vectors of VR-SGD are set to the average and last iterate of the previous epoch, respecti…

Improved regret bounds for online convex optimization under stochastic and adversarial settings.

New algorithms minimize dynamic regret for strongly convex losses.

A new method solves convex optimization problems on manifolds efficiently.

New shuffling methods improve convergence without Lipschitz smoothness.

Introduces HMC method for sampling Gibbs densities.

Paper analyzes regret bounds for unconstrained online optimization.

Stochastic GD converges linearly for CV@R learning under certain conditions.

The paper explores symplectic foliations and their leaves on manifolds.

In this paper we study the differentially private Empirical Risk Minimization (ERM) problem in different settings. For smooth (strongly) convex loss function with or without (non)-smooth regularization, we give algorithms that achieve either optimal or near optimal utility bounds with less gradient complexity compared …

We study stochastic gradient descent {\em without replacement} (\sgdwor) for smooth convex functions. \sgdwor is widely observed to converge faster than true \sgd where each sample is drawn independently {\em with replacement} \cite{bottou2009curiously} and hence, is more popular in practice. But it's convergence prope…

Improved Clipped-SGD achieves near-optimal heavy-tailed statistical estimation in streaming settings.

The paper examines curvature-dimension bounds on sub-Finsler Heisenberg groups.

We consider the stochastic composition optimization problem proposed in \cite{wang2017stochastic}, which has applications ranging from estimation to statistical and machine learning. We propose the first ADMM-based algorithm named com-SVR-ADMM, and show that com-SVR-ADMM converges linearly for strongly convex and Lipsc…

AGNES accelerates gradient descent with noisy gradients.

Paper proves CLT for quantile SGD with constant learning rate.

We propose a family of optimization methods that achieve linear convergence using first-order gradient information and constant step sizes on a class of convex functions much larger than the smooth and strongly convex ones. This larger class includes functions whose second derivatives may be singular or unbounded at th…

Paper solves robust convex problems with heavy-tailed noise.

Lower bounds for geodesically convex optimization show curvature negatively impacts complexity.

We consider stochastic gradient descent algorithms for minimizing a non-smooth, strongly-convex function. Several forms of this algorithm, including suffix averaging, are known to achieve the optimal $O(1/T)$ convergence rate in expectation. We consider a simple, non-uniform averaging strategy of Lacoste-Julien et al. …

The paper tackles minimax optimality in continuum contextual bandits with Hölder continuity.

SGD converges to global minimum for structured non-convex functions.