arXiv research

Paper closes convergence gap for SGD without replacement.

Improved SHB method for faster convergence on strongly-convex quadratics.

FedExProx's performance is no better than GD for quadratic optimization.

We consider the problem of minimizing the sum of an average function of a large number of smooth convex components and a general, possibly non-differentiable, convex function. Although many methods have been proposed to solve this problem with the assumption that the sum is strongly convex, few methods support the non-…

The DANE algorithm is an approximate Newton method popularly used for communication-efficient distributed machine learning. Reasons for the interest in DANE include scalability and versatility. Convergence of DANE, however, can be tricky; its appealing convergence rate is only rigorous for quadratic objective, and for …

In large-scale distributed learning, security issues have become increasingly important. Particularly in a decentralized environment, some computing units may behave abnormally, or even exhibit Byzantine failures -- arbitrary and potentially adversarial behavior. In this paper, we develop distributed learning algorithm…

Analysis of momentum methods on quadratic models, showing SGD's superiority.

The cyclic block coordinate descent-type (CBCD-type) methods, which performs iterative updates for a few coordinates (a block) simultaneously throughout the procedure, have shown remarkable computational performance for solving strongly convex minimization problems. Typical applications include many popular statistical…

Random permutations can offer faster convergence than with-replacement sampling for some functions.

New lower bounds for bilevel optimization with first-order oracles.

Study large deviations rates for SGD with strongly convex functions.

New algorithm extends LMC to more complex potentials.

In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder p…

This paper improves the convergence rates of bilevel optimization algorithms.

We consider the convex-concave saddle point problem $\min_{x}\max_{y} f(x)+y^\top A x-g(y)$ where $f$ is smooth and convex and $g$ is smooth and strongly convex. We prove that if the coupling matrix $A$ has full column rank, the vanilla primal-dual gradient method can achieve linear convergence even if $f$ is not stron…

Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precisi…

The framework of Integral Quadratic Constraints (IQC) reduces the computation of upper bounds on the convergence rate of several optimization algorithms to a semi-definite program (SDP). In the case of over-relaxed Alternating Direction Method of Multipliers (ADMM), an explicit and closed form solution to this SDP was …

New analysis improves SGD for robust and quantile regression with sub-quadratic convergence.

In this work we propose a differential geometric motivation for Nesterov's accelerated gradient method (AGM) for strongly-convex problems. By considering the optimization procedure as occurring on a Riemannian manifold with a natural structure, The AGM method can be seen as the proximal point method applied in this cur…

We study the trade-offs between convergence rate and robustness to gradient errors in designing a first-order algorithm. We focus on gradient descent (GD) and accelerated gradient (AG) methods for minimizing strongly convex functions when the gradient has random errors in the form of additive white noise. With gradient…

We propose a randomized second-order method for optimization known as the Newton Sketch: it is based on performing an approximate Newton step using a randomly projected or sub-sampled Hessian. For self-concordant functions, we prove that the algorithm has super-linear convergence with exponentially high probability, wi…

Simple stochastic Newton and cubic Newton methods with fast convergence.

Unified analysis of first-order methods for smooth games using IQCs.

OMGD algorithm optimizes online convex optimization with switching costs and delayed gradients.

In this paper, we propose a simple variant of the original stochastic variance reduction gradient (SVRG), where hereafter we refer to as the variance reduced stochastic gradient descent (VR-SGD). Different from the choices of the snapshot point and starting point in SVRG and its proximal variant, Prox-SVRG, the two vec…

Strongly convex bodies can be approximated by smooth ones.

The conjugate gradient (CG) method is an efficient iterative method for solving large-scale strongly convex quadratic programming (QP). In this paper we propose some generalized CG (GCG) methods for solving the $\ell_1$-regularized (possibly not strongly) convex QP that terminate at an optimal solution in a finite numb…

The paper simplifies strongly convex problems to simplicial structures.

Optimal control methods achieve significantly smaller regret than previously thought.

New method estimates minimizer and minimum value of a regression function.

In this paper, we prove that a strongly convex complex Finsler metric $F$ on a domain $D\subset\mathbb{C}^n$ is projectively flat (resp. dually flat) if and only if $F$ comes from a strongly convex complex Minkowski metric.

Fewer data weight updates lead to faster convergence in machine learning models.

SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.

Improved algorithm reduces stochastic gradient complexity for large-scale learning problems.

Alt-GDA outperforms Sim-GDA in minimax games with near-optimal local convergence.

CWGD measures gradient diversity weighted by curvature, improving SGD convergence.

Characterizes Anosov representations and strongly convex cocompact groups with eigenvalue gaps.

In this paper, we study locally strongly convex centroaffine hypersurfaces with parallel cubic form with respect to the Levi-Civita connection of the centroaffine metric. As the main result, we obtain a complete classification of such centroaffine hypersurfaces. The result of this paper is a centroaffine version of the…

In this paper, we establish a general inequality for locally strongly convex centroaffine hypersurfaces in $\mathbb{R}^{n+1}$ involving the norm of the covariant derivatives of both the difference tensor $K$ and the Tchebychev vector field $T$. Our result is optimal in that, applying our recent classification for local…

The problem of stochastic convex optimization with bandit feedback (in the learning community) or without knowledge of gradients (in the optimization community) has received much attention in recent years, in the form of algorithms and performance upper bounds. However, much less is known about the inherent complexity …

A multiobjective optimization problem is simplicial if the Pareto set and front are homeomorphic to a simplex and, under the homeomorphisms, each face of the simplex corresponds to the Pareto set and front of a subproblem. In this paper, we show that strongly convex problems are simplicial under a mild assumption on th…

The paper optimizes estimating transport maps between distributions.

Many classical algorithms are found until several years later to outlive the confines in which they were conceived, and continue to be relevant in unforeseen settings. In this paper, we show that SVRG is one such method: being originally designed for strongly convex objectives, it is also very robust in non-strongly co…

Paper improves algorithms for convex-concave minimax optimization problems.

Improved SGD for non-strongly-convex regression with faster convergence.

The paper proves properties of complex Finsler metrics on specific domains.

Almost all local minima in neural networks are strongly convex.

New lower bounds for gradient methods in strongly convex finite-sum optimization.