arXiv research

Blind Descent avoids gradient issues, using a different learning approach.

We propose accelerated randomized coordinate descent algorithms for stochastic optimization and online learning. Our algorithms have significantly less per-iteration complexity than the known accelerated gradient algorithms. The proposed algorithms for online learning have better regret performance than the known rando…

New algorithm reduces optimization complexity in adaptive mirror descent.

Gradient descent variants improve phase retrieval accuracy.

In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to train. Nowadays, most of the deep learning model training still relies on the back…

Paper uses Mirror Descent for efficient risk budgeting portfolios.

Study shows how steepest descent algorithms' geometric margin increases during training.

This monograph presents a class of algorithms called coordinate descent algorithms for mathematicians, statisticians, and engineers outside the field of optimization. This particular class of algorithms has recently gained popularity due to their effectiveness in solving large-scale optimization problems in machine lea…

Develops parameter-free online mirror descent for optimal dynamic regret.

Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Eucli…

Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…

Develops DP-SCD for stochastic coordinate descent, making it differentially private.

Gravilon improves gradient descent for neural networks.

Information geometry applies concepts in differential geometry to probability and statistics and is especially useful for parameter estimation in exponential families where parameters are known to lie on a Riemannian manifold. Connections between the geometric properties of the induced manifold and statistical properti…

Langevin algorithms enhance training of deep neural networks for stochastic control problems.

Study on privacy leakage in noisy gradient descent algorithms.

We propose an algorithm, semismooth Newton coordinate descent (SNCD), for the elastic-net penalized Huber loss regression and quantile regression in high dimensional settings. Unlike existing coordinate descent type algorithms, the SNCD updates each regression coefficient and its corresponding subgradient simultaneousl…

Lazy Gradient Descent outperforms existing polytope algorithms in pseudo-regret.

A new algorithm solves sparse optimization problems on measures efficiently.

Study on gradient descent in Hilbert spaces with Markov chains, focusing on mixing coefficients.

Continuous-time distributed mirror descent with integral feedback converges to global optimum.

Paper characterizes gradient descent in high-dimensional learning problems.

New loss function helps learn unstable dynamical systems.

Paper analyzes SVGD algorithm for non-asymptotic convergence.

New stochastic gradient descent with random search directions improves efficiency and convergence.

New bounds for model generalization under deterministic gradient descent.

We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton's …

Paper analyzes stability and generalization of SCO algorithms.

New learning algorithm for real analytic functions without gradient descent.

New algorithm reduces complexity for SPD manifold optimization.

We propose a new randomized coordinate descent method for a convex optimization template with broad applications. Our analysis relies on a novel combination of four ideas applied to the primal-dual gap function: smoothing, acceleration, homotopy, and coordinate descent with non-uniform sampling. As a result, our method…

A new method improves stochastic gradient descent for faster and more efficient estimation.

The paper develops efficient algorithms for variational inference with mixtures of isotropic Gaussians.

ECD algorithm speeds up non-convex optimization, offering quantum and stochastic enhancements.

The paper generalizes optimization algorithms using category theory.

New algorithm optimizes Bayesian network learning from Gaussian data.

Stochastic gradient descent on manifolds improves low-rank approximation.

Adaptive step-size improves optimization in complex geometries.

We introduce Kalman Gradient Descent, a stochastic optimization algorithm that uses Kalman filtering to adaptively reduce gradient variance in stochastic gradient descent by filtering the gradient estimates. We present both a theoretical analysis of convergence in a non-convex setting and experimental results which dem…

We present a generic framework for parallel coordinate descent (CD) algorithms that includes, as special cases, the original sequential algorithms Cyclic CD and Stochastic CD, as well as the recent parallel Shotgun algorithm. We introduce two novel parallel algorithms that are also special cases---Thread-Greedy CD and …

Paper proposes an algorithm to solve complex minimax problems efficiently.

Large-scale L1-regularized loss minimization problems arise in high-dimensional applications such as compressed sensing and high-dimensional supervised learning, including classification and regression problems. High-performance algorithms and implementations are critical to efficiently solving these problems. Building…

New inequalities help optimize first-order algorithms for statistical risk analysis.

A new method compresses large datasets for gradient descent algorithms efficiently.

Mirror descent method improved RL algorithms.

EM algorithm converges in KL divergence for exponential families via mirror descent.

New PnP algorithm converges with relaxed proximal gradient descent.

Training deep neural networks (DNNs) efficiently is a challenge due to the associated highly nonconvex optimization. The backpropagation (backprop) algorithm has long been the most widely used algorithm for gradient computation of parameters of DNNs and is used along with gradient descent-type algorithms for this optim…