arXiv research

Derives Mirror Descent from gradient flow on a Riemannian manifold.

Riemannian stochastic gradient descent approximates a diffusion process called Riemannian stochastic modified flow.

We provide the first experimental results on non-synthetic datasets for the quasi-diagonal Riemannian gradient descents for neural networks introduced in [Ollivier, 2015]. These include the MNIST, SVHN, and FACE datasets as well as a previously unpublished electroencephalogram dataset. The quasi-diagonal Riemannian alg…

Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Eucli…

Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.

Proposes ridge regression on Riemannian manifolds for time-series prediction.

Accelerates Riemannian gradient methods with extrapolation.

We develop Riemannian Stein Variational Gradient Descent (RSVGD), a Bayesian inference method that generalizes Stein Variational Gradient Descent (SVGD) to Riemann manifold. The benefits are two-folds: (i) for inference tasks in Euclidean spaces, RSVGD has the advantage over SVGD of utilizing information geometry, and …

Gradient descent on MMD GAN parameter space converges globally to target distribution.

Gradient descent with geometrically adapted metrics drives $\mathcal{L}^2$ cost to global minimum at uniform rate.

New algorithm reduces complexity for SPD manifold optimization.

Optimizes shapes on non-standard manifolds.

Riemannian stochastic gradient descent converges faster with increasing batch size.

This paper extends Mirror Descent to Riemannian manifolds for optimization.

Accelerated method finds critical points faster on manifolds.

Information geometry applies concepts in differential geometry to probability and statistics and is especially useful for parameter estimation in exponential families where parameters are known to lie on a Riemannian manifold. Connections between the geometric properties of the induced manifold and statistical properti…

The paper introduces a differentially private method for optimization on Riemannian manifolds.

New method avoids spurious critical points for low-rank matrix recovery.

Algorithm improves online canonical correlation analysis.

Natural gradient learning improves synaptic plasticity in spiking neurons.

The paper extends gradient flow and relaxation studies to non-flat Riemannian manifolds.

New method accelerates gradient descent on curved spaces.

Characterizes problems solvable via linear convergence algorithms.

Deep networks without non-linearities are equivalent to shallow ones.

Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.

New method classifies manifold-valued data using Riemannian geometry.

We consider the minimization of a function defined on a Riemannian manifold $\mathcal{M}$ accessible only through unbiased estimates of its gradients. We develop a geometric framework to transform a sequence of slowly converging iterates generated from stochastic gradient descent (SGD) on $\mathcal{M}$ to an averaged i…

The paper studies geometric properties of group equivariant operators and their Riemannian structure.

Several first order stochastic optimization methods commonly used in the Euclidean domain such as stochastic gradient descent (SGD), accelerated gradient descent or variance reduced methods have already been adapted to certain Riemannian settings. However, some of the most popular of these optimization tools - namely A…

Stochastic Gradient Descent improved for various Hilbert scales and misspecified models.

Gradient descent, or negative gradient flow, is a standard technique in optimization to find minima of functions. Many implementations of gradient descent rely on discretized versions, i.e., moving in the gradient direction for a set step size, recomputing the gradient, and continuing. In this paper, we present an appr…

This paper shows equivalence between SVGD and BBVI using kernel gradient flows.

We present here a new model and algorithm which performs an efficient Natural gradient descent for Multilayer Perceptrons. Natural gradient descent was originally proposed from a point of view of information geometry, and it performs the steepest descent updates on manifolds in a Riemannian space. In particular, we ext…

Stochastic mirror descent improves performance on ensemble models.

Guaranteed convergence for tensor factorization using Riemannian gradient descent.

In this paper, the Riemannian gradient algorithm and the natural gradient algorithm are applied to solve descent direction problems on the manifold of positive definite Hermitian matrices, where the geodesic distance is considered as the cost function. The first proposed problem is control for positive definite Hermiti…

Study uses outer metrics for PDE-constrained shape optimization over diffeomorphism group.

Paper proposes a new classifier for hyperbolic spaces using horospherical boundaries.

We propose a novel Riemannian manifold preconditioning approach for the tensor completion problem with rank constraint. A novel Riemannian metric or inner product is proposed that exploits the least-squares structure of the cost function and takes into account the structured symmetry that exists in Tucker decomposition…

The paper develops methods for unconstrained optimization on Riemannian manifolds.

In recent years, stochastic variance reduction algorithms have attracted considerable attention for minimizing the average of a large but finite number of loss functions. This paper proposes a novel Riemannian extension of the Euclidean stochastic variance reduced gradient (R-SVRG) algorithm to a manifold search space.…

Inexact Riemannian optimization converges to stationary points efficiently.

No accelerated gradient method for hyperbolic convex functions.

A novel approach to computing barycenters on graph-supported probability measures.

Decentralized optimization on dynamic manifolds with improved regret bound.

Blind Descent avoids gradient issues, using a different learning approach.

We study the stochastic Riemannian gradient algorithm for matrix eigen-decomposition. The state-of-the-art stochastic Riemannian algorithm requires the learning rate to decay to zero and thus suffers from slow convergence and sub-optimal solutions. In this paper, we address this issue by deploying the variance reductio…

Stochastic variance reduction algorithms have recently become popular for minimizing the average of a large, but finite, number of loss functions. In this paper, we propose a novel Riemannian extension of the Euclidean stochastic variance reduced gradient algorithm (R-SVRG) to a compact manifold search space. To this e…