arXiv research

This paper strengthens the central limit theorem for order statistics using relative entropy.

In this paper, we study stochastic non-convex optimization with non-convex random functions. Recent studies on non-convex optimization revolve around establishing second-order convergence, i.e., converging to a nearly second-order optimal stationary points. However, existing results on stochastic non-convex optimizatio…

We develop high-order approximations for the Heston model.

We consider the case of derivative-free algorithms for non-convex optimization, also known as zero order algorithms, that use only function evaluations rather than gradients. For a wide variety of gradient approximators based on finite differences, we establish asymptotic convergence to second order stationary points u…

Recent advances in Bayesian learning with large-scale data have witnessed emergence of stochastic gradient MCMC algorithms (SG-MCMC), such as stochastic gradient Langevin dynamics (SGLD), stochastic gradient Hamiltonian MCMC (SGHMC), and the stochastic gradient thermostat. While finite-time convergence properties of th…

A new method for faster optimization in high dimensions.

We introduce TrustVI, a fast second-order algorithm for black-box variational inference based on trust-region optimization and the reparameterization trick. At each iteration, TrustVI proposes and assesses a step based on minibatches of draws from the variational distribution. The algorithm provably converges to a stat…

A new hybrid-ordered SGD method reduces communication and complexity for non-convex optimization.

Efficient RNN algorithm guarantees convergence in online learning.

Modified Metropolis algorithm ensures convergence for multivariate binary distributions with fixed-order updates.

New method shows how order of gradient updates impacts stability and convergence in deep learning.

First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the prohibitive computational cost in calculating the second-order information. In this pa…

Study shows rate of convergence for particle approximation of PDEs in Wasserstein space.

Gradient flow of ReLU networks converges in low-correlation high-dimensional data.

OSGM uses online learning to adapt stepsize for faster convergence.

A new algorithm for solving constrained convex optimization problems efficiently.

This paper analyzes and guarantees convergence of prior-guided ZO algorithms.

Cubic-regularized Newton's method (CR) is a popular algorithm that guarantees to produce a second-order stationary solution for solving nonconvex optimization problems. However, existing understandings of the convergence rate of CR are conditioned on special types of geometrical properties of the objective function. In…

New high-order approximations for CIR process using random grids.

Negative step sizes improve second-order methods for neural networks.

TOLD++ improves convergence of diffusion models by critically damping the forward transition matrix.

Paper proposes a second-order method for faster SVI convergence.

In this paper, we design and analyze a new zeroth-order online algorithm, namely, the zeroth-order online alternating direction method of multipliers (ZOO-ADMM), which enjoys dual advantages of being gradient-free operation and employing the ADMM to accommodate complex structured regularizers. Compared to the first-ord…

We simplify deep learning convergence analysis using basic math.

Momentum is a popular technique to accelerate the convergence in practical training, and its impact on convergence guarantee has been well-studied for first-order algorithms. However, such a successful acceleration technique has not yet been proposed for second-order algorithms in nonconvex optimization.In this paper, …

Unified framework for analyzing batch updating methods with noisy gradients.

In this paper, we are interested in the strong convergence properties of the Ninomiya-Victoir scheme which is known to exhibit weak convergence with order 2. We prove strong convergence with order $1/2$. This study is aimed at analysing the use of this scheme either at each level or only at the finest level of a multil…

This paper improves convergence guarantees for SGD algorithms in non-convex smooth functions.

The paper proposes a time-dependent Markov model for a limit order book.

Improved Local SGD convergence for general convex objectives with bounded second-order heterogeneity.

Sharp convergence analysis for nonconvex regression models.

New method accelerates convergence for entropy-regularized reinforcement learning problems.

New algorithm provably converges to second-order stationary points in NMF.

This paper analyzes convergence of RMSProp and Adam in non-convex optimization with tight complexity bounds.

First-order method solves stochastic bilevel optimization with linear constraints.

Zeroth-order methods favor flat minima in machine learning.

Study proves fluid limits of fragmented limit-order markets.

New algorithm improves convergence of gradient boosting trees.

Policy gradient methods find Nash equilibrium in noisy games.

Local search heuristics for non-convex optimizations are popular in applied machine learning. However, in general it is hard to guarantee that such algorithms even converge to a local minimum, due to the existence of complicated saddle point structures in high dimensions. Many functions have degenerate saddle points su…

We study convergence properties of the full truncation Euler scheme for the Cox-Ingersoll-Ross process in the regime where the boundary point zero is inaccessible. Under some conditions on the model parameters (precisely, when the Feller ratio is greater than three), we establish the strong order 1/2 convergence in $L^…

Many problems in machine learning and game theory can be formulated as saddle-point problems, for which various first-order methods have been developed and proven efficient in practice. Under the general convex-concave assumption, most first-order methods only guarantee an ergodic convergence rate, that is, the uniform…

Improved fourth-order compact scheme for option valuation with Robin boundary condition.

COS method convergence conditions expanded for heavy-tailed distributions.

PWGF escapes saddle points in nonconvex optimization.

Sparse perturbations improve convergence in SZO methods for faster training.

We study the limiting behaviour of Darboux and Calapso transforms of polarized curves in the conformal n-dimensional sphere, when the polarization has a pole of first or second order at some point. We prove that for a pole of first order, as the singularity is approached all Darboux transforms converge to the original …

The study computes indicial roots and metric convergence orders for Ricci-flat conifolds.