arXiv research

Localized sum-of-norms clustering separates balls in data.

This paper deals with a natural stochastic optimization procedure derived from the so-called Heavy-ball method differential equation, which was introduced by Polyak in the 1960s with his seminal contribution [Pol64]. The Heavy-ball method is a second-order dynamics that was investigated to minimize convex functions f .…

New method shows stochastic momentum can converge quickly on optimization problems.

In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder p…

Nonparametric adaptive robust control tackles model uncertainty in stochastic processes.

A new Bayesian filtering method speeds up stochastic Newton optimization.

The paper analyzes convergence rates for SGD and SHB methods.

BalLOT uses optimal transport for balanced k-means clustering.

Framework for robust control under model uncertainty, improving financial derivatives hedging.

New tool for parallel and private stochastic convex optimization reduces query complexity.

The study extends stochastic completeness to landmark spaces with any number of landmarks.

Heavy Ball method speeds up finding global optima in non-convex problems.

Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of t…

In this paper we study several classes of stochastic optimization algorithms enriched with heavy ball momentum. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic dual subspace ascent. This is the first time momentum variants of several of these metho…

Joint sparsity offers powerful structural cues for feature selection, especially for variables that are expected to demonstrate a "grouped" behavior. Such behavior is commonly modeled via group-lasso, multitask lasso, and related methods where feature selection is effected via mixed-norms. Several mixed-norm based spar…

Paper proves SHB convergence with biased gradients and approximate step sizes.

New algorithm reduces regret in stochastic bandit convex optimization.

In this paper, we revisit the convergence of the Heavy-ball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/k) rate result of the Heavy-ball algorithm with constant step size for coercive objective functions. For objective functions satisfying a re…

GoTube verifies neural networks over time, scaling to large horizons.

We study computational and statistical consequences of problem geometry in stochastic and online optimization. By focusing on constraint set and gradient geometry, we characterize the problem families for which stochastic- and adaptive-gradient methods are (minimax) optimal and, conversely, when nonlinear updates -- su…

Improved SHB method for faster convergence on strongly-convex quadratics.

Paper analyzes SHB method for neural networks, proving stability, connectivity, and global convergence.

Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's accelerated gradient (AG) as well as accelerated projected gradient (APG) method have been commonly used in machine learning practice, but their performance is quite sensitive to noise in the gradients. We study these methods under a first-order stoch…

New metric derived for robust optimization in stochastic control problems.

We propose a method for zeroth order stochastic convex optimization that attains the suboptimality rate of $\tilde{\mathcal{O}}(n^{7}T^{-1/2})$ after $T$ queries for a convex bounded function $f:{\mathbb R}^n\to{\mathbb R}$. The method is based on a random walk (the \emph{Ball Walk}) on the epigraph of the function. Th…

Stochastic momentum methods have been widely adopted in training deep neural networks. However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored. This paper aims to bridge the gap between practice and theory by analyzing the stochast…

Analysis of momentum methods on quadratic models, showing SGD's superiority.

Round balls minimize liquid drop model volumes ≤ 1.

There is widespread sentiment that it is not possible to effectively utilize fast gradient methods (e.g. Nesterov's acceleration, conjugate gradient, heavy ball) for the purposes of stochastic optimization due to their instability and error accumulation, a notion made precise in d'Aspremont 2008 and Devolder, Glineur, …

Framework for optimizing portfolios under model uncertainty.

We introduce a novel and efficient sampling algorithm for the Multiplicative Attribute Graph Model (MAGM - Kim and Leskovec (2010)}). Our algorithm is \emph{strictly} more efficient than the algorithm proposed by Yun and Vishwanathan (2012), in the sense that our method extends the \emph{best} time complexity guarantee…

Large-scale machine learning training suffers from two prior challenges, specifically for nuclear-norm constrained problems with distributed systems: the synchronization slowdown due to the straggling workers, and high communication costs. In this work, we propose an asynchronous Stochastic Frank Wolfe (SFW-asyn) metho…

We study curvature dimension inequalities for the sub-Laplacian on contact Riemannian manifolds. This new curvature dimension condition is then used to obtain: 1) Geometric conditions ensuring the compactness of the underlying manifold (Bonnet-Myers type results); 2) Volume estimates of metric balls; 3) Gradient bounds…

Optimizes web page freshness with limited crawling frequencies.

3-balls in 4-sphere become isotopic in 5-ball.

Study on ball widths and minimal submanifolds in space forms.

Balls-and-Bins sampling improves DP-SGD privacy and utility.

Localized uncertainty attacks target uncertain regions to create imperceptible adversarial examples.

Paper studies stochastic optimization methods with momentum, proving convergence and avoiding traps.

This paper describes a method to construct standard 4-balls from homotopy 4-balls in $\mathbb{C}^2$.

Study minimal networks on spheres and balls near standard metrics.

In this paper, we study punctured spheres in two dimensional ball quotient compactifications $(X, D)$. For example, we show that smooth toroidal compactifications of ball quotients cannot contain properly holomorphically embedded $3$-punctured spheres. We also use totally geodesic punctured spheres to prove ampleness o…

Two minimal hypersurfaces in a ball intersect in any half-ball.

Paper uses Stochastic Mirror Descent for large-scale sparse recovery problems.

The choice of how to retain information about past gradients dramatically affects the convergence properties of state-of-the-art stochastic optimization methods, such as Heavy-ball, Nesterov's momentum, RMSprop and Adam. Building on this observation, we use stochastic differential equations (SDEs) to explicitly study t…

Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide significant improvements over stochastic gradient descent (SGD). Rigorously speak…

New surface area measures defined for ball-convex bodies, leading to entropy and inequalities.

Sharp lower bound found for geodesic ball eigenvalues.