arXiv research

The height function of various surfaces decomposes into finite sums of scaled and translated versions of itself.

SVRN accelerates Newton methods by reducing variance and improving performance.

We describe a novel optimization method for finite sums (such as empirical risk minimization problems) building on the recently introduced SAGA method. Our method achieves an accelerated convergence rate on strongly convex smooth problems. Our method has only one parameter (a step size), and is radically simpler than o…

The problem of minimizing sum-of-nonconvex functions (i.e., convex functions that are average of non-convex ones) is becoming increasingly important in machine learning, and is the core machinery for PCA, SVD, regularized Newton's method, accelerated non-convex optimization, and more. We show how to provably obtain an …

The popular cubic smoothing spline estimate of a regression function arises as the minimizer of the penalized sum of squares $\sum_j(Y_j - μ(t_j))^2 + λ\int_a^b [μ"(t)]^2 dt$, where the data are $t_j,Y_j$, $j=1,..., n$. The minimization is taken over an infinite-dimensional function space, the space of all functions wi…

We propose an optimization method for minimizing the finite sums of smooth convex functions. Our method incorporates an accelerated gradient descent (AGD) and a stochastic variance reduction gradient (SVRG) in a mini-batch setting. Unlike SVRG, our method can be directly applied to non-strongly and strongly convex prob…

Stochastic optimization algorithms with variance reduction have proven successful for minimizing large finite sums of functions. Unfortunately, these techniques are unable to deal with stochastic perturbations of input data, induced for example by data augmentation. In such cases, the objective is no longer a finite su…

Novel analysis of EFP for finite-sum problems in neural networks.

We study the conditions under which one is able to efficiently apply variance-reduction and acceleration schemes on finite sum optimization problems. First, we show that, perhaps surprisingly, the finite sum structure by itself, is not sufficient for obtaining a complexity bound of $\tilde{\cO}((n+L/μ)\ln(1/ε))$ for $L…

Recent advances in randomized incremental methods for minimizing $L$-smooth $μ$-strongly convex finite sums have culminated in tight complexity of $\tilde{O}((n+\sqrt{n L/μ})\log(1/ε))$ and $O(n+\sqrt{nL/ε})$, where $μ>0$ and $μ=0$, respectively, and $n$ denotes the number of individual functions. Unlike incremental me…

Paper establishes lower bounds for finite-sum optimization problems using novel construction methods.

Study best-response learning dynamics in zero-sum polymatrix games under full and minimal information settings.

Two new Frank-Wolfe algorithms improve convergence for constrained optimization.

SignSVRG improves SignSGD by reducing variance, achieving similar convergence rates.

This paper presents a lower bound for optimizing a finite sum of $n$ functions, where each function is $L$-smooth and the sum is $μ$-strongly convex. We show that no algorithm can reach an error $ε$ in minimizing all functions from this class in fewer than $Ω(n + \sqrt{n(κ-1)}\log(1/ε))$ iterations, where $κ=L/μ$ is a …

We consider manifolds $M^{2n}$ which admit smooth maps into a connected sum of $S^1\times S^n$ with only finitely many critical points, for $n\in\{2,4,8\}$, and compute the minimal number of critical points.

We consider the problem of minimizing the composition of a smooth (nonconvex) function and a smooth vector mapping, where the inner mapping is in the form of an expectation over some random variable or a finite sum. We propose a stochastic composite gradient method that employs an incremental variance-reduced estimator…

This paper develops a Hoeffding inequality for the partial sums $\sum_{k=1}^n f (X_k)$, where $\{X_k\}_{k \in \mathbb{Z}_{> 0}}$ is an irreducible Markov chain on a finite state space $S$, and $f : S \to [a, b]$ is a real-valued function. Our bound is simple, general, since it only assumes irreducibility and finiteness…

Study finds bound on energy of minimal spheres on complex manifolds.

DESTRESS optimizes decentralized nonconvex optimization with optimal IFO complexity and efficient communication.

In this paper, we consider immersed two-sided minimal hypersurfaces in $\mathbb{R}^n$ with finite total curvature. We prove that the sum of the Morse index and the nullity of the Jacobi operator is bounded from below by a linear function of the number of ends and the first Betti number of the hypersurface. When $n=4$, …

In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder p…

Characterizes compact complex surfaces with finite homotopy rank-sum.

A new algorithm improves convergence rates for convex optimization problems.

The paper analyzes the variance of different shuffling methods in stochastic gradient descent.

We propose the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG me…

Recent advances in optimization theory have shown that smooth strongly convex finite sums can be minimized faster than by treating them as a black box "batch" problem. In this work we introduce a new method in this class with a theoretical convergence rate four times faster than existing methods, for sums with sufficie…

In this note we complete the discussion of minimality of symplectic fiber sums. We find, that for fiber sums along spheres the minimality of the sum is determined by the cases discussed by M. Usher and one additional case: If the sum is the result of the rational blow-down of a symplectic -4-sphere in X, then it is non…

We exhibit many examples of closed symplectic manifolds on which there is an autonomous Hamiltonian whose associated flow has no nonconstant periodic orbits (the only previous explicit example in the literature was the torus T^2n (n\geq 2) with an irrational symplectic structure). The underlying smooth manifolds of our…

Characterizes Stein surfaces with finite homotopy rank-sum.

Many machine learning, statistical inference, and portfolio optimization problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG)…

Genus 3 Heegaard groups of lens space connected sums are finitely generated.

In supervised learning using kernel methods, we often encounter a large-scale finite-sum minimization over a reproducing kernel Hilbert space (RKHS). Large-scale finite-sum problems can be solved using efficient variants of Newton method, where the Hessian is approximated via sub-samples of data. In RKHS, however, the …

We prove for any positive integer $n$ there exist boundary-sum irreducible ${\mathbb Z}_n$-corks with Stein structure. Here `boundary-sum irreducible' means the manifold is indecomposable with respect to boundary-sum. We also verify that some of the finite order corks admit hyperbolic boundary by HIKMOT.

We prove qualitative estimates on the total curvature of closed minimal hypersurfaces in closed Riemannian manifolds in terms of their index and area, restricting to the case where the hypersurface has dimension less than seven. In particular, we prove that if we are given a sequence of closed minimal hypersurfaces of …

Freya PAGE optimizes nonconvex optimization with heterogeneous, asynchronous workers.

We give a short proof of a conjecture of Stipsicz on the minimality of fiber sums of Lefschetz fibrations, which was proved earlier by Usher. We then construct the first examples of genus g > 1 Lefschetz fibrations on minimal symplectic 4-manifolds which, up to diffeomorphisms of the summands, admit unique decompositio…

New methods optimize sums of bivariate functions on finite domains.

We propose a fast proximal Newton-type algorithm for minimizing regularized finite sums that returns an $ε$-suboptimal point in $\tilde{\mathcal{O}}(d(n + \sqrt{κd})\log(\frac{1}ε))$ FLOPS, where $n$ is number of samples, $d$ is feature dimension, and $κ$ is the condition number. As long as $n > d$, the proposed method…

Smooth finite-sum optimization has been widely studied in both convex and nonconvex settings. However, existing lower bounds for finite-sum optimization are mostly limited to the setting where each component function is (strongly) convex, while the lower bounds for nonconvex finite-sum optimization remain largely unsol…

New lower bounds for gradient methods in strongly convex finite-sum optimization.

We consider the problem of finding the minimizer of a function $f: \mathbb{R}^d \rightarrow \mathbb{R}$ of the finite-sum form $\min f(w) = 1/n\sum_{i}^n f_i(w)$. This problem has been studied intensively in recent years in the field of machine learning (ML). One promising approach for large-scale data is to use a stoc…

We prove existence and a.e. regularity of an area minimizing soap film with a bound on energy spanning a given Jordan curve in R^3. The energy of a film is defined to be the sum of its surface area and the length of its singular branched set. The class of surfaces over which area is minimized includes images of disks, …

We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-…

The warping sum $e(K)$ of a knot $K$ is the minimal value of the sum of the warping degrees of a minimal diagram of $K$ with both orientations. In this paper, knots $K$ with $e(K) \le 3$ are characterized, and some knots $K$ with $e(K)=4$ are given.

Unified framework for stability and generalization of Push-Sum in decentralized learning over directed graphs.

Characterizes symmetric Bernoulli distributions with minimal convex sums.

New minimal tori found in curved spaces.