Paper proves Pareto efficient insurance for multiple entities.
problem Optimizing insurance for multiple policyholders and insurers.
method Sum-minimization characterization and pairwise implementability analysis.
result Characterization of Pareto efficient insurance arrangements.
A new algorithm improves convergence rates for convex optimization problems.
problem Convex optimization problems with finite-sum structure.
method Nesterov Accelerated Shuffling Gradient (NASG) integrating Nesterov's acceleration with different shuffling schemes.
result Improved convergence rate of O(1/T) for unified shuffling schemes.
The paper analyzes the variance of different shuffling methods in stochastic gradient descent.
problem Understanding the variance of different shuffling methods in stochastic gradient descent.
method Power spectral density analysis to study the noise sequences of stochastic gradients.
result The stationary variances of iterates decrease in the order of SGD, SGD-RR, and SGD-SO.
Unified framework for decentralized optimization combining gradient tracking and variance reduction.
problem Solving finite-sum minimization problems in distributed systems with privacy and resource constraints.
method Unified algorithmic framework combining variance-reduction and gradient tracking.
result Unified methods achieve robust performance and fast convergence for smooth and strongly-convex objectives, and are applicable to non-convex problems.
Two new Frank-Wolfe algorithms improve convergence for constrained optimization.
problem Solving optimization problems with structured constraints in machine learning.
method Two new variants of the Frank-Wolfe (FW) method for stochastic finite-sum minimization.
result Best convergence guarantees for convex and non-convex objective functions.
Novel Newton method for large-scale kernel methods using random features.
problem Efficiently solving large-scale finite-sum minimization problems in RKHS.
method Randomized feature-based Newton method for empirical risk minimization.
result Local superlinear and global linear convergence of the method.
In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder p…
Authors find small triangulations for specific 4-manifolds.
problem Finding optimal triangulations for 4-manifolds.
method Triangulated connected sums of CP^2 and S^2×S^2, conjectured minimal pentachora.
result Triangulations have the smallest number of pentachora for their types.
SVRN accelerates Newton methods by reducing variance and improving performance.
problem Improving the efficiency of Newton methods for large-scale optimization problems.
method Stochastic Variance-Reduced Newton (SVRN) algorithm that accelerates Subsampled Newton and Iterative Hessian Sketch algorithms.
result SVRN accelerates Newton methods by reducing the number of passes over the data, achieving a significant improvement in performance.
New algorithm reduces distributed optimization time with stochastic delays.
problem Optimizing distributed data with stochastic delays.
method Developed ADSAGA, a variant of SAGA for distributed-data settings with stochastic delays.
result ADSAGA converges in $ ilde{O}\left(\left(n + \sqrt{m}κ
ight)\log(1/ε)
ight)$ iterations under mean delay m. Decentralized solutions to finite-sum minimization are of significant importance in many signal processing, control, and machine learning applications. In such settings, the data is distributed over a network of arbitrarily-connected nodes and raw data sharing is prohibitive often due to communication or privacy constr…
This paper proposes a framework of L-BFGS based on the (approximate) second-order information with stochastic batches, as a novel approach to the finite-sum minimization problems. Different from the classical L-BFGS where stochastic batches lead to instability, we use a smooth estimate for the evaluations of the gradie…
We target the problem of finding a local minimum in non-convex finite-sum minimization. Towards this goal, we first prove that the trust region method with inexact gradient and Hessian estimation can achieve a convergence rate of order O(1/k2/3) as long as those differential estimations are sufficientl…
In this paper, we propose a StochAstic Recursive grAdient algoritHm (SARAH), as well as its practical variant SARAH+, as a novel approach to the finite-sum minimization problems. Different from the vanilla SGD and other modern stochastic methods such as SVRG, S2GD, SAG and SAGA, SARAH admits a simple recursive framewor…
An algorithm is proposed for solving stochastic and finite sum minimization problems. Based on a trust region methodology, the algorithm employs normalized steps, at least as long as the norms of the stochastic gradient estimates are within a specified interval. The complete algorithm---which dynamically chooses whethe…
We consider variants of trust-region and cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under mild conditions on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve ε-approximate seco…
Trust region and cubic regularization methods have demonstrated good performance in small scale non-convex optimization, showing the ability to escape from saddle points. Each iteration of these methods involves computation of gradient, Hessian and function value in order to obtain the search direction and adjust the r…
Push-SAGA is a decentralized algorithm for directed graphs that converges linearly.
problem Finite-sum minimization over directed graphs with stochastic gradients.
method Combines variance reduction, gradient tracking, and consensus algorithms.
result Achieves linear convergence for smooth and strongly convex problems.
AdaSVRG combines adaptive gradient with SVRG for robust optimization.
problem Variance reduction in finite-sum minimization with unknown constants.
method AdaSVRG uses AdaGrad in SVRG's inner loop to make it robust.
result AdaSVRG achieves optimal gradient evaluations with no need for problem-dependent constants.
New bounds for SGD show improved performance in various settings.
problem Improving convergence bounds for SGD with random permutations.
method Analyzing convergence of SGD with random reshuffling and arbitrary permutations.
result Tighter lower bounds for weighted average iterates in both convex and strongly-convex cases.
A fast method for decentralized non-convex optimization over networks.
problem Decentralized non-convex optimization problems over a network of nodes.
method GT-SAGA, a randomized incremental gradient method that evaluates one component gradient per node per iteration.
result GT-SAGA achieves almost sure and mean-squared convergence to a first-order stationary point for general smooth non-convex problems.
Paper establishes lower bounds for finite-sum optimization problems using novel construction methods.
problem Lower complexity bounds for finite-sum optimization problems with various component functions.
method Developed novel approach to construct hard instances and analyzed PIFO algorithms.
result Established lower complexity bounds for convex-concave and nonconvex-strongly-concave objectives.
Improved robustness in optimization methods using second-order information.
problem Scalability and sensitivity to mini-batch size in optimization methods.
method Mini-Batch Stochastic Variance-Reduced Newton (extttMb−SVRN) algorithm incorporating partial second-order information. result Achieves a fast linear convergence rate independent of mini-batch size for large data sizes.