New algorithm reduces regret in stochastic bandit convex optimization.
problem Optimizing decisions in uncertain environments with convex losses.
method Introduces a second-order method for zeroth-order stochastic convex bandits.
result Regret bound of ( 1 + r / d ) [ d 1.5 n + d 3 ] p o l y l o g ( n , d , r ) (1 + r/d)[d^{1.5} \sqrt{n} + d^3] polylog(n, d, r) ( 1 + r / d ) [ d 1.5 n + d 3 ] p o l y l o g ( n , d , r ) . In this paper, we study stochastic non-convex optimization with non-convex random functions. Recent studies on non-convex optimization revolve around establishing second-order convergence, i.e., converging to a nearly second-order optimal stationary points. However, existing results on stochastic non-convex optimizatio…
Study on convex ordering in stochastic control for swing contracts, proving value function convexity.
problem Pricing of swing contracts under stochastic dynamics.
method Discrete-time stochastic optimal control problem, convexity propagation, Brownian diffusion model, Stein's formula.
result Value function is convex in underlying asset price, relaxation of convexity assumption for semi-convexity.
Expands learning paradigm to stochastic orders using Choquet-Toland distance and Variational Dominance Criterion.
problem Learning high-dimensional distributions with stochastic orders.
method Introduces Choquet-Toland distance and Variational Dominance Criterion, uses input convex maxout networks (ICMNs).
result Proposes surrogates for Choquet-Toland distance and Variational Dominance Criterion with parametric rates.
Optimized method tackles convex optimization with heavy-tailed noise.
problem Convex optimization problems with noisy gradients.
method Vanilla stochastic proximal subgradient method without gradient clipping or normalization.
result Achieves optimal complexity for various convex optimization types under heavy-tailed noise.
New characterization of second-order stochastic dominance with applications in risk management.
problem Characterizing second-order stochastic dominance.
method Properties of Expected Shortfall risk measures.
result New interpretation and proof techniques for second-order stochastic dominance.
New algorithm finds approximate stationary points in non-convex optimization.
problem Finding approximate stationary points in non-convex stochastic optimization.
method Design of an algorithm using O ( ε − 3 ) O(ε^{-3}) O ( ε − 3 ) stochastic gradient and Hessian-vector products. result Optimal rate of O ( ε − 3 ) O(ε^{-3}) O ( ε − 3 ) for finding ε ε ε -approximate stationary points, matching lower bounds. New model shows VIX futures are more expensive than local volatility model suggests.
problem VIX futures pricing under local volatility model is incorrect.
method Developed a continuous stochastic volatility model to show VIX futures are more expensive than local volatility model.
result Inversion of convex ordering between local and stochastic variances observed in SPX market for short maturities.
New adaptive methods solve weakly convex stochastic optimization problems.
problem Solving weakly convex stochastic optimization problems.
method Adaptive first and zeroth-order methods using exponential moving averages.
result Established non-asymptotic convergence rates for nonsmooth and nonconvex problems.
SPIDER optimizes non-convex problems with reduced gradient computations.
problem Non-convex optimization problems with limited gradient information.
method Stochastic Path-Integrated Differential Estimator (SPIDER) combined with gradient descent.
result SPIDER-SFO and SPIDER-SFO extsuperscript{+} achieve optimal gradient computation costs for non-convex optimization.
New algorithm optimizes convex functions with noisy evaluations in one dimension.
problem Optimizing convex functions with noisy zero-order evaluations in one dimension.
method Proposed a computationally efficient algorithm achieving O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) convergence rate. result Achieved the optimal O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) convergence rate, closing the gap in one dimension. Exact second-order optimization for deep learning reduces computational cost and improves performance.
problem Inadequate use of second-order optimization methods in deep learning due to high computational cost and non-convexity.
method Developed an exact stochastic second-order Newton method that addresses the non-convexity issue and provides an expression for the stochastic Hessian.
result Exact second-order Newton direction formula and its application in deep learning datasets.
We propose a method for zeroth order stochastic convex optimization that attains the suboptimality rate of O ~ ( n 7 T − 1 / 2 ) \tilde{\mathcal{O}}(n^{7}T^{-1/2}) O ~ ( n 7 T − 1/2 ) after T T T queries for a convex bounded function f : R n → R f:{\mathbb R}^n\to{\mathbb R} f : R n → R . The method is based on a random walk (the \emph{Ball Walk}) on the epigraph of the function. Th…
We consider the problem of stochastic comparison of general Garch-like processes, for different parameters and different distributions of the innovations. We identify several stochastic orders that are propagated from the innovations to the Garch process itself, and discuss their interpretations. We focus on the convex…
New algorithms optimize non-smooth, non-convex objectives with improved complexity.
problem Optimizing non-smooth, non-convex stochastic objectives.
method Reduction to online learning, applying optimistic online learning techniques.
result Improved complexity for finding ( δ , ε ) (δ,ε) ( δ , ε ) -stationary points. Stochastic methods tackle inexact Hessian and gradient computations in large-scale non-convex optimization.
problem Efficiently solving non-convex optimization problems with inexact Hessian and gradient computations.
method Stochastic trust region and cubic regularization methods with inexact gradient, Hessian, and function values.
result Achieves ε-approximate second-order optimality with similar iteration complexity as exact computations.
Optimal algorithms for online convex optimization with random order.
problem Online convex optimization with random order and non-convex loss functions.
method Stochastic gradient descent and algorithmic stability analysis.
result Achieves optimal bounds and significantly outperforms previous methods.
Proposes a new method for optimizing large-scale models using Nyström approximation of the Hessian.
problem Optimizing non-convex functions like deep learning models using second-order methods.
method Nyström-approximated curvature for stochastic optimization of large-scale empirical risk minimization.
result The proposed method achieves performance competitive with state-of-the-art first-order and stochastic quasi-Newton methods.
New algorithm tackles risk-aware learning problems efficiently.
problem Risk-aware learning with mean-semideviation objective.
method Zeroth-order compositional stochastic optimization algorithm.
result Algorithm converges to optimal solutions with explicit rates.
Unified approach for first-order methods with Markovian noise in stochastic optimization and variational inequalities.
problem Stochastic optimization problems with Markovian noise.
method Unified theoretical analysis of first-order gradient methods using randomized batching and multilevel Monte Carlo.
result Optimal (linear) dependence on the mixing time of the noise sequence, eliminating previous limiting assumptions.
Novel BSG method for efficient stochastic optimization.
problem Efficient optimization of non-convex surfaces in stochastic settings.
method Binary search combined with first order gradient optimization.
result BSG produces more promising results and better generalization than other methods.
Optimizes CM for stochastic convex optimization with progressive precision.
problem Stochastic nature of objective function in convex optimization.
method Iterative coordinate minimization with optimal precision control.
result Order-optimal regret performance for strongly convex and nonsmooth functions.
New lower bounds for bilevel optimization with first-order oracles.
problem Complexity of bilevel optimization with first-order oracles.
method Development of hard instances and proof of lower bounds.
result Nontrivial lower bounds for first-order zero-respecting algorithms.
Defines diversification as a binary relationship between financial portfolios.
problem Defines diversification in a new binary relationship for financial portfolios.
method Proposes a new definition of diversification based on convex linear combinations and second order stochastic dominance.
result The proposed definition coincides with second order stochastic dominance.
Accelerates stochastic optimization for convex and strongly convex problems.
problem Improving convergence rates in noisy stochastic optimization.
method Extends Catalyst approach to stochastic settings, handles inexact proximal operators.
result Achieves optimal worst-case complexity for noise-dominated regions.
Paper tackles private optimization for non-smooth objectives efficiently.
problem Private stochastic convex optimization for non-smooth objectives.
method Noisy mirror descent algorithm.
result Achieves optimal rates in statistical complexity and number of queries.
New model estimates higher-order interactions in stochastic processes using lower-dimensional projections.
problem Estimating higher-order interaction effects in stochastic processes with limited data.
method Additive Poisson Process (APP) combines information geometry and generalized additive models to model intensity functions in lower dimensions.
result The model can estimate higher-order intensity functions with sparse data.
New adaptive step-size method for convex optimization without tuning.
problem Optimizing convex functions efficiently with stochastic gradients.
method Adapted Adaptive Gradient Descent Without Descent to stochastic setting.
result Stochastic gradient descent converges under various assumptions.
Adaptive algorithm AMSGrad converges for weakly convex constrained optimization problems.
problem Solving constrained stochastic optimization problems with weakly convex objectives.
method Analysis of AMSGrad algorithm for a specific class of problems.
result AMSGrad achieves a convergence rate of i l d e O ( t − 1 / 4 ) \mathcal{ ilde O}(t^{-1/4}) i l d e O ( t − 1/4 ) for the norm of the gradient of the Moreau envelope. A new hybrid-ordered SGD method reduces communication and complexity for non-convex optimization.
problem Balancing communication, computational complexity, and convergence rate in distributed non-convex optimization.
method Hybrid-ordered distributed SGD with pre-shared scalers and periodic vector communication.
result Order-wise faster convergence compared to existing methods.
Paper tackles non-convex inf-projection problems with stochastic optimization.
problem Non-convex and possibly non-smooth inf-projection minimization problems.
method Developed stochastic algorithms for finding (nearly) stationary solutions.
result Established first-order convergence for non-convex inf-projection problems.
Paper finds optimal transport measures for arbitrage strategies.
problem Link between convex order and arbitrage strategies.
method Develops algorithms and models for finding optimal transport measures.
result Constructs a model-independent arbitrage strategy.
Study shows Stochastic Mirror Descent optimizes convex problems with infinite noise variance.
problem Optimizing convex problems with infinite noise variance.
method Stochastic Mirror Descent algorithm with uniformly convex mirror maps.
result Demonstrates convergence rate quantified in terms of iterations, dimensionality, and geometric parameters.
Proposes a new stochastic quasi-Newton method with Nesterov's acceleration.
problem Improving convergence in large-scale non-convex optimization problems.
method Stochastic quasi-Newton method with Nesterov's accelerated gradient.
result Improved performance compared to classical and popular methods.
Adaptive and robust algorithm for convex optimization under noisy feedback.
problem Minimizing unknown convex functions with noisy gradient feedback.
method Biased random walk on a binary tree with adaptive confidence bounds.
result Achieves optimal regret orders for various function classes.
Geodesic convexity generalizes the notion of (vector space) convexity to nonlinear metric spaces. But unlike convex optimization, geodesically convex (g-convex) optimization is much less developed. In this paper we contribute to the understanding of g-convex optimization by developing iteration complexity analysis for …
New algorithm reduces dimensionality in stochastic optimization.
problem Stochastic optimization in high-dimensional problems.
method Proposes a sparsity-inducing stochastic gradient-free (SI-SGF) algorithm.
result Proves dimension-free query complexity in convex and strongly convex cases.
This paper deals with a natural stochastic optimization procedure derived from the so-called Heavy-ball method differential equation, which was introduced by Polyak in the 1960s with his seminal contribution [Pol64]. The Heavy-ball method is a second-order dynamics that was investigated to minimize convex functions f .…
Efficient algorithm for zeroth-order bandit convex optimization with bounds on regret.
problem Optimizing in unknown, noisy environments with limited information.
method Online Newton Method for bandit convex optimization, proving regret bounds.
result Regret bounds for both adversarial and stochastic settings.
A new method for distributed optimization with noisy function evaluations.
problem Distributed optimization with noisy function evaluations.
method Zero-order one-point estimate with distributed stochastic gradient-tracking technique.
result The method converges almost surely to the optimum with a rate of O ( 1 k ) O(\frac{1}{\sqrt{k}}) O ( k 1 ) . Stochastic second-order methods converge fast under interpolation conditions.
problem Minimizing smooth and strongly-convex functions efficiently.
method Regularized subsampled Newton method (R-SSN) and stochastic BFGS algorithms.
result R-SSN achieves global linear convergence and quadratic rate in a local neighbourhood.
Paper tackles zeroth-order optimization for nonconvex problems with constraints, high-dimensions, and saddle-points.
problem Optimization of nonconvex functions with constraints and high-dimensionality, avoiding saddle-points.
method Proposes zeroth-order stochastic approximation algorithms, including conditional gradient and truncated gradient methods, and a zeroth-order cubic regularization Newton's method.
result Demonstrates algorithms achieving rates similar to standard stochastic gradient methods, with rates dependent on poly-logarithmic dimensionality.
The stochastic gradient (SG) method can minimize an objective function composed of a large number of differentiable functions, or solve a stochastic optimization problem, to a moderate accuracy. The block coordinate descent/update (BCD) method, on the other hand, handles problems with multiple blocks of variables by up…
We propose a new stochastic L-BFGS algorithm and prove a linear convergence rate for strongly convex and smooth functions. Our algorithm draws heavily from a recent stochastic variant of L-BFGS proposed in Byrd et al. (2014) as well as a recent approach to variance reduction for stochastic gradient descent from Johnson…
A novel distributed method tracks gradients for convex optimization over networks.
problem Distributed optimization of strongly-convex functions over a network.
method S-AB algorithm using auxiliary variables and row/column stochastic weights.
result Linear convergence to a neighborhood of the global minimizer.
Lower bounds on queries needed for finding stationary points in non-convex optimization.
problem Finding ε ε ε -stationary points in non-convex stochastic optimization. method Proving lower bounds on the number of queries required by stochastic first-order methods.
result Lower bounds on the number of queries required to find ε ε ε -stationary points are tight and optimal. In this note we establish some appropriate conditions for stochastic equality of two random variables/vectors which are ordered with respect to convex ordering or with respect to supermodular ordering. Multivariate extensions of this result are also considered.
We present a novel statistical inference framework for convex empirical risk minimization, using approximate stochastic Newton steps. The proposed algorithm is based on the notion of finite differences and allows the approximation of a Hessian-vector product from first-order information. In theory, our method efficient…