New algorithm reduces complexity for optimizing complex machine learning tasks.
problem Optimizing complex machine learning objectives like reinforcement learning and portfolio management.
method Developed SARAH-Compositional algorithm using Stochastic Recursive Gradient Descent.
result Achieved optimal IFO complexity bounds for stochastic compositional optimization.
New bounds on complexity for finding near-stationary points in stochastic convex optimization.
problem Finding near-stationary points in stochastic convex optimization.
method Joint analysis of local stochastic oracle and global oracle models; extensions of recursive regularization technique.
result Logarithmic dependence on smoothness in global oracle model for finding near-stationary points.
Method solves complex optimization problems with high probability bounds.
problem Nonlinear equality constrained stochastic optimization problems.
method Step-search sequential quadratic programming method.
result High-probability bound on iteration complexity for first-order stationarity.
Simplified calculus for semimartingales makes complex transformations easier.
problem Complex transformations of semimartingales.
method Unified treatment of transformations for real and complex semimartingales.
result Unified calculus for semimartingales simplifies various transformations.
Paper proposes an algorithm to solve complex minimax problems efficiently.
problem Stochastic nonconvex-concave minimax problems in various fields.
method Accelerated first-order regularized momentum descent ascent algorithm (FORMDA).
result Achieves best-known complexity bound of ildeO(ε−6.5) for single-loop algorithms. New method solves complex optimization problems with reduced sample complexity.
problem Solving nonconvex stochastic nested optimization problems.
method Stochastic ADMM approach to find ε-stationary points.
result Total sample complexity of O(ε^(-3)) for online case and O((2N_1 + N_2) + (2N_1 + N_2)^(1/2)ε^(-2)) for finite sum case.
Improved loss scaling for stochastic momentum algorithms in high dimensions.
problem Improving loss scaling for stochastic momentum algorithms in high dimensions.
method Dimension-adapted Nesterov acceleration (DANA) scales momentum hyperparameters based on model size and data complexity.
result DANA improves loss scaling exponents across various data and target complexities.
This paper constructs Brownian motion on complex flag manifolds and finds joint distribution of stochastic areas.
problem Modeling stochastic areas on complex partial flag manifolds.
method Constructs Brownian motion on complex partial flag manifolds and uses it to find joint distribution of stochastic areas.
result Limit law of stochastic areas is a multivariate Cauchy distribution.
Study on stochastic mean curvature flow on networks using Ito calculus.
problem Understanding the dynamics of network structures under random influences.
method Application of Ito calculus to derive a stochastic differential equation (SDE) for network edges.
result New insights into the stability, long-term behavior, and pattern formation of complex networks under stochastic influences.
New method reduces variance in stochastic optimization with high confidence.
problem Achieving high-probability guarantees in stochastic optimization with weaker noise assumptions.
method Stochastic proximal point method combining proximal subproblem solver and probability booster.
result Demonstrates convergence with low sample complexity under bounded variance assumptions.
New hybrid SGD algorithms improve stochastic nonconvex optimization complexity.
problem Solving stochastic nonconvex optimization problems efficiently.
method Hybrid SARAH-SGD algorithm combining biased and unbiased estimators.
result Achieves better complexity bound for ε-stationary points. The paper analyzes the sample complexity of SAA for CSO problems.
problem Finding efficient methods for solving CSO problems with limited samples.
method Establishes sample complexity of SAA for CSO under various structural assumptions.
result Total sample complexity improvements from d/eps^4 to d/eps^3 and 1/eps^2 under different conditions.
Improved zeroth-order algorithms tackle nonconvex minimax problems with reduced complexity.
problem Nonconvex minimax optimization problems in machine learning.
method Design and analysis of Zeroth-Order Gradient Descent Ascent ( exttt{ZO-GDA}) and Zeroth-Order Gradient Descent Multi-Step Ascent ( exttt{ZO-GDMSA}) algorithms.
result Oracle complexity improvements for minimax optimization problems.
New algorithms solve non-convex optimization problems efficiently.
problem Non-convex stochastic compositional optimization problems.
method Developed two stochastic Gauss-Newton algorithms.
result Established global oracle complexity for stochastic Gauss-Newton methods.
New algorithms reduce complexity for solving nonconvex optimization problems with stochastic objectives and constraints.
problem Solving nonconvex optimization problems with stochastic objectives and constraints.
method Single-loop quadratic penalty and augmented Lagrangian algorithms with variance reduction techniques.
result Achieved best-known complexity guarantees for solving nonconvex optimization problems with stochastic objectives and constraints.
New algorithms solve stochastic variational inequalities without bounded variance assumption.
problem Solving stochastic variational inequalities without bounded variance assumption.
method Developed algorithms for two classes of problems: monotone and structured nonmonotone VIs.
result Oracle complexity of O(ε^-4) for solving VIs with unbounded domains and possibly unbounded variance.
New technique reduces bias in CSO problems, improving sample complexity.
problem Reducing bias in conditional stochastic optimization problems.
method Introducing a stochastic extrapolation technique combined with variance reduction.
result Achieved significantly better sample complexity for nonconvex smooth objectives.
Proposes a method to reduce parallel complexity of MLMC in SGD.
problem Poor scalability of MLMC in SGD on parallel platforms.
method Proposes a delayed MLMC gradient estimator to reduce parallel complexity.
result Proves reduction in average parallel complexity per iteration at the cost of slightly worse convergence rate.
Stochastic Variance-Reduced Cubic regularization (SVRC) algorithms have received increasing attention due to its improved gradient/Hessian complexities (i.e., number of queries to stochastic gradient/Hessian oracles) to find local minima for nonconvex finite-sum optimization. However, it is unclear whether existing SVR…
This work introduces a new model for complex stochastic processes.
problem Difficulties in representing non-stationary distributions with conventional models.
method Recurrent Autoregressive Flows using normalizing flows with recurrent neural connections.
result Demonstrates the effectiveness of the proposed model through experiments.
A new biased gradient descent method for conditional stochastic optimization.
problem Challenges in constructing unbiased gradient estimators for conditional stochastic optimization.
method Proposes a biased stochastic gradient descent (BSGD) algorithm and analyzes its sample complexities.
result Establishes sample complexities of BSGD for various objectives and shows that BSpiderBoost matches the lower bound complexity.
Improved complexity for machine learning optimization methods.
problem Optimizing over-parametrized models in machine learning.
method Stochastic conditional gradient methods with interpolation-like conditions.
result Improved oracle complexities for finding optimal solutions.
New methods solve optimization problems with heavy-tailed noise, improving upon existing complexity bounds.
problem Optimization problems with heavy-tailed noise and weakly average smoothness.
method Normalized stochastic first-order methods with Polyak, multi-extrapolated, and recursive momentum.
result First-order oracle complexity results for finding approximate stochastic stationary points under heavy-tailed noise.
Optimized method tackles convex optimization with heavy-tailed noise.
problem Convex optimization problems with noisy gradients.
method Vanilla stochastic proximal subgradient method without gradient clipping or normalization.
result Achieves optimal complexity for various convex optimization types under heavy-tailed noise.
Proves minimax sample complexity for turn-based stochastic games.
problem Proving theoretical guarantees for reinforcement learning in turn-based stochastic games.
method Developing absorbing TBSG and reward perturbation techniques to handle statistical dependence.
result Empirical Nash equilibrium strategy approximates true Nash equilibrium in turn-based stochastic games.
We solve a complex optimization problem for Wasserstein barycenters using stochastic methods.
problem Optimizing the average of multiple probability distributions in a streaming data setting.
method We reformulate the problem as a convex-concave saddle-point problem and propose a stochastic optimization algorithm.
result Our algorithm has better complexity than existing methods for arbitrary distributions.
New algorithms solve complex multi-level optimization problems with improved efficiency.
problem Smooth stochastic multi-level composition optimization problems.
method Two algorithms using moving-average and linearized stochastic estimates.
result Achieved sample complexities of O(1/ε^4) and O(1/ε^6).
Paper solves discounted stochastic games with near-optimal time and sample complexity.
problem Solving discounted stochastic two-player games with optimal complexity.
method Generalizes Q-learning to two-player strategy computation, overcoming limitations of existing methods.
result Near-optimal ε-strategy computation with polylogarithmic factors in 1−γ and ε−2. Optimal algorithms for Riemannian optimization with reduced complexity.
problem Stochastic optimization on Riemannian manifolds with limited data.
method Zeroth-order Riemannian Averaging Stochastic Approximation algorithms using Riemannian moving-average estimators and novel geometric conditions.
result Achieves optimal sample complexities for generating approximate first-order stationary solutions.
Deep learning solves complex stochastic control with jumps.
problem Solving high-dimensional stochastic control tasks with jumps.
method Model-based approach using two neural networks, iteratively trained with objectives derived from the Hamilton-Jacobi-Bellman equation.
result Demonstrates effectiveness in solving complex high-dimensional stochastic control tasks.
Paper tackles Hessian/Jacobian-free stochastic bilevel optimization with O(ε−1.5) complexity.
problem Nonconvex-strongly-convex bilevel optimization problem.
method FdeHBO optimizer with finite-difference Hessian/Jacobian-vector approximation and momentum.
result FdeHBO achieves O(ε−1.5) iterations for ε-accurate stationary point. Improved stochastic Halpern iteration for fixed-point approximation in normed spaces.
problem Approximating fixed-points of nonexpansive and contractive operators in normed finite-dimensional spaces.
method Stochastic Halpern iteration with minibatch, analyzing oracle complexity.
result Improved oracle complexity for nonexpansive operators, with a lower bound of Ω(ε−3). Paper introduces a new deep-learning method for quantum mechanics.
problem Simulating time-evolving Schrödinger equations efficiently.
method Generative diffusion models and stochastic mechanics.
result Significantly lower computational complexity compared to existing methods.
Improved time complexity for parallel stochastic optimization in heterogeneous systems.
problem Time complexity in parallel stochastic optimization for large-scale machine learning models.
method Proposes Rennala MVR, a variance-reduced extension of Rennala SGD based on momentum-based variance reduction.
result Variance reduction improves time complexity in relevant parameter regimes for parallel stochastic optimization in heterogeneous systems.
A new hybrid algorithm reduces stochastic gradient evaluations for nonconvex optimization.
problem Solving stochastic composite nonconvex optimization problems efficiently.
method Proposes a new hybrid variance-reduced proximal gradient method with a stochastic gradient estimator.
result Achieves optimal stochastic oracle complexity bound with one less gradient evaluation.
Stochastic gradient descent is the method of choice for large-scale machine learning problems, by virtue of its light complexity per iteration. However, it lags behind its non-stochastic counterparts with respect to the convergence rate, due to high variance introduced by the stochastic updates. The popular Stochastic …
New PG methods tackle nonconvex optimization with auto-conditioned stepsizes.
problem Optimizing nonconvex functions over convex sets.
method Auto-conditioned projected gradient (AC-PG) methods and stochastic variants.
result Achieved optimal iteration complexity for finding approximate stationary points.
A faster ADMM method for nonconvex optimization with improved complexity.
problem Nonconvex optimization problems in machine learning.
method SPIDER-ADMM, a stochastic ADMM method using a new differential estimator.
result Achieves optimal IFO complexity of O(n+n1/2ε−1) for finding an ε-approximate stationary point. In this paper we study stochastic quasi-Newton methods for nonconvex stochastic optimization, where we assume that noisy information about the gradients of the objective function is available via a stochastic first-order oracle (SFO). We propose a general framework for such methods, for which we prove almost sure conve…
Develops minibatch stochastic proximal gradient for large-scale learning models.
problem Finding optimal predictors with complex regularizers in large-scale learning models.
method Minibatch variants of stochastic proximal gradient algorithm for composite objective functions.
result Minibatch size N after O(Nε1) iterations achieves ε−suboptimality in expected quadratic distance. Scalable Gaussian process models trained with unbiased stochastic ELBO.
problem Training large capacity Gaussian process models on huge datasets.
method Unbiased stochastic variational inference for scalable GPs.
result Accurate inference on large datasets with up to 10 million basis functions.
A new method for efficient Gaussian process regression reduces complexity and improves scalability.
problem Efficient Gaussian process regression for large datasets.
method Learnable coreset-based variational inference for Gaussian processes.
result CVGP reduces the dimensionality of the variational parameter search space to linear complexity.
SFLS method finds feasible solutions faster with less data.
problem Efficiently solving SOECs with near-feasibility and near-optimality.
method SFLS method that emphasizes feasibility before convergence.
result SFLS maintains high-probability feasibility at each iteration.
PASTIS selects minimal models from stochastic dynamics data.
problem Overfitting in model selection for stochastic dynamics.
method Combining likelihood-estimation statistics with extreme value theory.
result PASTIS reliably identifies minimal models, even with low sampling rates or error.
Relative to the large literature on upper bounds on complexity of convex optimization, lesser attention has been paid to the fundamental hardness of these problems. Given the extensive use of convex optimization in machine learning and statistics, gaining an understanding of these complexity-theoretic issues is importa…
New methods reduce constraint violations to certainty in stochastic optimization.
problem Finding a point with certain constraint satisfaction and near-stationarity.
method Single-loop variance-reduced stochastic first-order methods with truncated momentum schemes.
result Achieves strong convergence guarantees for ε-stochastic stationary points with certain constraint satisfaction. In the paper, we study the stochastic alternating direction method of multipliers (ADMM) for the nonconvex optimizations, and propose three classes of the nonconvex stochastic ADMM with variance reduction, based on different reduced variance stochastic gradients. Specifically, the first class called the nonconvex stoch…
Improved algorithm reduces stochastic gradient complexity for large-scale learning problems.
problem High stochastic gradient complexity for large-scale learning problems.
method Hybrid Stochastic-Deterministic Minibatch Proximal Gradient (HSDMPG) algorithm.
result Achieves nearly optimal generalization in less than a single pass over data.