Paper solves robust convex problems with heavy-tailed noise.
problem Solving convex compositional problems with heavy-tailed noise.
method Sub-Gaussian confidence bounds under weak heavy-tailed noise assumptions, using boosting strategy.
result Achieves nearly optimal high probability convergence result.
In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for compos…
A new ADMM-based algorithm for stochastic composition optimization.
problem Stochastic composition optimization problems in estimation and machine learning.
method com-SVR-ADMM, converges linearly for strongly convex and Lipschitz smooth objectives, and has improved convergence rates.
result com-SVR-ADMM converges linearly for strongly convex and Lipschitz smooth objectives with a rate of O ( log S / S ) O( \log S/S) O ( log S / S ) . Accelerates stochastic optimization for convex and strongly convex problems.
problem Improving convergence rates in noisy stochastic optimization.
method Extends Catalyst approach to stochastic settings, handles inexact proximal operators.
result Achieves optimal worst-case complexity for noise-dominated regions.
Paper introduces variance reduction for infinite datasets with finite-sum structure.
problem Optimizing composite and strongly convex objectives with stochastic perturbations.
method Variance reduction approach for stochastic optimization with composite and strongly convex objectives.
result Convergence rate outperforms SGD with a smaller constant factor.
A generalized optimistic method for saddle point problems with improved complexity.
problem Solving convex-concave saddle point problems efficiently.
method Proposes a generalized optimistic method that includes the optimistic gradient method as a special case, handling constrained saddle point problems with composite objective functions and arbitrary norms.
result Best-known global iteration complexity bounds for first-, second-, and higher-order methods.
Paper analyzes adaptive optimization algorithms and provides new insights.
problem Adaptive optimization algorithms for non-convex and composite objectives.
method New regret decomposition, Bregman divergences, modular analysis.
result Improved variational bounds and new optimistic MD algorithms.
New method solves non-strongly convex optimization problems without quadratic regularization.
problem Minimizing the sum of an average of smooth convex components and a non-differentiable convex function.
method Accelerated randomized mirror descent algorithm without strongly convex assumption.
result Performance of algorithms improved without quadratic regularization.
Paper shows linear convergence of ISTA and FISTA for ill-conditioned images.
problem Solving linear inverse problems with sparse representation in signal and image processing.
method Revisits iterative shrinkage-thresholding algorithms (ISTA) and improves their convergence properties.
result Linear convergence of ISTA and FISTA for strongly convex smooth parts, even in ill-conditioned cases.
Classical stochastic gradient methods are well suited for minimizing expected-value objective functions. However, they do not apply to the minimization of a nonlinear function involving expected values or a composition of two expected-value functions, i.e., problems of the form $\min_x \mathbf{E}_v [f_v\big(\mathbf{E}_…
New method solves complex optimization problems efficiently.
problem Optimizing complex functions with inner expectations in machine learning.
method Combines variance reduction methods with duality-free techniques.
result Proves linear convergence for convex and non-convex cases.
New algorithm solves complex optimization problems with two regularization terms efficiently.
problem Complex optimization problems with two regularization terms, especially composed with linear functions.
method Stochastic Primal-Dual Proximal ExtraGradient descent (SPDPEG) for convex and strongly convex objectives.
result Converges with rates matching best first-order stochastic algorithms.
Adaptive sampling method solves constrained and composite optimization problems.
problem Solving constrained optimization problems with stochastic objectives and deterministic constraints.
method Proximal gradient method with adaptive sampling to improve gradient approximation quality.
result Convergence results established for both strongly convex and general convex objectives.
New approach for distributed online optimization of non-convex losses with sublinear regret.
problem Regret evaluation and consensus in distributed, multi-agent systems with non-convex losses.
method Composite regret metric and consensus-based online normalized gradient (CONGD) approach for pseudo-convex losses; offline optimization oracle for general non-convex losses.
result First sublinear regret bound for general distributed online non-convex learning.
Proposes an accelerated optimization algorithm for composite objectives.
problem Gradient-based optimization with sparse solutions and high dimensions.
method Inexact variable-metric proximal point algorithm (QNing) with limited-memory BFGS.
result Significant improvements over competing methods in training machine learning models.
Extends geometric descent method for convex composite problems.
problem Nonsmooth and strongly convex composite problems.
method Geometric Proximal Gradient Method (GeoPG)
result Achieves optimal linear convergence rate of (1-1/\sqrtκ).
A new biased gradient descent method for conditional stochastic optimization.
problem Challenges in constructing unbiased gradient estimators for conditional stochastic optimization.
method Proposes a biased stochastic gradient descent (BSGD) algorithm and analyzes its sample complexities.
result Establishes sample complexities of BSGD for various objectives and shows that BSpiderBoost matches the lower bound complexity.
A new method for faster optimization of machine learning problems.
problem Minimization of composition of expected value functions.
method C-SAG, a novel extension of SAG for FS-CEVF problems.
result C-SAG achieves lower oracle query complexity per iteration than C-SVRG and converges faster.
Many classical algorithms are found until several years later to outlive the confines in which they were conceived, and continue to be relevant in unforeseen settings. In this paper, we show that SVRG is one such method: being originally designed for strongly convex objectives, it is also very robust in non-strongly co…
Paper proposes iLPA for solving DC composite optimization problems, with applications to matrix completion with outliers.
problem Solving nonconvex and nonsmooth DC composite optimization problems.
method Inexact linearized proximal algorithm (iLPA) for DC composite optimization problems.
result The iLPA achieves local R-linear convergence rate under the Kurdyka-Łöjasiewicz property.
This paper advances FL algorithms for composite optimization and statistical recovery.
problem Federated learning optimization and statistical recovery in composite settings.
method Proposes Fast Federated Dual Averaging for strongly convex and smooth loss, and Multi-stage Federated Dual Averaging for restricted strongly convex and smooth loss.
result Establishes state-of-the-art iteration and communication complexity, and high probability complexity bound with linear speedup.
Majorization-minimization algorithms consist of successively minimizing a sequence of upper bounds of the objective function. These upper bounds are tight at the current estimate, and each iteration monotonically drives the objective function downhill. Such a simple principle is widely applicable and has been very popu…
Analysis of a stochastic system showing convergence to an averaged model with Gaussian deviations.
problem Convergence analysis of a perturbed compositional gradient flow system.
method Separation of scales and averaging principle applied to stochastic differential equations.
result The slow motion of the system can be approximated by a standard perturbed gradient flow or SCGD algorithm.
We consider in this paper a class of composite optimization problems whose objective function is given by the summation of a general smooth and nonsmooth component, together with a relatively simple nonsmooth term. We present a new class of first-order methods, namely the gradient sliding algorithms, which can skip the…
New algorithm samples efficiently from complex composite potentials.
problem Sampling from densities with smooth and non-smooth components.
method Metropolis-Hastings framework with proximal-based proposal.
result Mixes to target density in O ( d log ( d / ε ) ) O(d \log (d/\varepsilon)) O ( d log ( d / ε )) iterations. Two accelerated methods for linearly constrained convex programming are proposed, improving convergence rates.
problem Efficiently solving structured linearly constrained convex programming problems.
method Two accelerated methods: LALM and LADMM, for composite convex objectives.
result Accelerated methods achieve faster convergence rates compared to non-accelerated methods.
Improved subgradient method tackles ill-conditioned composite optimization problems.
problem Slow convergence of subgradient method for composite optimization problems.
method Preconditioned subgradient method with Levenberg-Marquardt approach.
result Linear convergence rate for composite optimization problems under mild conditions.
Improved quasi-Newton method for convex optimization with linear and accelerated convergence.
problem Efficiently solving composite optimization problems with strong convexity.
method Proximal quasi-Newton algorithm with accelerated variant.
result Acceleration may not improve convergence in quasi-Newton setting.
New algorithm reduces complexity for optimizing complex machine learning tasks.
problem Optimizing complex machine learning objectives like reinforcement learning and portfolio management.
method Developed SARAH-Compositional algorithm using Stochastic Recursive Gradient Descent.
result Achieved optimal IFO complexity bounds for stochastic compositional optimization.
Continuous-time SGD converges to optimal parameters via CLT.
problem Learning continuous-time models efficiently.
method Stochastic gradient descent in continuous time (SGDCT).
result Proves a central limit theorem for SGDCT's convergence.
New algorithm reduces error in regression problems.
problem Minimizing composite objective functions with quadratic and convex components.
method Stochastic dual averaging with constant step-size, proving convergence rate O(1/n).
result Extends least-squares regression to various convex regularizers and geometries.
Improved online learning with time-varying constraints for complex domains.
problem Constrained online convex optimization with time-varying constraints.
method Constructing a composite surrogate loss and using the online Frank-Wolfe method.
result Novel regret and cumulative constraint violation bounds for strongly convex losses.
AdaGrad fails to adapt to Hölder-smoothness in composite optimization problems.
problem AdaGrad's convergence rate is suboptimal for composite objectives.
method Exhibited a simple one-dimensional convex problem to highlight AdaGrad's limitations.
result AdaGrad does not achieve the classical convergence rate for Hölder-smooth objectives.
A new method for distributed optimization reduces communication rounds without minibatches.
problem Efficient training in distributed machine learning with different data distributions.
method A primal-dual method (GA-MSGD) applied to the Lagrangian of distributed optimization.
result Achieves linear convergence in communication rounds for strongly convex objectives.
New method improves mini-batch efficiency for convex optimization problems.
problem Efficiently solving convex regularized empirical risk minimization with mini-batches.
method Doubly Accelerated Stochastic Variance Reduced Dual Averaging Method with double acceleration and variance reduction.
result Significantly reduces the number of mini-batches needed for optimal performance.
New algorithms solve large-scale low-rank and nonsmooth optimization problems efficiently.
problem Solving large-scale composite convex optimization problems with nonsmooth and low-rank terms.
method Stochastic optimization algorithms combining variance reduction and weak proximal oracle.
result First algorithm with nearly optimal sample complexity, single low-rank SVD per iteration, and log 1 / ε \log{1/ε} log 1/ ε thin-SVD computations. In this paper we develop proximal methods for statistical learning. Proximal point algorithms are useful in statistics and machine learning for obtaining optimization solutions for composite functions. Our approach exploits closed-form solutions of proximal operators and envelope representations based on the Moreau, Fo…
In this paper we develop a randomized block-coordinate descent method for minimizing the sum of a smooth and a simple nonsmooth block-separable convex function and prove that it obtains an ε ε ε -accurate solution with probability at least 1 − ρ 1-ρ 1 − ρ in at most O ( n ε log 1 ρ ) O(\tfrac{n}ε \log \tfrac{1}ρ) O ( ε n log ρ 1 ) iterations, where n n n is the numbe…
New algorithm tackles risk-aware learning problems efficiently.
problem Risk-aware learning with mean-semideviation objective.
method Zeroth-order compositional stochastic optimization algorithm.
result Algorithm converges to optimal solutions with explicit rates.
Optimizes average of convex functions with tight bounds.
problem Minimizing the average of m convex functions with gradient and prox oracles.
method Tight upper and lower bounds on complexity for deterministic and randomized optimization.
result Optimal methods for smooth and non-smooth functions, showing significant gap between deterministic and randomized settings.
Estimates input from output of nonlinear systems using ANN.
problem Estimating unknown compositional input from system output.
method Artificial Neural Networks (ANNs) for nonlinear system inversion.
result ANNs can compete with optimal bounds for linear systems and demonstrate promising results for nonlinear systems.
The paper shows how to simplify complex optimization problems into simpler ones.
problem Complex multiobjective optimization problems.
method Proving strongly convex problems are simplicial under certain conditions and demonstrating transformations.
result Strongly convex problems can be simplified into simpler ones via generic linear perturbations.
Unified analysis of matching pursuit and coordinate descent methods.
problem Optimization of linear spaces using first-order methods.
method Unified analysis of matching pursuit and coordinate descent, providing rates for smooth and strongly convex objectives.
result Unified analysis leading to tightest known rates for steepest coordinate descent and accelerated convergence for matching pursuit.
Boosting is a popular way to derive powerful learners from simpler hypothesis classes. Following previous work (Mason et al., 1999; Friedman, 2000) on general boosting frameworks, we analyze gradient-based descent algorithms for boosting with respect to any convex objective and introduce a new measure of weak learner p…
Algorithm samples from composite log-concave distributions using gradient evaluations and restricted Gaussian oracles.
problem Sampling from composite log-concave distributions with limited gradient evaluations.
method Proximal gradient algorithm with RGO for g g g and strong/strongly convex conditions for f f f . result Achieves ε ε ε error in total variation distance in O ~ ( κ d log 4 ( 1 / ε ) ) \widetilde{\mathcal O}(κ\sqrt d \log^4(1/ε)) O ( κ d log 4 ( 1/ ε )) iterations. Alternative proof for convergence rate of three operator splitting scheme.
problem Optimizing composite functions with specific operator properties.
method Three operator splitting scheme for optimizing composite functions.
result Sublinear rate of convergence proved for the method.
Optimizes convergence rate of stochastic proximal algorithms for composite convex problems.
problem Solving composite convex optimization problems with composite regularizers.
method Analyzed proximal stochastic gradient method and randomized incremental proximal method under relaxed variance assumptions.
result Proves O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) convergence rate for last iterate of both algorithms under componentwise convexity and smoothness. New methods reduce variance in stochastic dual averaging for sparse solutions.
problem Regularized empirical risk minimization problems in machine learning.
method Stochastic dual averaging with variance reduction for sparser solutions.
result Achieve best known convergence rates for both strongly and non-strongly convex regularizers.