Quantized Stochastic Primal-Dual Methods for Distributed Optimization
problem Distributed optimization with stochastic gradients and finite-bit communication
method q-PDGD, a quantized stochastic primal-dual method
result Linear contraction to an explicit neighborhood under RSI, O(1/k) convergence under PL inequality
New methods reduce variance in stochastic dual averaging for sparse solutions.
problem Regularized empirical risk minimization problems in machine learning.
method Stochastic dual averaging with variance reduction for sparser solutions.
result Achieve best known convergence rates for both strongly and non-strongly convex regularizers.
Optimizes stochastic convex optimization with local minimax theory and adaptive methods.
problem Optimizing stochastic convex optimization problems with local complexity measures.
method Local minimax theory, adaptive fully online methods, Nesterov's dual averaging, Riemannian stochastic gradient methods.
result Achieves optimal convergence guarantees for stochastic convex optimization problems.
NESTT tackles nonconvex optimization problems in a distributed and stochastic manner.
problem Nonconvex optimization problems with a sum of nonconvex functions and a nonsmooth regularizer.
method NESTT algorithm that splits the problem into N subproblems and uses an augmented Lagrangian based primal-dual scheme.
result NESTT achieves ε-stationary solution using O((\sum_{i=1}^N\sqrt{L_i/N})^2/ε) gradient evaluations, up to N times better than gradient descent methods.
New algorithms solve convex-concave problems faster than previous methods.
problem Solving min-max problems without bilinear structure.
method Stochastic primal-dual algorithms with logarithmic dual updates.
result Faster convergence rates than O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) for certain problems. Stochastic Gradient Descent (SGD) has become popular for solving large scale supervised machine learning optimization problems such as SVM, due to their strong theoretical guarantees. While the closely related Dual Coordinate Ascent (DCA) method has been implemented in various software packages, it has so far lacked go…
Stochastic dual coordinate ascent (SDCA) is an effective technique for solving regularized loss minimization problems in machine learning. This paper considers an extension of SDCA under the mini-batch setting that is often used in practice. Our main contribution is to introduce an accelerated mini-batch version of SDC…
A new method for distributed optimization reduces communication rounds without minibatches.
problem Efficient training in distributed machine learning with different data distributions.
method A primal-dual method (GA-MSGD) applied to the Lagrangian of distributed optimization.
result Achieves linear convergence in communication rounds for strongly convex objectives.
A fast method for training linear classifiers maximizes margins.
problem Training linear classifiers with maximum margins.
method Momentum-based gradient method derived from convex dual with Nesterov acceleration.
result Exponentially faster convergence rate compared to standard methods.
Dual IHT algorithm solves NP-hard non-convex sparse minimization problems.
problem Non-convex sparse minimization with ℓ 2 \ell_2 ℓ 2 -regularized loss function. method Developed a dual IHT algorithm for maximizing the non-smooth dual objective.
result Sparse recovery performance is invariant to RIP, superior to primal IHT algorithms.
Uniform sampling of training data has been commonly used in traditional stochastic optimization algorithms such as Proximal Stochastic Gradient Descent (prox-SGD) and Proximal Stochastic Dual Coordinate Ascent (prox-SDCA). Although uniform sampling can guarantee that the sampled stochastic quantity is an unbiased estim…
Gradient method achieves linear convergence for saddle point problems without strong convexity.
problem Solving saddle point problems with non-strongly convex functions.
method Primal-dual gradient method with a novel analysis technique.
result Linear convergence achieved without strong convexity of f f f . The paper analyzes momentum variants of various stochastic optimization methods.
problem Improving the convergence rates of stochastic optimization methods.
method Stochastic gradient descent, Newton, proximal point, and subspace ascent methods with momentum.
result Global linear convergence rates for various measures of success.
Improves SGD for convex functions with mini-batches, proving linear convergence.
problem Minimizing convex functions with constraints.
method Projected semi-stochastic gradient descent with mini-batches.
result Linear convergence under weak strong convexity assumption.
Proposes a new stochastic optimization method for MLR models.
problem Slow convergence of SGD in big data scenarios.
method Dual Stochastic Natural Gradient Descent (DNSGD) based on manifold optimization.
result DNSGD converges and has linear computational complexity.
New method improves optimization algorithms without Lipschitz smoothness.
problem Improving optimization algorithms in the absence of Lipschitz smoothness.
method Dual kernel conditioning (DKC) to provide dual Lipschitz continuity.
result First complexity bounds and iterate convergence for random reshuffling mirror descent.
Paper tackles robust model training with a new stochastic algorithm.
problem Training robust models against data distribution shift.
method Derives a novel dual formulation and proposes a nested stochastic gradient descent algorithm.
result Establishes polynomial iteration and sample complexities for large-scale DRO problems.
Stochastic gradient descent improves Gaussian process regression.
problem Efficiently solving large linear systems in Gaussian process regression.
method Developed a stochastic dual descent algorithm using insights from optimisation and kernel communities.
result Stochastic gradient descent is highly effective when done right.
Two algorithms solve nonconvex minimax problems with linear constraints, achieving complexity guarantees.
problem Nonconvex minimax problems with coupled linear constraints.
method Zeroth-order primal-dual alternating projected gradient (ZO-PDAPG) and zeroth-order regularized momentum primal-dual projected gradient (ZO-RMPDPG) algorithms.
result Iteration complexity guarantees for solving nonconvex-(strongly) concave minimax problems with coupled linear constraints.
A new method uses deep learning for optimal stopping problems.
problem Solving optimal stopping problems in financial mathematics.
method Deep primal-dual BSDE framework with a novel loss function.
result The method provides a true upper bound for the optimal value.
Study shows convergence of stochastic gradient method for unregularized Wasserstein optimization.
problem Wasserstein distributionally robust optimization under potential distribution shifts.
method Regularized approximation with stochastic gradient methods, convergence analysis.
result Stochastic gradient method converges to subgradients of unregularized objective as regularization vanishes.
New scalable methods for unbalanced optimal transport improve efficiency and applicability.
problem Scalable algorithms for unbalanced optimal transport remain underexplored.
method Analysis of semi-dual formulation and adaptive gradient methods.
result SGD methods achieve a convergence rate of O(n/εT) for large-scale applications.
Improves SDCA convergence for convex objectives with linear constraints.
problem Minimizing convex objectives with linear constraints under gradient-Lipschitz assumption failure.
method Shifted Stochastic Dual Coordinate Ascent (SDCA) under smoothness assumption.
result Obtains linear convergence rate for Poisson regression and Hawkes process objectives.
ProxQuant improves quantized neural networks using proximal operators.
problem Making neural networks work on devices with limited resources.
method Formulates quantized network training as a regularized learning problem and optimizes it via the prox-gradient method.
result ProxQuant outperforms state-of-the-art results on binary quantization and is on par with state-of-the-art on multi-bit quantization.
Develops an online method for solving constrained optimization problems with debiasing techniques.
problem Online inference of solutions to constrained optimization problems with equality and inequality constraints.
method Stochastic Sequential Quadratic Programming (SSQP) with momentum debiasing.
result Achieves global almost-sure convergence and local asymptotic normality with optimal primal-dual limiting covariance.
New algorithm approximates continuous Wasserstein barycenters efficiently.
problem Computing Wasserstein barycenters for continuous distributions.
method Stochastic algorithm using dual potentials and stochastic gradient descent.
result Efficient online approximation of continuous Wasserstein barycenters.
Paper solves investment and consumption problem with utility duality.
problem Continuous-time consumption and investment problem with incomplete markets and stochastic differential utility.
method Introduces a dual problem to study the problem, establishing duality for Epstein-Zin utility and identifying optimal strategies.
result Optimal strategies identified without technical conditions, with dual minimizer interpreted as least favorable market completion.
Develops an online dual coordinate ascent algorithm for streaming data.
problem Lack of scalability and online adaptability in existing dual coordinate ascent algorithms.
method Introduces an online dual coordinate ascent (O-DCA) algorithm that can handle streaming data without revisiting past data.
result The O-DCA algorithm enables continuous adaptation and learning for online learning scenarios.
DSPI connects natural policy gradient to policy iteration, proving global convergence.
problem Optimizing policies in reinforcement learning.
method DSPI framework, combining smoothed policy iteration and natural policy gradient.
result DSPI achieves geometric convergence and optimal complexity for policy optimization.
Polyak-Łojasiewicz inequality simplifies linear convergence proofs.
problem Proving linear convergence without strong convexity.
method Using Polyak-Łojasiewicz inequality to analyze various optimization methods.
result New analyses and proofs of linear convergence for multiple machine learning problems.
The paper presents methods to improve policy evaluation in reinforcement learning.
problem Policy evaluation in reinforcement learning with linear function approximation.
method Transformed into a convex-concave saddle point problem, primal-dual batch gradient method, and two stochastic variance reduction methods.
result Achieved linear convergence even with only strong concavity in dual variables.
New method speeds up optimization with non-uniform sampling.
problem Optimizing complex stochastic problems efficiently.
method Stochastic Primal Dual Coordinate Method with Optimality Violation-based Sampling.
result The proposed method and variants outperform other methods in speed.
New method improves mini-batch efficiency for convex optimization problems.
problem Efficiently solving convex regularized empirical risk minimization with mini-batches.
method Doubly Accelerated Stochastic Variance Reduced Dual Averaging Method with double acceleration and variance reduction.
result Significantly reduces the number of mini-batches needed for optimal performance.
First-order method solves stochastic bilevel optimization with linear constraints.
problem Stochastic bilevel optimization with linear constraints and noise.
method Developed a novel framework using gradient-based techniques and smoothed penalty functions.
result Achieved finite-time convergence guarantees for ( δ , ε ) (δ, ε) ( δ , ε ) -Goldstein stationary points. MDA optimizer performs similarly to SGD+M in CV and Adam in NLP.
problem Performance degradation due to choosing the wrong optimizer.
method Modernized Dual Averaging (MDA) optimizer, inspired by dual averaging.
result MDA performs as well as SGD+M in CV and as Adam in NLP.
New PDHG approach tackles high-cost stochastic minimization with linear composite terms.
problem High cost and lack of closed-form proximal mapping for composite regularization terms.
method Stochastic PDHG with data point sampling, high-probability iteration complexity analysis.
result High-probability convergence analysis supports practical performance.
New algorithm reduces complexity for distributed and stochastic optimization.
problem Optimization of distributed and stochastic problems with multiple agents.
method Random Gradient Extrapolation Method (RGEM)
result Achieves optimal complexity bounds for both distributed and stochastic optimization.
TiAda adapts adaptive gradient methods for nonconvex minimax optimization.
problem Nonconvex minimax optimization challenges in achieving convergence.
method TiAda is a time-scale adaptive GDA algorithm for nonconvex minimax optimization.
result TiAda achieves near-optimal complexities in deterministic and stochastic settings.
New algorithm optimizes stochastic optimization with circular dependency.
problem Circular dependency between decision variable and importance sampling.
method Single-loop stochastic approximation algorithm based on Nesterov's dual averaging.
result Achieves minimal asymptotic variance and resolves circular optimization challenge.
New method uses continuous OT for fairness, outperforming discrete OT.
problem Fairness issues in machine learning models.
method Stochastic-gradient fairness method based on continuous optimal transport.
result Continuous OT method outperforms discrete OT when data is limited.
Optimizes power allocation for WDM in RoFSO systems.
problem Maximizing total capacity with power and eye safety constraints.
method Model-based Stochastic Dual Gradient algorithm and model-free Primal-Dual Deep Learning algorithm.
result Deep Learning algorithm outperforms average equal power allocation.
Optimal privacy-preserving algorithm for solving saddle point problems.
problem Solving convex-concave stochastic saddle point problems under differential privacy constraints.
method Recursive regularization technique repurposed for saddle point problems, achieving strong gap rate of O(1/√n + √d/nε).
result Achieves nearly optimal strong gap rate of O(1/√n + √d/nε) with gradient complexity O(min{n^2ε^(1.5)/√d, n^(3/2)}).
We consider stochastic strongly convex optimization with a complex inequality constraint. This complex inequality constraint may lead to computationally expensive projections in algorithmic iterations of the stochastic gradient descent~(SGD) methods. To reduce the computation costs pertaining to the projections, we pro…
This paper shows how to use ℓ 1 \ell_1 ℓ 1 regularization effectively in training sparse CNNs.
problem Why ℓ 1 \ell_1 ℓ 1 regularization hasn't been used in sparse deep learning models like CNNs. method Demonstrated that SGD is not suitable for ℓ 1 \ell_1 ℓ 1 regularization and replaced it with a new training algorithm based on regularized dual averaging (RDA). result Achieved state-of-the-art sparsity for CNNs using RDA with ℓ 1 \ell_1 ℓ 1 regularization, achieving 95% sparsity for ResNet18 on CIFAR-10. This dissertation advances the theoretical foundation of local optimization methods in Federated Learning.
problem Theoretical understanding of local optimization methods in Federated Learning is lacking.
method The dissertation proposes and analyzes new methods to improve convergence rates and communication efficiency in Federated Learning.
result Sharp bounds and convergence rates for FedAvg are established, and new methods like FedAc and Federated Dual Averaging are proposed.
Given a convex optimization problem and its dual, there are many possible first-order algorithms. In this paper, we show the equivalence between mirror descent algorithms and algorithms generalizing the conditional gradient method. This is done through convex duality, and implies notably that for certain problems, such…
Study efficient convergence of RL algorithm with function approximation.
problem Convergence of actor-critic algorithm with nonlinear function approximation.
method Stochastic gradient descent ascent with adaptive proximal term, Polyak-Łojasiewicz condition.
result First efficient convergence result with rate of O(sqrt{ln(N d G^2) / N}).
Paper introduces a new reinforcement learning method with improved performance.
problem Designing and analyzing efficient reinforcement learning algorithms.
method Proximal gradient temporal difference learning (GTD) with accelerated algorithm GTD2-MP.
result GTD algorithms have linear complexity and improved convergence rate.