Subgradient algorithm achieves optimal regret for both adversarial and i.i.d. costs on the simplex.
problem Achieving optimal regret for both adversarial and i.i.d. costs on the simplex.
method Demonstrates the universality of the Subgradient algorithm for online learning on the simplex.
result Shows simultaneous O ( N ) O(\sqrt N) O ( N ) regret for adversarial costs and O ( 1 ) O(1) O ( 1 ) pseudo-regret for i.i.d. costs. The paper guarantees global stability for stochastic subgradient methods in nonsmooth nonconvex optimization.
problem Minimizing nonsmooth nonconvex functions with convergence guarantees.
method Developed a framework for stochastic subgradient methods with global stability guarantees.
result Iterates are uniformly bounded and asymptotically stabilize around the stable set of the differential inclusion.
A distributed subgradient method tackles non-convex optimization problems in networks.
problem Solving non-convex optimization problems in distributed networks.
method Proposes a distributed stochastic subgradient method (stoDPSM) with theoretical guarantees.
result Global convergence of stoDPSM using Moreau envelope stationarity measure, and linear convergence under sharpness condition.
Improved subgradient method tackles ill-conditioned composite optimization problems.
problem Slow convergence of subgradient method for composite optimization problems.
method Preconditioned subgradient method with Levenberg-Marquardt approach.
result Linear convergence rate for composite optimization problems under mild conditions.
New method bounds stochastic subgradient methods with heavy-tailed noise.
problem Bounding stochastic subgradient methods under heavy-tailed noise.
method Clipped version of projected stochastic subgradient method.
result Near optimal any-time and finite horizon bounds for averaging schemes.
Proof of convergence for multi-objective optimization using inverse reinforcement learning.
problem Proving convergence in multi-objective optimization problems.
method Wasserstein inverse reinforcement learning with projective subgradient method and gradient descent.
result Convergence of inverse reinforcement learning for multi-objective optimization.
Develops methods for solving convex optimization problems with improved accuracy and convergence.
problem Solving stochastic convex optimization problems with improved accuracy and robustness.
method Approximate-proximal point (aProx) family, including stochastic subgradient, proximal point, and bundle methods.
result Improved models converge with probability 1 and enjoy optimal asymptotic normality results under weak assumptions.
New Max-Plus neural network exploits subgradient sparsity for efficient training.
problem Training Max-Plus neural networks is challenging due to dense subgradients.
method Proposes a sparse subgradient algorithm tailored to Max-Plus models.
result Achieves more efficient updates while retaining theoretical guarantees.
Unified Lagrangian-based methods for nonsmooth nonconvex optimization.
problem Minimizing nonsmooth nonconvex functions with constraints.
method Developed a unified framework for Lagrangian-based methods using subgradient updates.
result Global convergence guarantees for the proposed framework under mild conditions.
Study proves convergence of subgradients for optimal transport-based objectives.
problem Ensuring statistical consistency and optimization stability in transport-based models.
method Proves graphical convergence of subdifferentials to the subdifferential of the population objective.
result Standard subgradient methods consistently approach stationary points of the population-level problem.
New algorithms achieve high-probability parameter-free regret in online convex optimization with heavy-tailed data.
problem Achieving high-probability parameter-free regret in online convex optimization with heavy-tailed data.
method Developed new regularization techniques to handle exponentially large iterates and heavy-tailed subgradients.
result Achieved regret bound of O ( ∥ u ∥ T 1 / p log ( 1 / δ ) ) O(\| \mathbf{u} \| T^{1/\mathfrak{p}} \log (1/δ)) O ( ∥ u ∥ T 1/ p log ( 1/ δ )) with high probability for subgradients with bounded p t h p^{th} p t h moments. Study on portfolio optimization and risk analysis, proving non-uniqueness and suggesting a method to resolve it.
problem Non-uniqueness in solution of portfolio optimization and risk analysis problems.
method Proof of non-uniqueness, introduction of Stainer point as a unique subgradient.
result Identification of a unique 'special' subgradient to resolve non-uniqueness in portfolio optimization and risk analysis.
The paper accelerates ISTA and FISTA algorithms for composite optimization problems.
problem Improving convergence rates of ISTA and FISTA for composite optimization.
method Improved proximal subgradient norm minimization using Lyapunov function.
result Convergence rates of ISTA and FISTA are accelerated.
We describe novel subgradient methods for a broad class of matrix optimization problems involving nuclear norm regularization. Unlike existing approaches, our method executes very cheap iterations by combining low-rank stochastic subgradients with efficient incremental SVD updates, made possible by highly optimized and…
Boosting algorithms in linear regression are analyzed using subgradient optimization.
problem Improving linear regression models through boosting algorithms.
method Subgradient optimization and modern first-order methods in convex optimization.
result Boosting algorithms can be interpreted as subgradient descent to minimize a specific loss function.
New theory accelerates stochastic optimization by leveraging local growth rate.
problem First-order stochastic convex optimization convergence rate.
method Developed two accelerated stochastic subgradient methods.
result Optimal iteration complexity of O ( 1 / ε 2 ( 1 − θ ) ) O(1/ε^{2(1-θ)}) O ( 1/ ε 2 ( 1 − θ ) ) for achieving ε ε ε -optimal solution. A new GAN training method using primal-dual subgradient methods.
problem Training GANs to avoid mode collapse and generate diverse samples.
method Relating GANs to convex optimization via Lagrangian perspective and primal-dual subgradient methods.
result The proposed method resolves mode collapse and generates diverse samples.
Parameter-free online convex optimization with sub-exponential noise achieves optimal regret.
problem Online convex optimization with sub-exponential noise, especially when subgradients are unbounded.
method Designing a novel parameter-free algorithm BANCO via a reduction to betting on noisy coins.
result BANCO achieves the optimal regret rate in the problem of unconstrained online convex optimization with sub-exponential noise.
Subgradient descent learns orthogonal dictionaries efficiently.
problem Sparse coding and dictionary learning for data representation.
method Subgradient descent algorithm with random initialization.
result Subgradient descent can recover orthogonal dictionaries under mild assumptions.
New algorithms solve large-scale convex regression problems.
problem Large-scale convex regression with subgradient regularization.
method Active set type algorithm on dual QP, approximate optimization, randomized augmentation.
result Solves problems with n=10^5 and d=10 in minutes.
Inexact subgradient methods work well for semialgebraic functions with additive errors.
problem Approximate gradients in machine learning and optimization.
method Inexact subgradient methods with persistent additive errors in semialgebraic functions.
result Iterates eventually fluctuate near the critical set with a proximity of O ( ε ρ ) O(ε^ρ) O ( ε ρ ) , where ε ε ε is the magnitude of subgradient evaluation errors. Paper proves equivalences in portfolio optimization with new risk measures.
problem Portfolio optimization with novel risk measures.
method Derive subgradients and gradients for negative expectile and omega ratio.
result Negative expectile can be used as a portfolio optimization objective.
New algorithms optimize spectral risk measures, improving interpolation between average and worst-case performance.
problem Optimizing spectral risk measures for learning systems.
method Developed stochastic algorithms to optimize spectral risk measures by characterizing their subdifferential and addressing challenges like biasedness of subgradient estimates and non-smoothness.
result Our approach outperforms out-of-the-box stochastic subgradient and dual averaging methods in optimizing spectral risk measures.
This work establishes uniform convergence of subdifferentials in stochastic optimization.
problem Understanding how empirical stationary points approximate population ones in nonsmooth, nonconvex stochastic optimization.
method Reduction principle for weakly convex stochastic objectives, focusing on subgradient convergence.
result Sharp uniform convergence rates for subdifferential mappings in stochastic convex-composite optimization.
We propose a randomized block-coordinate variant of the classic Frank-Wolfe algorithm for convex optimization with block-separable constraints. Despite its lower iteration cost, we show that it achieves a similar convergence rate in duality gap as the full Frank-Wolfe algorithm. We also show that, when applied to the d…
A new method solves convex optimization on curved spaces.
problem Optimization on curved spaces with non-smooth functions.
method Convex bundle method on Riemannian manifolds.
result The method converges to a minimizer under mild conditions.
RSG method reduces subgradient method's complexity for convex optimization.
problem Finding optimal solutions for convex optimization problems efficiently.
method Periodically restarts the standard subgradient method to reduce complexity.
result RSG method finds ε ε ε -optimal solutions with lower complexity than SG. Optimized method tackles convex optimization with heavy-tailed noise.
problem Convex optimization problems with noisy gradients.
method Vanilla stochastic proximal subgradient method without gradient clipping or normalization.
result Achieves optimal complexity for various convex optimization types under heavy-tailed noise.
Study robust recovery of low-rank matrices from corrupted measurements without rank prior.
problem Robust recovery of low-rank matrices from corrupted Gaussian measurements with unknown rank.
method Subgradient method with diminishing stepsizes for nonconvex nonsmooth problem.
result Subgradient method converges to exact low-rank solution at sublinear rate under RDPP condition.
Paper presents an efficient algorithm for learning minimax risk classifiers with large-scale data.
problem Efficient learning of minimax risk classifiers for large-scale data with multiple classes.
method Combination of constraint and column generation for efficient learning.
result 10x speedup for general large-scale data and 100x speedup with many classes.
Given a convex optimization problem and its dual, there are many possible first-order algorithms. In this paper, we show the equivalence between mirror descent algorithms and algorithms generalizing the conditional gradient method. This is done through convex duality, and implies notably that for certain problems, such…
New algorithms accelerate model-based optimization for stochastic problems.
problem Optimizing model-based stochastic optimization problems efficiently.
method Proposed new model-based algorithms with acceleration and minibatch techniques.
result Non-asymptotic convergence guarantees with linear speedup in minibatch size.
The paper derives subgradient estimates for a specific nonlinear subparabolic equation on pseudo-Hermitian manifolds.
problem Deriving subgradient estimates for positive solutions to a nonlinear subparabolic equation on pseudo-Hermitian manifolds.
method Using the CR sub-Laplacian comparison property, the paper derives local subgradient estimates for positive solutions to the given equation.
result The paper establishes subgradient estimates for positive solutions to the nonlinear subparabolic equation.
New method for optimization on Hadamard manifolds with curvature-independent guarantees.
problem Curvature-dependent complexity in geodesic convex optimization.
method Introducing horospherical convexity and developing algorithms for optimization.
result Curvature-independent convergence of subgradient descent and Nesterov's method.
Study on Adam-family methods for nonsmooth optimization with convergence guarantees.
problem Training nonsmooth neural networks with convergence guarantees.
method Two-timescale updating scheme and stochastic subgradient methods with gradient clipping.
result Convergence guarantees for various Adam-family methods in training nonsmooth neural networks.
Nesterov's extrapolation improves convergence in nonsmooth optimization.
problem Improving convergence rate in nonsmooth convex optimization.
method Nesterov's extrapolation applied to projected subgradient methods.
result Nesterov's extrapolation optimizes individual convergence for nonsmooth problems.
Stochastic algorithm achieves sublinear convergence for bi-objective optimization.
problem Optimizing two conflicting functions using gradient or subgradient descent.
method Stochastic alternating algorithm with varying steps for each objective.
result Achieves sublinear convergence rate of O(1/T) under strong convexity.
New algorithm improves stability and speed of adversarial training.
problem Discontinuity in solutions of inner maximization in minimax optimization.
method Epsilon-subgradient descent algorithm with K candidate solutions.
result Significant improvement in stability and convergence speed.
New algorithm samples from complex non-convex distributions.
problem Sampling from non-convex, non-smooth distributions with superlinear gradients.
method Subgradient Tamed Unadjusted Langevin Algorithm (SG-TULA)
result Non-asymptotic convergence bounds in Wasserstein-2 distance for SG-TULA.
New bounds on function optimization complexity using localized minimax analysis.
problem Optimizing stochastic convex functions and their hardest local alternatives.
method Localized minimax analysis of stochastic convex optimization, introducing computational modulus of continuity.
result Explicit bounds on the number of subgradient evaluations for optimization, demonstrating superefficiency.
Study currents from semi-convex functions, apply to Hessian measures.
problem Understanding currents from semi-convex functions.
method Analyze integer multiplicity rectifiable currents from subgradient graphs of semi-convex functions.
result Weak continuity theorem for currents with pointwise convergence.
New algorithm reduces privacy loss in SGD without learning rate tuning.
problem Locally differentially private stochastic optimization with high privacy loss.
method BANCO (Betting Algorithm for Noisy COins) for ε ε ε -LDP SGD. result Matches convergence rate of tuned SGD without learning rate tuning.
Paper presents a robust estimator for density ratio estimation that trims outliers.
problem Vulnerability of density ratio estimation to corrupted data points.
method Automatically identifies and trims outliers in density ratio estimation; uses convex formulation and subgradient descent.
result Global optimum can be obtained via subgradient descent; parameter estimation error analyzed under high-dimensional settings.
New iterative regularization method for convex loss functions.
problem Improving supervised learning with convex loss functions.
method Subgradient method-based iterative regularization without constraints.
result Finite sample bounds on excess risk achieved through stopping early.
Paper proposes fast, robust methods for low-rank matrix recovery.
problem Estimating low-rank matrices from incomplete or corrupted data.
method Scaled subgradient methods for nonsmooth, nonconvex formulations.
result Methods converge almost dimension-free and condition-number independent.
We present new algorithms to compute the mean of a set of empirical probability measures under the optimal transport metric. This mean, known as the Wasserstein barycenter, is the measure that minimizes the sum of its Wasserstein distances to each element in that set. We propose two original algorithms to compute Wasse…
Improved method reduces projection calls for nonsmooth convex optimization.
problem Optimizing nonsmooth convex functions with convex constraints.
method MOPES and MOLES methods combining Moreau-Yosida smoothing and accelerated first-order schemes.
result Achieves ε ε ε -suboptimality with significantly fewer projection calls. Efficiently samples Bayesian max-margin models for large datasets.
problem Challenges in Monte Carlo sampling for Bayesian max-margin models.
method Stochastic subgradient Hamiltonian Monte Carlo (HMC) methods.
result Effective solution for posterior inference of various Bayesian max-margin models.