We consider in this paper a class of composite optimization problems whose objective function is given by the summation of a general smooth and nonsmooth component, together with a relatively simple nonsmooth term. We present a new class of first-order methods, namely the gradient sliding algorithms, which can skip the…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “composite nonsmooth optimization”
Optimizes convergence rate of stochastic proximal algorithms for composite convex problems.
problem Solving composite convex optimization problems with composite regularizers.
method Analyzed proximal stochastic gradient method and randomized incremental proximal method under relaxed variance assumptions.
result Proves convergence rate for last iterate of both algorithms under componentwise convexity and smoothness.
SARAH and SPIDER are two recently developed stochastic variance-reduced algorithms, and SPIDER has been shown to achieve a near-optimal first-order oracle complexity in smooth nonconvex optimization. However, SPIDER uses an accuracy-dependent stepsize that slows down the convergence in practice, and cannot handle objec…
An inexact LPA for DC composite optimization and application to matrix completions with outliersmath.OC
Paper proposes iLPA for solving DC composite optimization problems, with applications to matrix completion with outliers.
problem Solving nonconvex and nonsmooth DC composite optimization problems.
method Inexact linearized proximal algorithm (iLPA) for DC composite optimization problems.
result The iLPA achieves local R-linear convergence rate under the Kurdyka-Łöjasiewicz property.
Consider the stochastic composition optimization problem where the objective is a composition of two expected-value functions. We propose a new stochastic first-order method, namely the accelerated stochastic compositional proximal gradient (ASC-PG) method, which updates based on queries to the sampling oracle using tw…
This work uses Lasry-Lions envelopes to solve nonconvex optimization problems.
problem Nonconvex and nonsmooth terms in optimization problems.
method Develops a homotopy approach using Lasry-Lions envelopes to approximate and solve the original problem.
result The method can solve composite minimization problems and is more effective than classical alternatives in certain domains.
Cyclic coordinate descent identifies models in finite time and converges linearly.
problem Model identification in composite nonsmooth optimization problems.
method Cyclic coordinate descent for a wide class of functions.
result Explicit local linear convergence rates for coordinate descent.
Analysis of Generalized Bregman Surrogate Algorithms for Nonsmooth Nonconvex Statistical Learningmath.OC
Paper develops algorithms for nonsmooth, nonconvex statistical learning problems.
problem Nonsmooth and nonconvex objectives in statistical learning.
method Bregman-surrogate algorithm framework, including local linear approximation, mirror descent, iterative thresholding, DC programming.
result Global convergence rates for nonconvex and nonsmooth objectives in high dimensions.
Stochastic Proximal Gradient Algorithm with Minibatches. Application to Large Scale Learning Modelscs.LG
Develops minibatch stochastic proximal gradient for large-scale learning models.
problem Finding optimal predictors with complex regularizers in large-scale learning models.
method Minibatch variants of stochastic proximal gradient algorithm for composite objective functions.
result Minibatch size after iterations achieves suboptimality in expected quadratic distance.
We propose an adaptive smoothing algorithm based on Nesterov's smoothing technique in \cite{Nesterov2005c} for solving "fully" nonsmooth composite convex optimization problems. Our method combines both Nesterov's accelerated proximal gradient scheme and a new homotopy strategy for smoothness parameter. By an appropriat…
Composite convex optimization problems which include both a nonsmooth term and a low-rank promoting term have important applications in machine learning and signal processing, such as when one wishes to recover an unknown matrix that is simultaneously low-rank and sparse. However, such problems are highly challenging t…
New Langevin Monte Carlo algorithms for sampling from nonsmooth distributions.
problem Sampling from distributions with nonsmooth convex composite potentials.
method Leveraging Bregman--Moreau envelopes and proximal operators in mirror descent.
result Efficiency in sampling from nonsmooth distributions, extending existing methods.
This work establishes uniform convergence of subdifferentials in stochastic optimization.
problem Understanding how empirical stationary points approximate population ones in nonsmooth, nonconvex stochastic optimization.
method Reduction principle for weakly convex stochastic objectives, focusing on subgradient convergence.
result Sharp uniform convergence rates for subdifferential mappings in stochastic convex-composite optimization.
Due to their simplicity and excellent performance, parallel asynchronous variants of stochastic gradient descent have become popular methods to solve a wide range of large-scale optimization problems on multi-core architectures. Yet, despite their practical success, support for nonsmooth objectives is still lacking, ma…
In this paper, we extend the geometric descent method recently proposed by Bubeck, Lee and Singh to tackle nonsmooth and strongly convex composite problems. We prove that our proposed algorithm, dubbed geometric proximal gradient method (GeoPG), converges with a linear rate and thus achieves the optimal …
New algorithm for fast nonsmooth optimization with applications in image processing and machine learning.
problem Minimizing the sum of three convex functions with specific properties.
method PDDY algorithm, based on Davis-Yin splitting in a primal-dual product space.
result Sublinear and linear convergence rates in various scenarios, including strong convexity.
Study efficient derivative computation for nondifferentiable maps in machine learning.
problem Efficiently compute derivatives of fixed-point of nondifferentiable contractions.
method Iterative Differentiation (ITD), Approximate Implicit Differentiation (AID), and New Stochastic Implicit Differentiation (NSID).
result Established convergence rates for ITD, AID, and NSID, matching or improving smooth setting rates.
Within the unmanageably large class of nonconvex optimization, we consider the rich subclass of nonsmooth problems that have composite objectives---this already includes the extensively studied convex, composite objective problems as a special case. For this subclass, we introduce a powerful, new framework that permits…
In this paper, a novel stochastic extra-step quasi-Newton method is developed to solve a class of nonsmooth nonconvex composite optimization problems. We assume that the gradient of the smooth part of the objective function can only be approximated by stochastic oracles. The proposed method combines general stochastic …
PPGD solves nonconvex nonsmooth optimization problems without KL property.
problem Nonconvex and nonsmooth optimization problems in statistics and machine learning.
method Projective Proximal Gradient Descent (PPGD) for solving a class of nonconvex and nonsmooth problems.
result PPGD achieves a fast convergence rate of O(1/k^2) for k ≥ k_0.
Nonsmoothness in Machine Learning: specific structure, proximal identification, and applicationsmath.OC
This paper tackles nonsmooth optimization in machine learning.
problem Nonsmoothness in machine learning optimization problems.
method Identifying specific structures and leveraging them for practical applications.
result Compression, acceleration, and dimension reduction are possible with nonsmooth optimization.
Neural Network Gaussian Process Considering Input Uncertainty for Composite Structures Assemblystat.ML
Model predicts composite structures assembly quality with input uncertainty.
problem Accurate prediction of dimensional deviations and residual stress in composite structures assembly.
method Neural Network Gaussian Process considering input uncertainty.
result NNGPIU model outperforms other methods for nonsmooth, nonlinear responses.
Study on Adam-family methods for nonsmooth optimization with convergence guarantees.
problem Training nonsmooth neural networks with convergence guarantees.
method Two-timescale updating scheme and stochastic subgradient methods with gradient clipping.
result Convergence guarantees for various Adam-family methods in training nonsmooth neural networks.
New methods help escape strict saddle points in nonsmooth optimization.
problem Escaping strict saddle points in nonsmooth optimization.
method An inexact stochastically perturbed gradient method applied to the Moreau envelope.
result A variety of algorithms for nonsmooth optimization can efficiently escape strict saddle points of the Moreau envelope.
Classical stochastic gradient methods are well suited for minimizing expected-value objective functions. However, they do not apply to the minimization of a nonlinear function involving expected values or a composition of two expected-value functions, i.e., problems of the form $\min_x \mathbf{E}_v [f_v\big(\mathbf{E}_…
In this work we consider the stochastic minimization of nonsmooth convex loss functions, a central problem in machine learning. We propose a novel algorithm called Accelerated Nonsmooth Stochastic Gradient Descent (ANSGD), which exploits the structure of common nonsmooth loss functions to achieve optimal convergence ra…
Nonsmooth High-Order Averaging Theory with Application to Extremum Seeking Optimization and Controlmath.OC
New theory for nonsmooth systems helps optimize and control complex functions.
problem Optimizing and controlling systems with nonsmooth functions.
method Higher-order averaging theory with nonsmooth near-identity transformation and lexicographic differentiation.
result Closed formula for nonsmooth first and second-order averaging.
Deep learning models viewed through tame geometry for convergence guarantees.
problem Understanding convergence guarantees in deep learning models.
method Introducing tame geometry concepts and tools for nonsmooth nonconvex settings.
result Illustrates tame geometry as a natural framework for AI systems, especially deep learning.
New stability bounds for SGD on nonsmooth convex losses.
problem Understanding stability of SGD on nonsmooth convex losses.
method Sharp upper and lower bounds for SGD and full-batch GD on nonsmooth convex losses.
result SGD can be less stable but still useful for generalization bounds.
We propose an algorithmic framework for convex minimization problems of a composite function with two terms: a self-concordant function and a possibly nonsmooth regularization term. Our method is a new proximal Newton algorithm that features a local quadratic convergence rate. As a specific instance of our framework, w…
New method solves nonsmooth low-rank matrix optimization problems efficiently.
problem Nonsmooth and low-rank matrix optimization problems in statistics and machine learning.
method Low-rank Extragradient Method with warm-start initialization.
result The extragradient method converges to an optimal solution with rate and requires only two low-rank SVDs per iteration.
Optimized method tackles convex optimization with heavy-tailed noise.
problem Convex optimization problems with noisy gradients.
method Vanilla stochastic proximal subgradient method without gradient clipping or normalization.
result Achieves optimal complexity for various convex optimization types under heavy-tailed noise.
We analyze stochastic algorithms for optimizing nonconvex, nonsmooth finite-sum problems, where the nonconvex part is smooth and the nonsmooth part is convex. Surprisingly, unlike the smooth case, our knowledge of this fundamental problem is very limited. For example, it is not known whether the proximal stochastic gra…
We study a hybrid conditional gradient - smoothing algorithm (HCGS) for solving composite convex optimization problems which contain several terms over a bounded set. Examples of these include regularization problems with several norms as penalties and a norm constraint. HCGS extends conditional gradient methods to cas…
Alternating direction method of multipliers (ADMM) is a popular optimization tool for the composite and constrained problems in machine learning. However, in many machine learning problems such as black-box attacks and bandit feedback, ADMM could fail because the explicit gradients of these problems are difficult or in…
Unified Lagrangian-based methods for nonsmooth nonconvex optimization.
problem Minimizing nonsmooth nonconvex functions with constraints.
method Developed a unified framework for Lagrangian-based methods using subgradient updates.
result Global convergence guarantees for the proposed framework under mild conditions.
The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimizationcs.LG
Nesterov's extrapolation improves convergence in nonsmooth optimization.
problem Improving convergence rate in nonsmooth convex optimization.
method Nesterov's extrapolation applied to projected subgradient methods.
result Nesterov's extrapolation optimizes individual convergence for nonsmooth problems.
Novel coordinate descent (CD) methods are proposed for minimizing nonconvex functions consisting of three terms: (i) a continuously differentiable term, (ii) a simple convex term, and (iii) a concave and continuous term. First, by extending randomized CD to nonsmooth nonconvex settings, we develop a coordinate subgradi…
Improved Last-Iterate Convergence of Shuffling Gradient Methods for Nonsmooth Convex Optimizationmath.OC
Improved shuffling gradient methods converge faster for nonsmooth convex optimization.
problem Improving convergence rates for nonsmooth convex optimization problems.
method Analysis of shuffling gradient methods, focusing on Random Reshuffle and Single Shuffle strategies.
result Shuffling gradient methods, particularly Random Reshuffle and Single Shuffle, converge faster than Proximal Gradient Descent for nonsmooth convex optimization.
Adam's convergence rate in nonsmooth nonconvex optimization is analyzed.
problem Understanding Adam's convergence behavior in nonsmooth nonconvex optimization.
method Finite-time analysis of Adam's classical form.
result Proven convergence rate of for nonsmooth nonconvex optimization.
Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifoldmath.OC
Two new methods solve nonsmooth optimization on Riemannian Stiefel manifold.
problem Optimization over nonsmooth, non-differentiable functions on Riemannian manifolds.
method R-ProxSGD and R-ProxSPB, generalizing proximal SGD and SpiderBoost.
result R-ProxSPB finds ε-stationary points with IFO complexity of Ø(ε^(-3)) in online and Ø(n + √nε^(-2)) in finite-sum cases.
Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimizationmath.OC
The paper guarantees global stability for stochastic subgradient methods in nonsmooth nonconvex optimization.
problem Minimizing nonsmooth nonconvex functions with convergence guarantees.
method Developed a framework for stochastic subgradient methods with global stability guarantees.
result Iterates are uniformly bounded and asymptotically stabilize around the stable set of the differential inclusion.
We consider a class of nonconvex nonsmooth optimization problems whose objective is the sum of a smooth function and a finite number of nonnegative proper closed possibly nonsmooth functions (whose proximal mappings are easy to compute), some of which are further composed with linear maps. This kind of problems arises …
New adaptive methods solve weakly convex stochastic optimization problems.
problem Solving weakly convex stochastic optimization problems.
method Adaptive first and zeroth-order methods using exponential moving averages.
result Established non-asymptotic convergence rates for nonsmooth and nonconvex problems.
We generalize Newton-type methods for minimizing smooth functions to handle a sum of two convex functions: a smooth function and a nonsmooth function with a simple proximal mapping. We show that the resulting proximal Newton-type methods inherit the desirable convergence behavior of Newton-type methods for minimizing s…
We present a stochastic setting for optimization problems with nonsmooth convex separable objective functions over linear equality constraints. To solve such problems, we propose a stochastic Alternating Direction Method of Multipliers (ADMM) algorithm. Our algorithm applies to a more general class of nonsmooth convex …
The paper relaxes assumptions for analyzing stochastic optimization algorithms.
problem Analyzing the convergence of stochastic gradient algorithms under weaker variance assumptions.
method Building on and extending a connection to the Halpern iteration, the paper analyzes algorithms for convex nonsmooth optimization and min-max problems.
result Rates for optimality measures are obtained without requiring boundedness of the feasible set for problems beyond simple constrained optimization.
Asynchronous Delay-Aware Accelerated Proximal Coordinate Descent for Nonconvex Nonsmooth Problemsmath.OC
Nonconvex and nonsmooth problems have recently attracted considerable attention in machine learning. However, developing efficient methods for the nonconvex and nonsmooth optimization problems with certain performance guarantee remains a challenge. Proximal coordinate descent (PCD) has been widely used for solving opti…