New algorithms optimize non-smooth, non-convex objectives with improved complexity.
problem Optimizing non-smooth, non-convex stochastic objectives.
method Reduction to online learning, applying optimistic online learning techniques.
result Improved complexity for finding (δ,ε)-stationary points. Paper tackles private optimization for non-smooth objectives efficiently.
problem Private stochastic convex optimization for non-smooth objectives.
method Noisy mirror descent algorithm.
result Achieves optimal rates in statistical complexity and number of queries.
Averaged SGD optimizes a smoothed objective, leading to better generalization.
problem Improving generalization performance in machine learning models.
method Analyzed the smoothed objective function of SGD and proved that averaged SGD can optimize this smoothed function efficiently.
result Averaged SGD can efficiently optimize a smoothed objective, leading to better generalization.
This paper studies continuum-armed bandits under Besov smoothness conditions and derives minimax rates.
problem Optimizing an unknown function with limited evaluations.
method Studies continuum-armed bandits under Besov smoothness conditions and derives minimax rates.
result Minimax rates over Besov spaces are identical to those over the smallest Hölder space into which Besov spaces embed.
Hierarchical randomized smoothing improves model robustness for complex data.
problem Certifying robustness on complex data (e.g. images, graphs) is challenging.
method Add random noise to a randomly selected subset of entities in a hierarchical manner.
result Hierarchical randomized smoothing yields stronger robustness guarantees with high accuracy.
New perspective on Sinkhorn algorithm using stochastic mirror descent.
problem Optimal transport with unbounded domain and non-smooth objective.
method Stochastic mirror descent applied to relative smoothness.
result Sinkhorn algorithm as a special case of stochastic mirror descent.
AsylADMM improves gossip-based learning for non-smooth objectives.
problem Efficient and robust decentralized learning on edge devices.
method Asynchronous gossip algorithm for non-smooth optimization.
result AsylADMM converges faster on non-smooth problems.
We consider the stochastic composition optimization problem proposed in \cite{wang2017stochastic}, which has applications ranging from estimation to statistical and machine learning. We propose the first ADMM-based algorithm named com-SVR-ADMM, and show that com-SVR-ADMM converges linearly for strongly convex and Lipsc…
Introduces symplectic hopfoids and their relation to double Lie groupoids.
problem Understanding symplectic structures in groupoid-like objects.
method Introduces symplectic hopfoids and shows their correspondence to symplectic double groupoids.
result Symplectic hopfoids and double Lie groupoids are equivalent.
New method tackles bilevel optimization with polyhedral constraints.
problem Challenges in bilevel optimization with active-set changes and expensive Hessian inversions.
method Logarithmic barrier smoothing and proxy-gradient algorithm for differentiable approximation.
result Stationarity rates of O(K−2/3) in deterministic setting and O(K−2/5) under stochastic noise. In this paper, we consider efficient differentially private empirical risk minimization from the viewpoint of optimization algorithms. For strongly convex and smooth objectives, we prove that gradient descent with output perturbation not only achieves nearly optimal utility, but also significantly improves the running …
Paper analyzes convergence of stochastic methods under heavy-tailed noise.
problem Analyzing convergence of stochastic methods under heavy-tailed noise.
method Investigates vanilla and clipped stochastic subgradient descent methods.
result Demonstrates convergence properties under sub-Weibull and p-BCM noise assumptions.
SVO improves particle diversity and variational bounds in filtering SMC.
problem Improving variational bounds in particle filtering with limited samples.
method Introduces Particle Smoothing Variational Objectives (SVO) for smoothed approximate posterior through subsampling.
result SVO outperforms filtered objectives with fewer Monte Carlo samples on nonlinear systems.
New algorithms optimize without knowing problem parameters.
problem Optimizing large-scale problems without knowing key parameters.
method Combining mirror descent with dual averaging techniques.
result Converges without prior knowledge of problem parameters.
AdaGrad fails to adapt to Hölder-smoothness in composite optimization problems.
problem AdaGrad's convergence rate is suboptimal for composite objectives.
method Exhibited a simple one-dimensional convex problem to highlight AdaGrad's limitations.
result AdaGrad does not achieve the classical convergence rate for Hölder-smooth objectives.
New method smooths optimization for sparse regularization.
problem Non-smooth, non-convex optimization problems for sparsity.
method Overparameterization and smooth surrogate penalties.
result Surrogate objective has identical global and local minima.
ConquerNet smooths quantile regression for deep learning with minimax guarantees.
problem Optimization challenges in quantile regression for deep models.
method ConquerNet uses convolution-smoothed quantile ReLU neural networks.
result ConquerNet provides minimax guarantees and outperforms standard quantile neural networks.
SGD optimality proven for convex objectives without smoothness assumptions.
problem Proving optimality of SGD for convex objectives without smoothness assumptions.
method Stochastic Gradient Descent (SGD) for convex objectives without smoothness or strict convexity assumptions.
result With high probability, the objective evaluated at the final candidate minimizer is close to the minimal value of the objective.
A method for learning from unlabeled time-series data using temporal smoothing and entropy maximization.
problem Learning from unlabeled time-series data efficiently and accurately.
method Training a feedforward neural network with two objectives: temporal smoothing and entropy maximization.
result The method extracts slowly evolving information from time-series data, filtering out noise.
Method approximates efficient frontier of chance-constrained programs.
problem Approximating the efficient frontier of chance-constrained nonlinear programs.
method Stochastic approximation method based on bi-objective viewpoint.
result Converges to stationary solutions of a smooth approximation of the original problem.
We compare various different definitions of "the category of smooth objects". The definitions compared are due to Chen, Frölicher, Sikorski, Smith, and Souriau. The method of comparison is to construct functors between the categories that enable us to see how the categories relate to each other. This produces a diagram…
Paper proves smoothness for variational inference, giving convergence guarantees.
problem Proving convergence guarantees for black-box variational inference.
method Describes gradients in an inner-product space, using Bessel's inequality.
result Objective is M-Lipschitz smooth if target is, excluding entropy.
A new biased gradient descent method for conditional stochastic optimization.
problem Challenges in constructing unbiased gradient estimators for conditional stochastic optimization.
method Proposes a biased stochastic gradient descent (BSGD) algorithm and analyzes its sample complexities.
result Establishes sample complexities of BSGD for various objectives and shows that BSpiderBoost matches the lower bound complexity.
New geometric objects generalize Lie groupoids, with nontrivial tangent bundle properties.
problem Generalizing Lie groupoids to nonassociative structures.
method Introducing quasiloopoids and loopoids, proving properties of their tangent bundles, and reformulating discrete mechanics.
result Tangent bundles of loopoids are canonically loopoids, but cotangent bundles are not.
If X is a smooth manifold then the R-algebra C∞(X) of smooth functions c:X→R is a C∞-ring. That is, for each smooth function f:Rn→R there is an n-fold operation Φf:C∞(X)n→C∞(X) acting by Φf:(c1,…,cn)↦f(c1,...,cn), a…
The paper introduces methods to learn smooth functions on hypergraphs with sparsity.
problem Learning smooth functions on hypergraphs with sparsity.
method General framework for smoothness measures, sparse learning on hypergraphs.
result Proposes sparsely smooth formulations that induce sparsity on hypergraphs and show benefits in handling irrelevant or noisy data.
New algorithms minimize noisy, irregular functions without gradients.
problem Minimizing noisy, irregular, and algebraically intractable functions.
method Generalized gradient descent recursion with smooth approximations.
result Convergence results under weak assumptions on function regularity.
Armijo line-search speeds up gradient descent for various functions.
problem Improving convergence rate of gradient descent.
method Applying Armijo line-search to adjust step-size in gradient descent.
result GD with Armijo line-search converges faster than GD with a fixed step-size.
Unified analysis of matching pursuit and coordinate descent methods.
problem Optimization of linear spaces using first-order methods.
method Unified analysis of matching pursuit and coordinate descent, providing rates for smooth and strongly convex objectives.
result Unified analysis leading to tightest known rates for steepest coordinate descent and accelerated convergence for matching pursuit.
Smooth Kahler-Einstein metrics have been studied for the past 80 years. More recently, singular Kahler-Einstein metrics have emerged as objects of intrinsic interest, both in differential and algebraic geometry, as well as a powerful tool in better understanding their smooth counterparts. This article is mostly a surve…
New method improves robustness of large models without sacrificing accuracy.
problem Improving robustness of large pre-trained models without accuracy loss.
method Multi-scale diffusion denoised smoothing, selectively applying smoothing at multiple noise scales.
result Strong certified robustness at high noise levels with accuracy close to non-smoothed classifiers.
This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms. Specifically, the smooth and nonsmooth terms are dealt with by gradient and proximal updates, respectively. The proposed algorithm is closely related to a p…
New algorithm finds local minima in non-convex, non-smooth problems.
problem Finding local minimizers in non-convex and non-smooth optimization.
method Perturbed Proximal Descent, tailored for non-smooth cases.
result First known results for non-smooth optimization.
Bayesian optimization on networks using Gaussian process models.
problem Optimizing expensive black-box functions on network structures.
method Developed Bayesian optimization algorithms with Gaussian process surrogates tailored to network geometry.
result Established regret bounds for smooth objective functions and analyzed practical cases.
SIXO improves inference by learning smoothing distributions from all observations.
problem Inference limitations due to ignoring future observations in filtering distributions.
method Density ratio estimation to warp filtering distributions into smoothing distributions, then use SMC with learned targets.
result Proves tighter log marginal lower bounds and more accurate inferences and estimates.
BézierGAN generates smooth curves from low-dimensional parameters.
problem Designing smooth curves for aerodynamic and hydrodynamic shapes.
method Generative model that maps low-dimensional latent representation to Bézier curve points.
result Generates diverse and realistic curves with consistent shape variation.
A new clustering algorithm considers data smoothness for better performance.
problem Clustering multi-scale data with varying cluster densities.
method Divide objects into tiny clusters, cluster centers form smooth graphs.
result Significantly outperforms state-of-the-art clustering algorithms.
Top-N-Rank improves top N item recommendations in scalable recommender systems.
problem Improving top N item recommendations in scalable recommender systems.
method Proposes a novel list-wise Learning-to-Rank model optimizing a variant of DCG objective function, incorporating weights for implicit feedback.
result Significant improvement in ranking quality for top N recommendations.
A new framework for training structured prediction models using smoothing.
problem Training smooth structured prediction models with non-smooth objectives.
method Smoothing over the maximum margin structured prediction objective to enable fast optimization.
result The proposed framework enables the use of efficient optimization algorithms for structured prediction.
MGDA converges under generalized smoothness for neural network optimization.
problem Optimizing neural networks with standard smoothness assumptions not holding.
method Revisited and analyzed MGDA and its stochastic version for generalized ℓ-smooth MOO problems. result MGDA and its variants converge to Pareto stationary points with guaranteed CA distance.
We consider the fundamental problem in non-convex optimization of efficiently reaching a stationary point. In contrast to the convex case, in the long history of this basic problem, the only known theoretical results on first-order non-convex optimization remain to be full gradient descent that converges in $O(1/\varep…
PPOS improves PPO by smoothing the surrogate objective function.
problem Performance instability and optimization inefficiency in PPO.
method Use of a functional clipping method instead of a flat clipping method.
result PPOS conducts more accurate updates and outperforms other PPO variants.
New DRO algorithm finds robust models without strong assumptions.
problem Learning robust models against distribution shift.
method Non-convex, non-smooth, non-asymptotic convergence guarantees for DRO.
result Algorithm finds first-order stationary points efficiently.
FeDualEx tackles saddle point optimization in federated learning with composite objectives.
problem Saddle point optimization with constraints and non-smooth regularization in federated learning.
method Federated Dual Extrapolation (FeDualEx) algorithm for saddle point optimization and composite objectives.
result FeDualEx effectively solves saddle point optimization problems with composite objectives in federated learning.
New method tackles non-smooth tensor data for better recovery.
problem Non-smooth changes in tensor data degrade traditional t-SVD methods.
method Learnable tensor nuclear norm, Alternating Proximal Multiplier Method (APMM), multi-objective tensor recovery framework.
result The proposed method effectively recovers tensor data with non-smooth changes.
Paper tackles optimization challenges in deep neural nets with nonconvex and non-smooth objectives.
problem Optimization of deep neural net models with nonconvex and non-smooth objectives.
method Summarizes challenges, state of the art, and presents numerical results on a specific class of problems.
result Numerical results on non-convex and non-smooth optimization problems.
We establish a relation between smooth 2-functors defined on the path 2-groupoid of a smooth manifold and differential forms on this manifold. This relation can be understood as a part of a dictionary between fundamental notions from category theory and differential geometry. We show that smooth 2-functors appear in se…
We review the basic definition of a stack and apply it to the topological and smooth settings. We then address two subtleties of the theory: the correct definition of a ``stack over a stack'' and the distinction between small stacks (which are algebraic objects) and large stacks (which are generalized spaces).