Adapts Hölder smoothness with normalized gradients.
problem Improving smoothness adaptation methods.
method Black-box adaptation of Levy's method using normalized gradients.
result Bound depends on local Hölder smoothness.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Gradient Smoothing”
Generalizes smoothness conditions for optimization methods.
problem Optimization under non-uniform smoothness conditions.
method Develops a new analysis technique for bounding gradients.
result Obtains convergence rates for gradient descent and Nesterov's method.
We generalize stochastic smoothing for gradient estimation of non-differentiable functions.
problem Gradient estimation for non-differentiable functions.
method Developed a general framework for relaxation and gradient estimation of non-differentiable black-box functions using stochastic smoothing with reduced assumptions.
result Empirically validated the effectiveness of variance reduction strategies for various non-differentiable tasks.
New algorithm tackles nonconvex machine learning problems with adaptive normalization and independent sampling.
problem Nonconvex machine learning problems with generalized-smoothness.
method Adaptive gradient normalization, independent sampling, and gradient clipping.
result Achieves an O(ε^(-4)) sample complexity for fast convergence.
Stochastic gradient methods are dominant in nonconvex optimization especially for deep models but have low asymptotical convergence due to the fixed smoothness. To address this problem, we propose a simple yet effective method for improving stochastic gradient methods named predictive local smoothness (PLS). First, we …
Develops accelerated methods for optimization using low-dimensional projected-gradient information.
problem Optimization with low-dimensional projected-gradient information and Nesterov acceleration.
method Randomized-subspace Nesterov accelerated gradient methods for smooth convex and strongly convex optimization.
result Established accelerated oracle-complexity guarantees and unified basis for comparing sketch families.
New algorithm solves federated minimax optimization problems.
problem Federated minimax optimization challenges.
method Federated Stochastic Smoothed Gradient Descent Ascent (FESS-GDA).
result FESS-GDA uniformly solves federated minimax problems.
AdaGrad fails to adapt to Hölder-smoothness in composite optimization problems.
problem AdaGrad's convergence rate is suboptimal for composite objectives.
method Exhibited a simple one-dimensional convex problem to highlight AdaGrad's limitations.
result AdaGrad does not achieve the classical convergence rate for Hölder-smooth objectives.
We establish local elliptic and parabolic gradient estimates for positive smooth solutions to a nonlinear parabolic equation on a smooth metric measure space. As applications, we determine various conditions on the equation's coefficients and the growth of solutions that guarantee the nonexistence of nontrivial positiv…
The paper provides gradient estimates for nonlinear heat-type equations on smooth metric measure spaces.
problem Proving gradient estimates for nonlinear heat-type equations on smooth metric measure spaces.
method Using Hamilton type and Li-Yau type estimates, the paper proves gradient estimates on positive solutions to generalized nonlinear parabolic equations on smooth metric measure spaces with compact boundary.
result Gradient estimates for nonlinear heat-type equations on smooth metric measure spaces.
The paper proves gradient estimates for nonlinear parabolic equations on smooth metric measure spaces.
problem Proving gradient estimates for nonlinear parabolic equations on smooth metric measure spaces.
method Using Souplet-Zhang type estimates and properties of Bakry-Emery Ricci tensor and weighted mean curvature.
result Gradient estimates for nonlinear parabolic equations on smooth metric measure spaces with Dirichlet boundary condition.
Gradient descent with logistic loss can interpolate deep networks with smoothed ReLU activations under certain conditions.
problem Conditions for gradient descent to drive logistic loss to zero in deep networks with smoothed ReLU activations.
method Gradient descent applied to fixed-width deep networks with smoothed ReLU approximations (e.g., Swish, Huberized ReLU).
result Gradient descent can drive logistic loss to zero under specific conditions, providing bounds on convergence rate.
New optimization method combines gradient clipping and non-Euclidean smoothness.
problem Improving optimization in non-Euclidean spaces for machine learning.
method Hybrid of steepest descent and conditional gradient, incorporating weight decay.
result Achieves optimal convergence rate and demonstrates effectiveness in deep learning.
Paper relaxes SGD privacy and generalization guarantees for non-smooth convex losses.
problem Privacy and generalization in SGD for non-smooth convex losses.
method Relaxes Lipschitz and strong smoothness assumptions to Hölder smoothness, proving -DP and optimal excess risk.
result Noisy SGD with -Hölder smooth losses achieves optimal excess risk with linear gradient complexity for .
In this article, we mathematically study several GAN related topics, including Inception score, label smoothing, gradient vanishing and the -log(D(x)) alternative. --- An advanced version is included in arXiv:1703.02000 "Activation Maximization Generative Adversarial Nets". Please refer Section 6 in 1703.02000 for deta…
Improved optimization technique reduces training complexity for non-convex problems.
problem Training non-convex optimization problems with exploding gradients.
method Employed variance reduction technique (SPIDER) with carefully designed learning rate.
result Improved stochastic gradient complexity to for -stationary solutions.
New SPS variant improves non-smooth optimization without small gradients.
problem Improving non-smooth optimization without small gradients.
method Safeguarded Stochastic Polyak Step Size (SPS) for non-smooth optimization.
result Rigorous convergence guarantees for non-smooth convex optimization without strong assumptions.
The paper provides new gradient estimates for solutions to a nonlinear elliptic equation on smooth metric measure spaces.
problem Gradient estimates for solutions to a specific nonlinear elliptic equation on smooth metric measure spaces.
method Nash-Moser iteration technique to obtain local gradient estimates.
result New local gradient estimates for positive solutions to the equation.
Uniform convergence of the smooth calibration error and its relationship with functional gradientstat.ML
The paper provides a uniform convergence bound for smooth calibration error and its relationship with functional gradient.
problem Limited theoretical understanding of learning algorithms achieving high accuracy and good calibration.
method Focuses on smooth calibration error, providing a uniform convergence bound and proving the relationship with functional gradient.
result Derives conditions for simultaneous classification and calibration guarantees in gradient boosting trees, kernel boosting, and neural networks.
The paper provides approximation guarantees for neural networks trained with gradient flow.
problem Approximating neural networks trained with gradient flow in continuous -norm.
method NTK argument for non-convex second but last layer, under-parametrized regime.
result Gradient flow convergence guarantees for neural networks under Sobolev smoothness assumptions.
Improved sampling from high-dimensional Gaussians using smoothed scores.
problem Sampling from high-dimensional Gaussian distributions with gradient information.
method Using smoothed scores, which are gradients of the logarithms of Gaussian-convolved densities, to overcome approximation barriers.
result Improved sampling efficiency with a complexity of \(O\left(\left(\logκ+\log(e\sqrt d/δ_{
m TV})
ight)\log(e\sqrt d/δ_{
m TV})
ight)\) smoothed-score queries.
Unified signSGD and gradient descent analysis for neural networks.
problem Performance of sign-based optimization methods in neural networks.
method Unified analysis of separable smoothness and -smoothness, isolating geometric properties affecting performance.
result Sign-based methods are preferable over gradient descent under specific Hessian properties in deep networks.
Shuffling the Stochastic Mirror Descent via Dual Lipschitz Continuity and Kernel Conditioningmath.OC
New method improves optimization algorithms without Lipschitz smoothness.
problem Improving optimization algorithms in the absence of Lipschitz smoothness.
method Dual kernel conditioning (DKC) to provide dual Lipschitz continuity.
result First complexity bounds and iterate convergence for random reshuffling mirror descent.
We provide tight upper and lower bounds on the complexity of minimizing the average of convex functions using gradient and prox oracles of the component functions. We show a significant gap between the complexity of deterministic vs randomized optimization. For smooth functions, we show that accelerated gradient de…
New method provides tighter robustness guarantees for adversarial attacks.
problem Ensuring robustness against adversarial attacks in machine learning models.
method Developed a Second-order Smoothing (SoS) robustness certificate using Gaussian random smoothing.
result SoS certificates are tighter and provide improved robustness on high-dimensional datasets.
Armijo line-search speeds up gradient descent for various functions.
problem Improving convergence rate of gradient descent.
method Applying Armijo line-search to adjust step-size in gradient descent.
result GD with Armijo line-search converges faster than GD with a fixed step-size.
Large-batch stochastic gradient descent (SGD) is widely used for training in distributed deep learning because of its training-time efficiency, however, extremely large-batch SGD leads to poor generalization and easily converges to sharp minima, which prevents naive large-scale data-parallel SGD (DP-SGD) from convergin…
We consider in this paper a class of composite optimization problems whose objective function is given by the summation of a general smooth and nonsmooth component, together with a relatively simple nonsmooth term. We present a new class of first-order methods, namely the gradient sliding algorithms, which can skip the…
New shuffling methods improve convergence without Lipschitz smoothness.
problem Lack of convergence guarantees for shuffling methods under non-Lipschitz conditions.
method Revisit shuffling methods, prove convergence under general bounded variance condition.
result Matched current best-known convergence rates without Lipschitz smoothness.
This work accelerates gradient descent with anytime convergence guarantees.
problem Improving the convergence rate of gradient descent methods.
method Proposes a stepsize schedule for gradient descent that achieves anytime convergence rates.
result Gradient descent can achieve convergence rates of for any stopping time .
This paper improves convergence guarantees for SGD algorithms in non-convex smooth functions.
problem Theoretical convergence properties of SGD algorithms for non-convex smooth functions.
method Analysis of SGD algorithms with arbitrary data ordering for non-convex smooth functions.
result Enhanced convergence guarantees for incremental gradient and single shuffle SGD, improving the optimization term of convergence guarantee.
We study a hybrid conditional gradient - smoothing algorithm (HCGS) for solving composite convex optimization problems which contain several terms over a bounded set. Examples of these include regularization problems with several norms as penalties and a norm constraint. HCGS extends conditional gradient methods to cas…
In this paper, we consider the following general evolution equation on smooth metric measure spaces . We give a local gradient estimate of Souplet-Zhang type for positive smooth solution of this equation provided that the Bakry-Émery curvature bounded from below. When …
New algorithm improves gradient-based ERM for smooth convex losses.
problem Empirical risk minimization of smooth, strongly convex loss functions.
method Iterative gradient-based method with local polynomial regression.
result Oracle complexity of for our algorithm.
New example of non-Kähler soliton with Kähler-like behavior at infinity.
problem Constructing non-Kähler expanding gradient Ricci solitons.
method Asymptotically conical (AC) construction with Kähler tangent cone at infinity.
result Example of a non-Kähler soliton with a Kähler-like behavior at infinity.
Improved analysis and new algorithm for gradient-free optimization of smooth functions.
problem Minimization of highly smooth functions with noisy oracle information.
method Two zero-order projected gradient descent algorithms based on randomization over the and spheres, with improved analysis and theoretical guarantees.
result Improved convergence rates and theoretical guarantees for various function classes.
Paper improves stochastic bilevel optimization methods for highly-smooth problems.
problem Finding -stationary points in stochastic bilevel optimization.
method Proposes FSA- methods using th-order finite differences for hyper-gradient approximation.
result Achieves upper complexity bound of for th-order smooth problems.
We consider first order gradient methods for effectively optimizing a composite objective in the form of a sum of smooth and, potentially, non-smooth functions. We present accelerated and adaptive gradient methods, called FLAG and FLARE, which can offer the best of both worlds. They can achieve the optimal convergence …
Untuned SGD converges but with an exponential dependence on smoothness, adaptive methods prevent this.
problem The exponential dependence on smoothness in untuned SGD's convergence rate.
method Untuned SGD with arbitrary stepsize η, adaptive methods like NSGD, AMSGrad, and AdaGrad.
result Adaptive methods prevent the exponential dependence on smoothness in SGD.
Hamilton's Gradient Estimates and A Monotonicity Formula for Heat Flows on Metric Measure Spacesmath.MG
In this paper, we extend the Hamilton's gradient estimates \cite{har93} and a monotonicity formula of entropy \cite{ni04} for heat flows from smooth Riemannian manifolds to (non-smooth) metric measure spaces with appropriate Riemannian curvature-dimension condition.
This paper analyzes the convergence of Federated Average under relaxed assumptions.
problem Lack of theoretical analysis for Federated Average under assumptions beyond smoothness.
method Relaxing assumptions of strong smoothness to semi-smoothness and semi-Lipschitz properties, and introducing a bound on the gradient.
result Provides a theoretical convergence study on Federated Learning under new assumptions.
Capillary -curvature problemmath.AP
Proves existence of smooth convex solutions to capillary curvature equations.
problem Proving existence of smooth convex solutions to capillary curvature equations.
method Gradient estimate for capillary curvature equations in half-space.
result Existence of even, smooth, strictly convex solutions for all and .
Optimal private ERM and SCO with subquadratic gradient complexity.
problem Private optimization of non-smooth convex functions.
method Subquadratic gradient complexity algorithm using subsampling and smoothing.
result Achieved optimal excess empirical risk and population loss.
Kernel-Gradient Drifting improves generative modeling for non-Euclidean data.
problem Challenges in generative modeling for non-Euclidean data.
method Replaces Euclidean displacement with kernel-induced directions, exposing score-based structure.
result Kernel-gradient drifting enables state-of-the-art one-step generation for non-Euclidean data.
This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms. Specifically, the smooth and nonsmooth terms are dealt with by gradient and proximal updates, respectively. The proposed algorithm is closely related to a p…
DSPI connects natural policy gradient to policy iteration, proving global convergence.
problem Optimizing policies in reinforcement learning.
method DSPI framework, combining smoothed policy iteration and natural policy gradient.
result DSPI achieves geometric convergence and optimal complexity for policy optimization.
We use matrix iteration theory to characterize acceleration in smooth games. We define the spectral shape of a family of games as the set containing all eigenvalues of the Jacobians of standard gradient dynamics in the family. Shapes restricted to the real line represent well-understood classes of problems, like minimi…
Introduces a new theoretical framework for exponential smoothing.
problem Theoretical foundation and robustness of simple exponential smoothing.
method Stochastic gradient ascent to optimize Gaussian log-likelihood functions.
result Simple exponential smoothing converges to the trend of a trend-stationary process.