This paper bounds the Lipschitz constants of neural networks and their gradients.
problem Estimating the Lipschitz constant of complex models like neural networks.
method Local upper and lower bounds on Lipschitz constants computed with respect to network parameters.
result It is impossible to derive global upper bounds for the Lipschitz constants of neural networks.
New methods for convex optimization with locally Lipschitz gradient, achieving faster convergence.
problem Optimization problems with locally Lipschitz continuous gradient.
method Accelerated proximal gradient (APG) methods and proximal augmented Lagrangian method.
result Achieved faster convergence rates for convex optimization problems with locally Lipschitz gradient.
We focus our attention on the notion of intrinsic Lipschitz graphs, inside a special class of metric spaces i.e. the Carnot groups. More precisely, we provide a characterization of locally intrinsic Lipschitz functions in Carnot groups of step 2 in terms of their intrinsic distributional gradients.
Introduces gradient decay in Softmax for better generalization.
problem Improving generalization performance in neural networks.
method Gradient decay hyperparameter in Softmax for varying gradient rates based on probability.
result Gradient decay rate affects generalization performance and can be tuned for better optimization.
New method improves optimization algorithms without Lipschitz smoothness.
problem Improving optimization algorithms in the absence of Lipschitz smoothness.
method Dual kernel conditioning (DKC) to provide dual Lipschitz continuity.
result First complexity bounds and iterate convergence for random reshuffling mirror descent.
We study distributed optimization algorithms for minimizing the average of \emph{heterogeneous} functions distributed across several machines with a focus on communication efficiency. In such settings, naively using the classical stochastic gradient descent (SGD) or its variants (e.g., SVRG) with a uniform sampling of …
Study Ricci-Deturck flow from rough metrics, proving short-time existence.
problem Short-time existence of Ricci-Deturck flow from rough metrics.
method Ricci-Deturck flow, bi-Lipschitz metrics, small gradient concentration.
result Proved short-time existence of Ricci-Deturck flow.
Adaptive methods improve gradient descent and proximal gradient for convex optimization.
problem Improving efficiency of gradient descent and proximal gradient methods.
method Adaptive versions of GD and ProxGD using local curvature information.
result Proved convergence with local Lipschitz gradient assumptions.
HALO uses local Lipschitz constants to optimize functions efficiently.
problem Efficiently solving global optimization problems with complex objective functions.
method Hybrid Adaptive Lipschizian Optimization (HALO) algorithm that estimates local Lipschitz constants and balances global and local information.
result HALO outperforms other global optimization algorithms on numerous test functions.
The paper analyzes neural network dynamics after weights escape the origin.
problem Understanding gradient flow dynamics of neural networks after the origin.
method Analyzes gradient flow of homogeneous neural networks with locally Lipschitz gradients.
result Characterizes the first saddle point encountered after escaping the origin.
Constructs a map with prescribed local Lipschitz constants on a subset of a manifold.
problem Creating a Lipschitz map with specific local Lipschitz constants on a subset of a manifold.
method Constructs a Lipschitz map that matches a given map on a subset and has a local Lipschitz constant defined by a continuous function.
result A Lipschitz map can be constructed with a local Lipschitz constant prescribed by a continuous function.
New bounds explain neural network generalization by considering local Lipschitz properties.
problem Existing bounds fail to account for initialization and SGD biases.
method Optimal transport interpretation of generalization problem.
result Instance-dependent bounds that depend on local Lipschitz regularity.
New method balances multivariate model fitting for mixed likelihoods.
problem Multivariate models often fit only a subset of observed variables.
method Lipschitz standardization for data preprocessing.
result Lipschitz standardization leads to more accurate multivariate models.
Gradient flow of curve length on Sobolev metrics preserves convexity.
problem Optimal low-regularity gradient flow of curve length.
method Explicit gradient formula, Picard-Lindelöf theorem, time-reparametrisation.
result Exponential decay of length and preservation of convexity.
Maps and measures on surfaces link best Lipschitz and least gradient functions.
problem Analyzing maps between surfaces and their geometric properties.
method Duality between best Lipschitz and least gradient maps, geodesic laminations, and transverse measures.
result The infinity harmonic map defines a geodesic lamination and the least gradient map defines a transverse measure.
The mean curvature flow is the gradient flow of volume functionals on the space of submanifolds. We prove a fundamental regularity result of the mean curvature flow in this paper: a Lipschitz submanifold with small local Lipschitz norm becomes smooth instantly along the mean curvature flow. This generalizes the regular…
We compute the local Lipschitz constant of ReLU networks precisely.
problem Estimating the local Lipschitz constant of ReLU networks is hard.
method We use a novel approach involving the generalized Jacobian and backpropagation.
result We provide an algorithm to compute the exact Lipschitz constant of ReLU networks.
GD converges in unstable regimes, even with oscillatory behavior.
problem Understanding convergence of GD in unstable regimes.
method Analysis of two-step gradient updates.
result Characterization of local conditions for convergence.
New method for differentially private optimization with general Lipschitz conditions.
problem Differentially private optimization under general Lipschitz conditions.
method Generalized Lipschitz condition for per-sample gradients, tuning clip norm based on minimum per-sample Lipschitz constant.
result Efficacy of the recommended clip norm tuning method verified on 8 datasets.
Extends Lipschitz functions while preserving local constants.
problem Extending Lipschitz functions on metric spaces while maintaining local constants.
method Extends Lipschitz functions on metric spaces while locally preserving the asymptotic Lipschitz constant.
result Sobolev spaces on metric measure spaces are invariant under isomorphism of mm-structures.
Let (X,d,μ) be a complete metric measure space, with μ a locally doubling measure, that supports a local weak L2-Poincaré inequality. By assuming a heat semigroup type curvature condition, we prove that Cheeger-harmonic functions are Lipschitz continuous on (X,d,μ). Gradient estimates for Cheeger-harmonic func…
Optimistic method adapted for faster convex-concave min-max problems.
problem Solving convex-concave min-max optimization problems efficiently.
method Adaptive, line search-free second-order methods combining optimistic updates and second-order information.
result Achieves optimal convergence rate without line search or backtracking.
The curse of dimensionality affects neural network optimization, especially with smooth functions.
problem The curse of dimensionality in neural network optimization.
method Examined through the evolution of the parameter distribution under 2-Wasserstein gradient flow.
result The curse of dimensionality persists in neural network optimization, even with smooth functions.
The Jacobian matrix (or the gradient for single-output networks) is directly related to many important properties of neural networks, such as the function landscape, stationary points, (local) Lipschitz constants and robustness to adversarial attacks. In this paper, we propose a recursive algorithm, RecurJac, to comput…
This paper analyzes convergence of large-scale Transformers with weight decay.
problem Understanding optimization guarantees in large-scale Transformer training.
method Construct mean-field limit, show gradient flow convergence to PDE, demonstrate global minimum consistency.
result Gradient flow reaches global minimum in large-scale Transformers with small weight decay.
New Transformers maintain Lipschitz continuity for robustness.
problem Ensuring robustness in Transformers for safety-sensitive applications.
method Introducing gradient-descent-type in-context Transformers with explicit Euler steps of negative gradient flows.
result Universal approximation theorem for Lipschitz continuous Transformers.
We construct short retractions of a CAT(1) space to its small convex subsets. This construction provides an alternative geometric description of an analytic tool introduced by Wilfrid Kendall. Our construction uses a tractrix flow which can be defined as a gradient flow for a family of functions of certain type. In an …
In this paper, we are concerned with a non-asymptotic analysis of sampling algorithms used in nonconvex optimization. In particular, we obtain non-asymptotic estimates in Wasserstein-1 and Wasserstein-2 distances for a popular class of algorithms called Stochastic Gradient Langevin Dynamics (SGLD). In addition, the afo…
New method estimates Riemannian derivatives from noisy function evaluations.
problem Optimizing functions on Riemannian manifolds with noisy data.
method Riemannian Gaussian smoothing for gradient and Hessian estimation.
result Oracle complexity independent of ambient dimension.
Lipschitz continuity recently becomes popular in generative adversarial networks (GANs). It was observed that the Lipschitz regularized discriminator leads to improved training stability and sample quality. The mainstream implementations of Lipschitz continuity include gradient penalty and spectral normalization. In th…
Lipschitz constraints under L2 norm on deep neural networks are useful for provable adversarial robustness bounds, stable training, and Wasserstein distance estimation. While heuristic approaches such as the gradient penalty have seen much practical success, it is challenging to achieve similar practical performance wh…
We give sufficient conditions for a Cc1-local diffeomorphism between Fréchet spaces to be a global one. We extend the Clarke's theory of generalized gradients to the more general setting of Fréchet spaces. As a consequence, we define the Chang Palais-Smale condition for Lipschitz functions and show that a functio…
We model how Lipschitz continuity changes during neural network training.
problem Understanding how Lipschitz continuity evolves during training.
method We use a system of stochastic differential equations to capture the dynamics of Lipschitz continuity under SGD.
result We identify three factors driving the evolution of Lipschitz continuity: gradient flow projection, gradient noise, and Hessian projection.
ModHiFi identifies critical components for model modification without gradients or loss function.
problem Modifying open weight models without access to training data or loss function.
method Theoretical analysis of Lipschitz-continuous networks, Subset Fidelity metric, and ModHiFi algorithm.
result ModHiFi-P and ModHiFi-U achieve significant performance improvements in model pruning and unlearning.
In this paper we describe the notion of a weak lipschitzianity of a mapping on a Cq stratification. We also distinguish a class of regularity conditions that are in some sense invariant under definable, locally Lipschitz and weakly bi-Lipschitz homeomorphisms. This class includes the Whitney (B) condition and the …
Training neural networks under a strict Lipschitz constraint is useful for provable adversarial robustness, generalization bounds, interpretable gradients, and Wasserstein distance estimation. By the composition property of Lipschitz functions, it suffices to ensure that each individual affine transformation or nonline…
We investigate the challenge of multi-output learning, where the goal is to learn a vector-valued function based on a supervised data set. This includes a range of important problems in Machine Learning including multi-target regression, multi-class classification and multi-label classification. We begin our analysis b…
Gradient descent converges linearly in finite-width networks with positive NTK and compatible conditions.
problem Local convergence of gradient descent in finite-width networks.
method Positive Neural Tangent Kernel (NTK), local Polyak-Łojasiewicz inequality, fixed-step containment in Locally Quasi-Convex Region (LQCR).
result Linear convergence achieved under specific conditions.
This paper analyzes the convergence of Federated Average under relaxed assumptions.
problem Lack of theoretical analysis for Federated Average under assumptions beyond smoothness.
method Relaxing assumptions of strong smoothness to semi-smoothness and semi-Lipschitz properties, and introducing a bound on the gradient.
result Provides a theoretical convergence study on Federated Learning under new assumptions.
New shuffling methods improve convergence without Lipschitz smoothness.
problem Lack of convergence guarantees for shuffling methods under non-Lipschitz conditions.
method Revisit shuffling methods, prove convergence under general bounded variance condition.
result Matched current best-known convergence rates without Lipschitz smoothness.
In this paper, we study the convergence of generative adversarial networks (GANs) from the perspective of the informativeness of the gradient of the optimal discriminative function. We show that GANs without restriction on the discriminative function space commonly suffer from the problem that the gradient produced by …
Generative adversarial networks (GANs) are one of the most popular approaches when it comes to training generative models, among which variants of Wasserstein GANs are considered superior to the standard GAN formulation in terms of learning stability and sample quality. However, Wasserstein GANs require the critic to b…
A Lipschitz hypersurface is a hypersurface which locally is the graph of a Lipschitz function. A Lipschitz (or C^1) hypersurface is said to be Levi-flat if it is locally foliated by complex manifolds of complex dimension (n-1). We shall prove that there exist no Lipschitz Levi-flat hypersurfaces in CP^n with n >= 3. Ou…
The paper examines partial regularity of Lipschitz solutions to minimal surface system.
problem Understanding the regularity of solutions to the minimal surface system.
method Investigation of stationary, integral weak, and viscosity solutions; interior gradient estimate using maximum principle.
result Partial regularity results for Lipschitz solutions, including interior gradient estimate.
GraN-GAN normalizes gradients for better GAN performance.
problem Improving image generation in GANs with piecewise linear discriminators.
method Piecewise Gradient Normalization (GraN) for input-dependent normalization.
result Significant performance gains in image generation across various datasets.
Sharp Lipschitz bounds and gradient estimates for fully nonlinear parabolic equations.
problem Understanding moduli of continuity for fully nonlinear parabolic equations.
method Proving moduli of continuity of viscosity solutions are subsolutions of one-dimensional parabolic equations.
result Sharp Lipschitz bounds and gradient estimates for fully nonlinear parabolic equations with bounded initial data.
New algorithms sample from log concave distributions without gradient Lipschitz continuity.
problem Sampling from log concave distributions without gradient Lipschitz continuity.
method Two algorithms based on monotone polygonal (tamed) Euler schemes.
result Non-asymptotic 2-Wasserstein distance bounds between the process and target measure.
Wasserstein GAN(WGAN) is a model that minimizes the Wasserstein distance between a data distribution and sample distribution. Recent studies have proposed stabilizing the training process for the WGAN and implementing the Lipschitz constraint. In this study, we prove the local stability of optimizing the simple gradien…