arXiv research

This paper bounds the Lipschitz constants of neural networks and their gradients.

New method for differentially private optimization with general Lipschitz conditions.

Adaptive sampling improves convergence in heterogeneous distributed optimization.

New PG methods tackle nonconvex optimization with auto-conditioned stepsizes.

GraN-GAN normalizes gradients for better GAN performance.

This paper analyzes challenges and solutions in deep learning optimization.

New Transformers maintain Lipschitz continuity for robustness.

Adaptive LR improves neural network Lipschitz regularity without slowing convergence.

Proposes a learning rate method for shallow nets based on gradient Lipschitz constant.

We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant---for multiple $p$-norms---of a feed forward neural network composed of commonly used layer types.…

Generative adversarial networks (GANs) are one of the most popular approaches when it comes to training generative models, among which variants of Wasserstein GANs are considered superior to the standard GAN formulation in terms of learning stability and sample quality. However, Wasserstein GANs require the critic to b…

Tight estimation of the Lipschitz constant for deep neural networks (DNNs) is useful in many applications ranging from robustness certification of classifiers to stability analysis of closed-loop systems with reinforcement learning controllers. Existing methods in the literature for estimating the Lipschitz constant su…

New method for efficient proximal mapping of 1-path-norm in shallow networks.

Constructs a map with prescribed local Lipschitz constants on a subset of a manifold.

New approach achieves optimal rates for differentially private stochastic convex optimization with heavy-tailed gradients.

Optimistic method adapted for faster convex-concave min-max problems.

We compute the local Lipschitz constant of ReLU networks precisely.

HALO uses local Lipschitz constants to optimize functions efficiently.

New MIP formulations for neural network Lipschitz constant estimation.

Enhances ENet's prediction accuracy while maintaining uncertainty estimation.

Paper investigates Lipschitz constants of self-attention modules in neural networks.

New algorithm improves online learning with reduced discretization.

We show that any smooth bi-Lipschitz $h$ can be represented exactly as a composition $h_m \circ ... \circ h_1$ of functions $h_1,...,h_m$ that are close to the identity in the sense that each $\left(h_i-\mathrm{Id}\right)$ is Lipschitz, and the Lipschitz constant decreases inversely with the number $m$ of functions com…

New metric measure space theory for Lipschitz constants.

Unified high-probability regret bounds for online convex optimisation with randomised gradient estimators.

Boosted Frank-Wolfe accelerates optimization for nonconvex problems.

Neural networks are dense among Lipschitz functions with fixed Lipschitz constant.

Extends Lipschitz functions while preserving local constants.

The paper studies properties of intrinsically Lipschitz constants in metric spaces.

We study the problem of sampling from a probability distribution $π$ on $\rset^d$ which has a density \wrt\ the Lebesgue measure known up to a normalization factor $x \mapsto \rme^{-U(x)} / \int_{\rset^d} \rme^{-U(y)} \rmd y$. We analyze a sampling method based on the Euler discretization of the Langevin stochastic dif…

We provide larger step-size restrictions for which gradient descent based algorithms (almost surely) avoid strict saddle points. In particular, consider a twice differentiable (non-convex) objective function whose gradient has Lipschitz constant L and whose Hessian is well-behaved. We prove that the probability of init…

New method improves neural network robustness to adversarial attacks.

Several recent papers have discussed utilizing Lipschitz constants to limit the susceptibility of neural networks to adversarial examples. We analyze recently proposed methods for computing the Lipschitz constant. We show that the Lipschitz constant may indeed enable adversarially robust neural networks. However, the m…

This paper analyzes the Lipschitz constants of deep neural networks with random weights.

This paper studies bounds for the Lipschitz constant of random neural networks.

This work focuses on important step in quantitative topology: given homotopic mappings from $S^m$ to $S^n$ of Lipschitz constant $L$, build the (asymptotically) simplest homotopy between them (meaning having the least Lipschitz constant). The present paper resolves this problem for the first case where Hopf invariant p…

The paper proposes a method to train NNs with a small Lipschitz constant to improve robustness.

Adjusting the learning rate schedule in stochastic gradient methods is an important unresolved problem which requires tuning in practice. If certain parameters of the loss function such as smoothness or strong convexity constants are known, theoretical learning rate schedules can be applied. However, in practice, such …

A new algorithm solves minimax problems without needing parameters.

New solver SR2 tackles deep neural network training with nonsmooth regularization.

The paper classifies capillary graphs on manifolds with Ricci lower bounds.

We study online optimization of smoothed piecewise constant functions over the domain [0, 1). This is motivated by the problem of adaptively picking parameters of learning algorithms as in the recently introduced framework by Gupta and Roughgarden (2016). Majority of the machine learning literature has focused on Lipsc…

Paper proves convergence for private FL on non-Lipschitz convex objectives using normalization instead of clipping.

LiPopt uses polynomial optimization to estimate neural network Lipschitz constants efficiently.

CLIP controls neural network stability by bounding Lipschitz constants.

The paper studies Lipschitz bounds for integral kernels under differentiability assumptions.

Improved variational inequality algorithms using adaptive step sizes.

We determine the asymptotic behavior of the optimal Lipschitz constant for the systole map from Teichmuller space to the curve complex.