Constructs a map with prescribed local Lipschitz constants on a subset of a manifold.
problem Creating a Lipschitz map with specific local Lipschitz constants on a subset of a manifold.
method Constructs a Lipschitz map that matches a given map on a subset and has a local Lipschitz constant defined by a continuous function.
result A Lipschitz map can be constructed with a local Lipschitz constant prescribed by a continuous function.
We compute the local Lipschitz constant of ReLU networks precisely.
problem Estimating the local Lipschitz constant of ReLU networks is hard.
method We use a novel approach involving the generalized Jacobian and backpropagation.
result We provide an algorithm to compute the exact Lipschitz constant of ReLU networks.
New MIP formulations for neural network Lipschitz constant estimation.
problem Ensuring robustness of neural networks by calculating their Lipschitz constant.
method Reformulating the neural network Lipschitz estimation problem as a Quadratically Constrained MIP (MIQCQP) problem.
result Solutions of the MIQCQP formulations provide bounds on the Lipschitz constant, with conditions for exactness.
Paper investigates Lipschitz constants of self-attention modules in neural networks.
problem Lipschitz constants of self-attention modules in neural networks.
method Proved standard dot-product self-attention is not Lipschitz for unbounded input domain. Proposed L2 self-attention that is Lipschitz. Derived upper bound on L2 self-attention's Lipschitz constant.
result Proved standard self-attention is not Lipschitz for unbounded input domain and proposed an alternative L2 self-attention that is Lipschitz.
New metric measure space theory for Lipschitz constants.
problem Defining and characterizing Cheeger energy in metric measure spaces.
method Adapting Cheeger theory to intrinsically Lipschitz sections.
result Characterization of intrinsic Cheeger energy in terms of relaxed slope.
Neural networks are dense among Lipschitz functions with fixed Lipschitz constant.
problem Characterizing neural network approximations to Lipschitz functions.
method Analyzing L-Lipschitz neural networks and their density in L-Lipschitz functions. result One layer neural networks are dense in the set of all L-Lipschitz functions. Extends Lipschitz functions while preserving local constants.
problem Extending Lipschitz functions on metric spaces while maintaining local constants.
method Extends Lipschitz functions on metric spaces while locally preserving the asymptotic Lipschitz constant.
result Sobolev spaces on metric measure spaces are invariant under isomorphism of mm-structures.
The paper studies properties of intrinsically Lipschitz constants in metric spaces.
problem Investigating properties of intrinsically Lipschitz constants.
method Introduced Leibniz and product formulas for intrinsic slope.
result Formulated Leibniz and product formulas for intrinsic slope.
This paper bounds the Lipschitz constants of neural networks and their gradients.
problem Estimating the Lipschitz constant of complex models like neural networks.
method Local upper and lower bounds on Lipschitz constants computed with respect to network parameters.
result It is impossible to derive global upper bounds for the Lipschitz constants of neural networks.
Several recent papers have discussed utilizing Lipschitz constants to limit the susceptibility of neural networks to adversarial examples. We analyze recently proposed methods for computing the Lipschitz constant. We show that the Lipschitz constant may indeed enable adversarially robust neural networks. However, the m…
This paper analyzes the Lipschitz constants of deep neural networks with random weights.
problem Estimating the Lipschitz constants of deep neural networks with random parameters.
method High probability upper and lower bounds derived for ReLU neural networks with He initialization.
result The behavior of the Lipschitz constant varies significantly between p∈[1,2) and p∈[2,∞]. This paper studies bounds for the Lipschitz constant of random neural networks.
problem Quantifying the worst-case robustness of neural networks against adversarial perturbations.
method Analyzes upper and lower bounds for the Lipschitz constant of random ReLU neural networks under specific initialization conditions.
result For deep networks, the upper bound is larger than the lower bound by a logarithmic factor in width.
This work focuses on important step in quantitative topology: given homotopic mappings from Sm to Sn of Lipschitz constant L, build the (asymptotically) simplest homotopy between them (meaning having the least Lipschitz constant). The present paper resolves this problem for the first case where Hopf invariant p…
The paper proposes a method to train NNs with a small Lipschitz constant to improve robustness.
problem Neural networks' susceptibility to adversarial perturbations in safety-critical applications.
method The paper introduces a framework to train multi-layer NNs by minimizing their Lipschitz constant, using an optimization scheme based on the Alternating Direction Method of Multipliers.
result The proposed training procedure successfully increases the robustness of neural networks.
LiPopt uses polynomial optimization to estimate neural network Lipschitz constants efficiently.
problem Estimating the Lipschitz constant of neural networks efficiently.
method Sparse polynomial optimization, leveraging network connectivity to reduce complexity.
result Superior estimates of the ℓ∞-Lipschitz constant compared to existing methods. New method for differentially private optimization with general Lipschitz conditions.
problem Differentially private optimization under general Lipschitz conditions.
method Generalized Lipschitz condition for per-sample gradients, tuning clip norm based on minimum per-sample Lipschitz constant.
result Efficacy of the recommended clip norm tuning method verified on 8 datasets.
CLIP controls neural network stability by bounding Lipschitz constants.
problem Neural networks lack mathematical guarantees of stability, especially to adversarial examples.
method Develops a variational regularization method (CLIP) to control the Lipschitz constant of neural networks.
result CLIP provides a tighter bound on the actual Lipschitz constant compared to layer-wise methods.
The paper studies Lipschitz bounds for integral kernels under differentiability assumptions.
problem Understanding the Lipschitz continuity of feature maps associated with integral kernels.
method Analyzes differentiability assumptions to derive explicit formulas for Lipschitz constants and conditions for non-Lipschitz continuity.
result Explicit formulas and conditions for Lipschitz continuity of feature maps associated with various kernels.
We determine the asymptotic behavior of the optimal Lipschitz constant for the systole map from Teichmuller space to the curve complex.
The paper calculates upper bounds on ReLU network Lipschitz constants.
problem Determining the maximum perturbation size for robustness of neural networks.
method Analyzing ReLU, affine-ReLU, and max pooling functions; combining results; tracking zero elements; using a computational approach.
result The method produces the largest known bounds on minimum adversarial perturbations for large networks.
In this note we prove that reconstruction from magnitudes of frame coefficients (the so called "phase retrieval problem") can be performed using Lipschitz continuous maps. Specifically we show that when the nonlinear analysis map α:H→Rm is injective, with (α(x))k=∣<x,fk>∣2, where $…
Scalable method bounds Lipschitz constant of generative models.
problem Bounding the Lipschitz constant of generative models.
method Layerwise convex approximations using zonotopes.
result Efficient and tight bounds on generative models.
Improved robustness of machine learning models with controlled Lipschitz constants.
problem Vulnerability of state-of-the-art models to adversarial attacks.
method Proposes a CLL loss that calibrates the margin and Lipschitz constant penalties, improving robustness certificates.
result Consistently outperforms other losses on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets.
Square inscribed in a curve made of two graph functions.
problem Finding inscribed squares in curves formed by graph functions.
method Analysis of spectral invariants of Jordan Floer homology under curve perturbations.
result Existence of inscribed squares in curves with specific Lipschitz constants.
ECP optimizes expensive functions without knowing Lipschitz constant.
problem Optimizing expensive, non-convex functions with unknown Lipschitz constants.
method ECP minimizes evaluations by focusing on potentially optimal regions, eliminating Lipschitz constant estimation.
result Guaranteed no-regret performance and minimax-optimal regret bounds.
Guarantees uniform convergence for square-root Lipschitz losses.
problem Uniform convergence guarantees for square-root Lipschitz losses.
method Using Rademacher complexity and square root of scalar loss function Lipschitz constant.
result Generalizes previous results and handles non-smooth loss functions.
Tight estimation of the Lipschitz constant for deep neural networks (DNNs) is useful in many applications ranging from robustness certification of classifiers to stability analysis of closed-loop systems with reinforcement learning controllers. Existing methods in the literature for estimating the Lipschitz constant su…
In this paper we consider Llarull's theorem in the foliation case and get a lower bound of the Lipschitz constant of the map M→Sn in the foliation case under the spin condition.
We show that a family of isolated complex hypersurface singularities with constant Milnor number may fail, in the strongest sense, to have constant bi-Lipschitz type. Our example is the Briac con--Speder family $X_t:=\{(x,y,z)\in\C^3 | x^5+z^{15}+y^7z+txy^6=0 \}$ of normal complex surface germs; we show the germ $(X_0,…
Push-forward models struggle to fit multimodal distributions due to high Lipschitz constants.
problem Expressivity of push-forward generative models in fitting multimodal distributions.
method Analyzing the Lipschitz constant and its relation to the total variation distance and Kullback-Leibler divergence.
result Push-forward models require high Lipschitz constants to approximate multimodal distributions, leading to a trade-off between expressivity and stability.
The paper calculates bounds on the local Lipschitz constants of neural network layers.
problem Understanding the Lipschitz constants of neural network layers for robustness analysis.
method Analytical approach to determine upper bounds on local Lipschitz constants of affine-ReLU functions.
result The method produces tighter bounds than the standard conservative bound, especially for small perturbations.
Two-layer neural networks need more neurons to be robust.
problem Understanding the robustness of two-layer neural networks and the role of overparametrization.
method Investigation of the tradeoffs between network size and robustness, using Lipschitz constant as a measure.
result A conjecture that robustness requires overparametrization, with precise bounds for different cases.
Abstract: Lipschitz homeomorphisms are deformed using Perelman's methods.
problem Deformation of Lipschitz homeomorphisms
method Lipschitz analogues of Siebenmann's and Perelman's homeomorphism theory
result Lipschitz stability theorem and gluing theorem
New method tightens Lipschitz bounds for CNNs efficiently.
problem Lipschitz regularization of Convolutional Neural Networks (CNNs).
method Using Toeplitz matrix theory, introduces a tight and computationally efficient upper bound for convolutional layers.
result Developed an algorithm to train Lipschitz regularized CNNs.
We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant---for multiple p-norms---of a feed forward neural network composed of commonly used layer types.…
In \cite{GrOrang}, Gromov asks the following question: given a nullhomotopic map f:Sm→Sn of Lipschitz constant L, how does the Lipschitz constant of an optimal nullhomotopy of f depend on L, m, and n? We establish that for fixed m and n, the answer is at worst quadratic in L. More precisely, we …
New method for efficient proximal mapping of 1-path-norm in shallow networks.
problem Efficiently handling the 1-path-norm of shallow neural networks.
method Closed-form proximal operator for efficient computation and upper bound on Lipschitz constant.
result Proximal mapping allows robust training against adversarial perturbations.
We give asymptotic bounds for the optimal Lipschitz constants for the systole map from the Teichmuller space to the curve complex. We give similar results to those known for closed surfaces in the cases when the genus is fixed or the ratio of genus and punctures is a rational number.
Maps between acute triangles with minimal stretch found and studied.
problem Finding the minimal stretch between acute triangles.
method Formula for the smallest Lipschitz constant and analysis of the metric space.
result Metric space of pairs of acute triangles with fixed area is Finsler and geodesics determined.
New law establishes robustness for neural networks with bounded weights.
problem Ensuring robustness of neural networks against adversarial attacks.
method Deriving a lower bound on Lipschitz constant for arbitrary model classes with bounded Rademacher complexity.
result Established a law of robustness for weight-bounded neural networks, requiring log(n) layers for robust fitting.
LALR adapts learning rate for faster convergence in regression and neural nets.
problem Finding optimal learning rates for faster convergence in regression and neural networks.
method Lipschitz continuity theory applied to Mean Absolute Error and Quantile loss functions.
result Adaptive learning rate policy enables up to 20x faster convergence.
We examine the impact of learning Lipschitz continuous models in the context of model-based reinforcement learning. We provide a novel bound on multi-step prediction error of Lipschitz models where we quantify the error using the Wasserstein metric. We go on to prove an error bound for the value-function estimate arisi…
Adaptive sampling improves convergence in heterogeneous distributed optimization.
problem Poor performance of classical SGD and SVRG in heterogeneous distributed settings.
method Adaptive sampling of machines with an adaptive estimate of local Lipschitz constants.
result Significantly accelerates convergence rate from maximum to average Lipschitz constant.
Efficient local Lipschitz bounds improve neural network robustness.
problem Certifying robustness of neural networks is challenging and often leads to over-regularization.
method Proposes an efficient trainable local Lipschitz upper bound by considering activation functions and weight matrices.
result Consistently outperforms state-of-the-art methods in clean and certified accuracy on various datasets.
The paper stabilizes invertible neural networks by using Gaussian mixture models.
problem Invertible neural networks can have exploding Lipschitz constants, leading to numerical errors.
method The authors use Gaussian mixture models to stabilize the latent distribution of invertible neural networks.
result Numerical simulations confirm that this modification improves sampling quality in multimodal applications.
A classical result of Milman roughly states that every Lipschitz function on Sn is almost constant on a sufficiently high-dimensional sphere Sm⊂Sn. In this paper we extend the result by proving that any Lipschitz function on a positively curved homogeneous space is almost consta…
Techniques known as Nonlinear Set Membership prediction, Kinky Inference or Lipschitz Interpolation are fast and numerically robust approaches to nonparametric machine learning that have been proposed to be utilised in the context of system identification and learning-based control. They utilise presupposed Lipschitz p…
The goal of the paper is to design sequential strategies which lead to efficient optimization of an unknown function under the only assumption that it has a finite Lipschitz constant. We first identify sufficient conditions for the consistency of generic sequential algorithms and formulate the expected minimax rate for…