Neural networks are dense among Lipschitz functions with fixed Lipschitz constant.
problem Characterizing neural network approximations to Lipschitz functions.
method Analyzing L-Lipschitz neural networks and their density in L-Lipschitz functions. result One layer neural networks are dense in the set of all L-Lipschitz functions. New MIP formulations for neural network Lipschitz constant estimation.
problem Ensuring robustness of neural networks by calculating their Lipschitz constant.
method Reformulating the neural network Lipschitz estimation problem as a Quadratically Constrained MIP (MIQCQP) problem.
result Solutions of the MIQCQP formulations provide bounds on the Lipschitz constant, with conditions for exactness.
New scalable Lipschitz bounds improve neural network robustness analysis.
problem Computing tight Lipschitz bounds for deep neural networks is challenging and computationally expensive.
method Derived new closed-form Lipschitz bounds using more general feasible points of LipSDP, avoiding SDP solvers.
result Improved scalability and precision of Lipschitz estimation for large neural networks.
GroupSort neural networks can approximate Lipschitz continuous functions.
problem Understanding and improving the expressive power of neural networks with Lipschitz constraints.
method Introduced and studied GroupSort neural networks with constraints on weights, proving their ability to approximate Lipschitz continuous functions.
result GroupSort networks can represent any Lipschitz continuous piecewise linear functions and are well-suited for approximating general Lipschitz continuous functions.
Paper investigates Lipschitz constants of self-attention modules in neural networks.
problem Lipschitz constants of self-attention modules in neural networks.
method Proved standard dot-product self-attention is not Lipschitz for unbounded input domain. Proposed L2 self-attention that is Lipschitz. Derived upper bound on L2 self-attention's Lipschitz constant.
result Proved standard self-attention is not Lipschitz for unbounded input domain and proposed an alternative L2 self-attention that is Lipschitz.
New approach to certifiably robust neural networks using Boolean function perspective.
problem Lack of principled understanding and certified robustness for ℓ∞ perturbations. method New perspective on Boolean functions, deriving impossibility results, and developing a unified Lipschitz network.
result Unified Lipschitz network that bypasses expressive power limitations and achieves better certified robustness.
The paper calculates upper bounds on ReLU network Lipschitz constants.
problem Determining the maximum perturbation size for robustness of neural networks.
method Analyzing ReLU, affine-ReLU, and max pooling functions; combining results; tracking zero elements; using a computational approach.
result The method produces the largest known bounds on minimum adversarial perturbations for large networks.
This paper bounds the Lipschitz constants of neural networks and their gradients.
problem Estimating the Lipschitz constant of complex models like neural networks.
method Local upper and lower bounds on Lipschitz constants computed with respect to network parameters.
result It is impossible to derive global upper bounds for the Lipschitz constants of neural networks.
LiPopt uses polynomial optimization to estimate neural network Lipschitz constants efficiently.
problem Estimating the Lipschitz constant of neural networks efficiently.
method Sparse polynomial optimization, leveraging network connectivity to reduce complexity.
result Superior estimates of the ℓ∞-Lipschitz constant compared to existing methods. Investigates Lipschitz continuity in neural networks across various settings.
problem Understanding the Lipschitz behavior of neural networks.
method Empirical investigation of Lipschitz bounds in different neural network architectures and datasets.
result Remarkable fidelity of the lower Lipschitz bound and a Double Descent trend in both upper and lower bounds.
This paper studies bounds for the Lipschitz constant of random neural networks.
problem Quantifying the worst-case robustness of neural networks against adversarial perturbations.
method Analyzes upper and lower bounds for the Lipschitz constant of random ReLU neural networks under specific initialization conditions.
result For deep networks, the upper bound is larger than the lower bound by a logarithmic factor in width.
This paper analyzes the Lipschitz constants of deep neural networks with random weights.
problem Estimating the Lipschitz constants of deep neural networks with random parameters.
method High probability upper and lower bounds derived for ReLU neural networks with He initialization.
result The behavior of the Lipschitz constant varies significantly between p∈[1,2) and p∈[2,∞]. This paper examines weight initialization for 1-Lipschitz networks to improve robustness against adversarial attacks.
problem Improving the robustness of deep neural networks against adversarial attacks.
method Examined weight parametrization of AOL and SLL networks, calculated weight variance bounds, and demonstrated weight decay.
result Weight initialization causes deep 1-Lipschitz networks to decay to zero, and weight variance does not affect output variance distribution.
New parameterization of neural networks with Lipschitz bounds for robustness.
problem Developing robust neural networks with Lipschitz bounds.
method Introducing a new parameterization that admits a Lipschitz bound during training without requiring projections or barrier functions.
result The new parameterization improves robustness to adversarial attacks in image classification.
New method tightens Lipschitz bounds for CNNs efficiently.
problem Lipschitz regularization of Convolutional Neural Networks (CNNs).
method Using Toeplitz matrix theory, introduces a tight and computationally efficient upper bound for convolutional layers.
result Developed an algorithm to train Lipschitz regularized CNNs.
The paper studies Lipschitz bounds for integral kernels under differentiability assumptions.
problem Understanding the Lipschitz continuity of feature maps associated with integral kernels.
method Analyzes differentiability assumptions to derive explicit formulas for Lipschitz constants and conditions for non-Lipschitz continuity.
result Explicit formulas and conditions for Lipschitz continuity of feature maps associated with various kernels.
Efficient local Lipschitz bounds improve neural network robustness.
problem Certifying robustness of neural networks is challenging and often leads to over-regularization.
method Proposes an efficient trainable local Lipschitz upper bound by considering activation functions and weight matrices.
result Consistently outperforms state-of-the-art methods in clean and certified accuracy on various datasets.
1-Lipschitz networks are as accurate as classical networks and offer robustness.
problem Misconceptions about 1-Lipschitz neural networks and their properties.
method Analysis of 1-Lipschitz neural networks' accuracy, robustness, and generalization.
result 1-Lipschitz neural networks are as accurate as classical networks and can fit arbitrarily difficult boundaries.
CLIP controls neural network stability by bounding Lipschitz constants.
problem Neural networks lack mathematical guarantees of stability, especially to adversarial examples.
method Develops a variational regularization method (CLIP) to control the Lipschitz constant of neural networks.
result CLIP provides a tighter bound on the actual Lipschitz constant compared to layer-wise methods.
The local Lipschitz constant of a neural network is a useful metric with applications in robustness, generalization, and fairness evaluation. We provide novel analytic results relating the local Lipschitz constant of nonsmooth vector-valued functions to a maximization over the norm of the generalized Jacobian. We prese…
This research explores principles of Lipschitz continuity in neural networks for robustness and generalization.
problem Ensuring robustness and generalization in neural networks, especially to small input perturbations and out-of-distribution data.
method Two complementary perspectives: internal (training dynamics) and external (frequency signal propagation).
result Advances in understanding the principles of Lipschitz continuity in neural networks.
Lipschitz constraints under L2 norm on deep neural networks are useful for provable adversarial robustness bounds, stable training, and Wasserstein distance estimation. While heuristic approaches such as the gradient penalty have seen much practical success, it is challenging to achieve similar practical performance wh…
Adaptive LR improves neural network Lipschitz regularity without slowing convergence.
problem Training neural networks with high Lipschitz constant.
method Adaptive learning rate (decay) in gradient descent for two-layer networks.
result Decaying learning rate maintains convergence rate and Lipschitz regularity.
Study shows shallow ReLU networks struggle with high-dimensional Lipschitz functions.
problem Expressing high-dimensional Lipschitz functions with shallow ReLU networks.
method Established lower bounds on shallow network complexity for polynomial approximation.
result Shallow ReLU networks suffer from the curse of dimensionality for Lipschitz functions.
Several recent papers have discussed utilizing Lipschitz constants to limit the susceptibility of neural networks to adversarial examples. We analyze recently proposed methods for computing the Lipschitz constant. We show that the Lipschitz constant may indeed enable adversarially robust neural networks. However, the m…
Develops risk measures on Lipschitz spaces for financial positions.
problem Lack of standard cash-additive methods in Lipschitz spaces.
method Proposes Lipschitz-free space, uses additivity along benchmark-deviation instruments.
result Derives dual representations for convex and coherent risk measures.
Scalable method bounds Lipschitz constant of generative models.
problem Bounding the Lipschitz constant of generative models.
method Layerwise convex approximations using zonotopes.
result Efficient and tight bounds on generative models.
The paper proposes a method to train NNs with a small Lipschitz constant to improve robustness.
problem Neural networks' susceptibility to adversarial perturbations in safety-critical applications.
method The paper introduces a framework to train multi-layer NNs by minimizing their Lipschitz constant, using an optimization scheme based on the Alternating Direction Method of Multipliers.
result The proposed training procedure successfully increases the robustness of neural networks.
Lipschitz normalization boosts deep attention models, especially for graph neural networks.
problem Gradient explosion in deep graph attention networks leads to poor performance.
method Enforcing Lipschitz continuity by normalizing attention scores.
result Deep GAT models with LipschitzNorm achieve state-of-the-art results for tasks with long-range dependencies.
Researchers develop a new framework to control neural network sensitivity.
problem Understanding and controlling the behavior of neural networks.
method Direct parameterization of bi-Lipschitzness in convex neural networks.
result A clear and tight control of neural network sensitivity achieved.
Lipschitz RNNs improve stability and performance in various tasks.
problem Improving stability and performance of RNNs.
method Introduced a Lipschitz recurrent unit with a linear and Lipschitz nonlinear component for stability analysis.
result Lipschitz RNNs outperform existing units on benchmark tasks.
We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant---for multiple p-norms---of a feed forward neural network composed of commonly used layer types.…
We examine the impact of learning Lipschitz continuous models in the context of model-based reinforcement learning. We provide a novel bound on multi-step prediction error of Lipschitz models where we quantify the error using the Wasserstein metric. We go on to prove an error bound for the value-function estimate arisi…
We model how Lipschitz continuity changes during neural network training.
problem Understanding how Lipschitz continuity evolves during training.
method We use a system of stochastic differential equations to capture the dynamics of Lipschitz continuity under SGD.
result We identify three factors driving the evolution of Lipschitz continuity: gradient flow projection, gradient noise, and Hessian projection.
GD-trained shallow ReLU nets learn Lipschitz functions with noise.
problem Learning Lipschitz functions with additive noise in overparameterized neural networks.
method Gradient Descent (GD) with early stopping, focusing on the Neural Tangent Kernel (NTK).
result Early-stopped GD achieves minimax optimal rates for learning Lipschitz functions.
Improved robustness of 1D CNNs for heart arrhythmia classification.
problem Improving the robustness of 1D CNNs for classification tasks.
method Parameterization using Cayley transform and controllability Gramian for Lipschitz-bounded CNNs.
result Improved robustness of trained Lipschitz-bounded 1D CNNs for heart arrhythmia classification.
Two-layer neural networks need more neurons to be robust.
problem Understanding the robustness of two-layer neural networks and the role of overparametrization.
method Investigation of the tradeoffs between network size and robustness, using Lipschitz constant as a measure.
result A conjecture that robustness requires overparametrization, with precise bounds for different cases.
Paper introduces a neural network for consistent estimation of optimal transport maps.
problem Statistically consistent estimation of optimal transport maps between probability distributions.
method Lipschitz-constrained GAN penalized by quadratic transportation cost.
result The generator converges uniformly to the optimal transport map as sample size increases.
The paper calculates bounds on the local Lipschitz constants of neural network layers.
problem Understanding the Lipschitz constants of neural network layers for robustness analysis.
method Analytical approach to determine upper bounds on local Lipschitz constants of affine-ReLU functions.
result The method produces tighter bounds than the standard conservative bound, especially for small perturbations.
New method for efficient proximal mapping of 1-path-norm in shallow networks.
problem Efficiently handling the 1-path-norm of shallow neural networks.
method Closed-form proximal operator for efficient computation and upper bound on Lipschitz constant.
result Proximal mapping allows robust training against adversarial perturbations.
We investigate robustness of deep feed-forward neural networks when input data are subject to random uncertainties. More specifically, we consider regularization of the network by its Lipschitz constant and emphasize its role. We highlight the fact that this regularization is not only a way to control the magnitude of …
LiST improves neural network robustness and calibration without manual tuning.
problem Developing robust and calibrated neural networks simultaneously.
method Lipschitz Scaling Training (LiST) that iteratively adjusts the global Lipschitz constant.
result LiST yields an out-of-the-box calibrated network with competitive accuracy and robustness.
New Lipschitz bound for ReLU networks resists weight rescaling.
problem Lack of robustness guarantees for ReLU networks under weight perturbations.
method Rescaling-invariant Lipschitz bound based on path-metrics.
result The new bound applies to various ReLU-DAG architectures and resists neuron-wise rescalings.
The paper uses neural networks to learn system dynamics from data with Lipschitz regularization.
problem Learning governing equations from time-sampled data.
method Lipschitz regularized deep neural networks for ODE system identification.
result Lipschitz regularization improves the smoothness and generalization of the learned function.
Training neural networks under a strict Lipschitz constraint is useful for provable adversarial robustness, generalization bounds, interpretable gradients, and Wasserstein distance estimation. By the composition property of Lipschitz functions, it suffices to ensure that each individual affine transformation or nonline…
Study identifies three quantization regimes for ReLU networks.
problem Approximation of Lipschitz functions by ReLU networks with finite-precision weights.
method Established through nonasymptotic tight lower and upper bounds on minimax approximation error.
result Memory-optimality achieved in proper quantization regime for deep networks.
Adversarial attacks against machine learning models are a rather hefty obstacle to our increasing reliance on these models. Due to this, provably robust (certified) machine learning models are a major topic of interest. Lipschitz continuous models present a promising approach to solving this problem. By leveraging the …
Study shows prior Lipschitz continuity can improve adversarial robustness of Bayesian Neural Networks.
problem Improving adversarial robustness of Bayesian Neural Networks.
method Analysis of i.i.d., zero-mean Gaussian priors and posteriors approximated via mean-field variational inference.
result Adversarial robustness is sensitive to the prior variance.