We introduce a variational framework to learn the activation functions of deep neural networks. Our aim is to increase the capacity of the network while controlling an upper-bound of the actual Lipschitz constant of the input-output relation. To that end, we first establish a global bound for the Lipschitz constant of …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Lipschitz activation”
Training neural networks under a strict Lipschitz constraint is useful for provable adversarial robustness, generalization bounds, interpretable gradients, and Wasserstein distance estimation. By the composition property of Lipschitz functions, it suffices to ensure that each individual affine transformation or nonline…
Efficient local Lipschitz bounds improve neural network robustness.
problem Certifying robustness of neural networks is challenging and often leads to over-regularization.
method Proposes an efficient trainable local Lipschitz upper bound by considering activation functions and weight matrices.
result Consistently outperforms state-of-the-art methods in clean and certified accuracy on various datasets.
New MIP formulations for neural network Lipschitz constant estimation.
problem Ensuring robustness of neural networks by calculating their Lipschitz constant.
method Reformulating the neural network Lipschitz estimation problem as a Quadratically Constrained MIP (MIQCQP) problem.
result Solutions of the MIQCQP formulations provide bounds on the Lipschitz constant, with conditions for exactness.
Tight estimation of the Lipschitz constant for deep neural networks (DNNs) is useful in many applications ranging from robustness certification of classifiers to stability analysis of closed-loop systems with reinforcement learning controllers. Existing methods in the literature for estimating the Lipschitz constant su…
RVFL networks can efficiently approximate Lipschitz functions in L∞ norm.
problem Efficiently approximating Lipschitz continuous functions in L∞ norm.
method Random Vector Functional Link (RVFL) network with ReLU activation functions, proving approximation in L∞ norm.
result An RVFL with ReLU activation functions can approximate Lipschitz continuous functions in L∞ norm.
Two-layer neural networks need more neurons to be robust.
problem Understanding the robustness of two-layer neural networks and the role of overparametrization.
method Investigation of the tradeoffs between network size and robustness, using Lipschitz constant as a measure.
result A conjecture that robustness requires overparametrization, with precise bounds for different cases.
Improved generalization bounds for CNNs using Rademacher complexity.
problem Establishing non-vacuous generalization bounds for deep learning models.
method Rademacher complexity framework with novel contraction lemmas for high-dimensional mappings.
result Enhanced generalization bounds for a broader class of activation functions.
New parameterization of neural networks with Lipschitz bounds for robustness.
problem Developing robust neural networks with Lipschitz bounds.
method Introducing a new parameterization that admits a Lipschitz bound during training without requiring projections or barrier functions.
result The new parameterization improves robustness to adversarial attacks in image classification.
Sparse neural networks can match dense models on Lipschitz functions.
problem Sparse networks are more efficient but lack theoretical guarantees.
method Formal model of sparse networks, LSH-based routing function, Lipschitz function approximation.
result Sparse networks can approximate dense networks on Lipschitz functions.
We study the robustness of active learning (AL) algorithms against prior misspecification: whether an algorithm achieves similar performance using a perturbed prior as compared to using the true prior. In both the average and worst cases of the maximum coverage setting, we prove that all -approximate algorithms are …
The paper calculates upper bounds on ReLU network Lipschitz constants.
problem Determining the maximum perturbation size for robustness of neural networks.
method Analyzing ReLU, affine-ReLU, and max pooling functions; combining results; tracking zero elements; using a computational approach.
result The method produces the largest known bounds on minimum adversarial perturbations for large networks.
This paper presents a novel nonmyopic adaptive Gaussian process planning (GPP) framework endowed with a general class of Lipschitz continuous reward functions that can unify some active learning/sensing and Bayesian optimization criteria and offer practitioners some flexibility to specify their desired choices for defi…
Adversarial attacks against machine learning models are a rather hefty obstacle to our increasing reliance on these models. Due to this, provably robust (certified) machine learning models are a major topic of interest. Lipschitz continuous models present a promising approach to solving this problem. By leveraging the …
Wide deep neural networks with Gaussian weights approximate Gaussian processes closely.
problem Understanding the approximation of deep neural networks with Gaussian weights to Gaussian processes.
method Established novel rates for the Gaussian approximation of random deep neural networks with Gaussian parameters and Lipschitz activation functions in the wide limit.
result The distance between the network output and the Gaussian approximation scales inversely with the width of the network.
The paper studies Lipschitz bounds for integral kernels under differentiability assumptions.
problem Understanding the Lipschitz continuity of feature maps associated with integral kernels.
method Analyzes differentiability assumptions to derive explicit formulas for Lipschitz constants and conditions for non-Lipschitz continuity.
result Explicit formulas and conditions for Lipschitz continuity of feature maps associated with various kernels.
New approach to certifiably robust neural networks using Boolean function perspective.
problem Lack of principled understanding and certified robustness for perturbations.
method New perspective on Boolean functions, deriving impossibility results, and developing a unified Lipschitz network.
result Unified Lipschitz network that bypasses expressive power limitations and achieves better certified robustness.
The curse of dimensionality affects neural network optimization, especially with smooth functions.
problem The curse of dimensionality in neural network optimization.
method Examined through the evolution of the parameter distribution under 2-Wasserstein gradient flow.
result The curse of dimensionality persists in neural network optimization, even with smooth functions.
Shuffling the Stochastic Mirror Descent via Dual Lipschitz Continuity and Kernel Conditioningmath.OC
New method improves optimization algorithms without Lipschitz smoothness.
problem Improving optimization algorithms in the absence of Lipschitz smoothness.
method Dual kernel conditioning (DKC) to provide dual Lipschitz continuity.
result First complexity bounds and iterate convergence for random reshuffling mirror descent.
Paper defines topology automaton for Barański carpets and proves Hölder equivalence conditions.
problem Tackles Hölder equivalence of Barański carpets.
method Defines topology automaton and applies method from previous studies.
result Obtains sufficient condition for Hölder equivalence of Barański carpets.
Deep neural networks (DNNs) have emerged as a popular mathematical tool for function approximation due to their capability of modelling highly nonlinear functions. Their applications range from image classification and natural language processing to learning-based control. Despite their empirical successes, there is st…
GD-trained shallow ReLU nets learn Lipschitz functions with noise.
problem Learning Lipschitz functions with additive noise in overparameterized neural networks.
method Gradient Descent (GD) with early stopping, focusing on the Neural Tangent Kernel (NTK).
result Early-stopped GD achieves minimax optimal rates for learning Lipschitz functions.
Improved robustness for deep neural networks with tighter bounds and attacks.
problem Loose upper bounds and prohibitive computation in existing adversarial robustness methods.
method Primal approach with exact Lipschitz certificates for ReLU networks and modern architectures, and novel Wasserstein Distributional Attacks.
result Tighter upper bounds and greater flexibility in attack points compared to existing methods.
New method for efficient proximal mapping of 1-path-norm in shallow networks.
problem Efficiently handling the 1-path-norm of shallow neural networks.
method Closed-form proximal operator for efficient computation and upper bound on Lipschitz constant.
result Proximal mapping allows robust training against adversarial perturbations.
Adaptive LR improves neural network Lipschitz regularity without slowing convergence.
problem Training neural networks with high Lipschitz constant.
method Adaptive learning rate (decay) in gradient descent for two-layer networks.
result Decaying learning rate maintains convergence rate and Lipschitz regularity.
i-DenseNets improve parameter efficiency and performance in density estimation.
problem Improving parameter efficiency and performance in density estimation models.
method Invertible Dense Networks (i-DenseNets) with learnable weighted concatenation and Concatenated LipSwish activation function.
result i-DenseNets outperform Residual Flows and other flow-based models in bits per dimension.
The paper calculates bounds on the local Lipschitz constants of neural network layers.
problem Understanding the Lipschitz constants of neural network layers for robustness analysis.
method Analytical approach to determine upper bounds on local Lipschitz constants of affine-ReLU functions.
result The method produces tighter bounds than the standard conservative bound, especially for small perturbations.
Kolmogorov-Arnold Networks offer improved interpretability and parsimony in science tasks.
problem Improving interpretability and parsimony in science-oriented tasks.
method Theoretical analysis of Kolmogorov-Arnold Networks (KAN) with generalization bounds and model complexity.
result Generalization bounds for KAN with various activation functions, scaling with the norm of coefficient matrices and Lipschitz constants.
Bounds on Gaussian approximation for neural networks with novel smoothing techniques.
problem Approximating the distribution of wide random neural networks.
method Stein's method, Gaussian smoothing, Laplacian operators, Cameron-Martin space.
result First bounds on Gaussian approximation of wide random neural networks.
We show that the log-likelihood of several probabilistic graphical models is Lipschitz continuous with respect to the lp-norm of the parameters. We discuss several implications of Lipschitz parametrization. We present an upper bound of the Kullback-Leibler divergence that allows understanding methods that penalize the …
Revisits shallow neural networks using Lipschitz norms and measures.
problem Existence and compactness of minimizers in neural network formulations.
method Mean field parametrization, signed measures, duality pairings, Kantorovich-Rubinstein norms.
result Compactness results and uniform large data limits for empirical risk minimization.
GraN-GAN normalizes gradients for better GAN performance.
problem Improving image generation in GANs with piecewise linear discriminators.
method Piecewise Gradient Normalization (GraN) for input-dependent normalization.
result Significant performance gains in image generation across various datasets.
We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machin…
Two-layer neural networks must be robust, even with arbitrary weights.
problem Proving the robustness of two-layer neural networks with arbitrary weights.
method Developed a new function-space covering method to prove the robustness law, replacing parameter-space covering.
result Proved the conjectured law for two-layer networks with arbitrary real weights, biases, and affine skip connections.
New method learns SIMs with arbitrary monotone activations without strong distributional assumptions.
problem Learning Single-Index Models with arbitrary monotone activations.
method Based on omniprediction with calibrated multiaccuracy and Bregman divergences.
result First agnostic learning result for SIMs with arbitrary monotone activations.
We show that any smooth bi-Lipschitz can be represented exactly as a composition of functions that are close to the identity in the sense that each is Lipschitz, and the Lipschitz constant decreases inversely with the number of functions com…
Effective regularisation of neural networks is essential to combat overfitting due to the large number of parameters involved. We present an empirical analogue to the Lipschitz constant of a feed-forward neural network, which we refer to as the maximum gain. We hypothesise that constraining the gain of a network will h…
Mixtures of neural operators reduce active complexity in operator learning.
problem Reduction of active complexity in operator learning models.
method Constructive comparison between routed mixtures of neural operators (MoNOs) and a fixed single-neural-operator construction.
result Every scalar uniformly continuous nonlinear operator can be approximated by a MoNO whose active expert has smaller depth, width, and rank scaling.
Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weightsstat.ML
Study Gaussian approximation for deep neural networks with random weights.
problem Understanding the distribution of deep neural networks with random weights.
method Established Gaussian approximation bounds in Wasserstein-1 norm.
result Convergence rates of order for deep networks with proportional layer widths.
The scope of research in the domain of activation functions remains limited and centered around improving the ease of optimization or generalization quality of neural networks (NNs). However, to develop a deeper understanding of deep learning, it becomes important to look at the non linear component of NNs more careful…
Although for neural networks with locally Lipschitz continuous activation functions the classical derivative exists almost everywhere, the standard chain rule is in general not applicable. We will consider a way of introducing a derivative for neural networks that admits a chain rule, which is both rigorous and easy to…
The Jacobian matrix (or the gradient for single-output networks) is directly related to many important properties of neural networks, such as the function landscape, stationary points, (local) Lipschitz constants and robustness to adversarial attacks. In this paper, we propose a recursive algorithm, RecurJac, to comput…
Flow-based generative models parameterize probability distributions through an invertible transformation and can be trained by maximum likelihood. Invertible residual networks provide a flexible family of transformations where only Lipschitz conditions rather than strict architectural constraints are needed for enforci…
A new method reduces computational cost for gene expression inference in large microarray data sets.
problem Efficiently predicting gene expression in large datasets with limited resources.
method Adaptive Lipschitz constant inspired learning rate, random sub-sampling, and A-ReLU activation function.
result Remarkable improvement in saving computational cost while maintaining prediction accuracy.
Study shows infoGAN's generalization error bound for two-layer networks.
problem Understanding generalization error in infoGAN for two-layer neural networks.
method Analyzes the difference between empirical and population objective functions, derives Rademacher complexity bounds.
result Derives error bound for infoGAN's generalization error in a two-layer network.
This paper analyzes challenges and solutions in deep learning optimization.
problem Gradient vanishing and exploding issues in deep learning.
method Improvement of gradient flow and constraints on Lipschitz constant.
result Enhanced understanding of Jacobian matrices and Lipschitz constants in deep learning modules.
We study the sample complexity of learning one-hidden-layer convolutional neural networks (CNNs) with non-overlapping filters. We propose a novel algorithm called approximate gradient descent for training CNNs, and show that, with high probability, the proposed algorithm with random initialization grants a linear conve…
CGNNs use wavelets for continuous function generation in infinite-dimensional spaces.
problem Generating continuous functions in infinite-dimensional spaces for applications like inverse problems.
method Inspired by DCGAN, CGNNs use wavelet multiresolution analysis with convolutional and nonlinear layers.
result CGNNs can be injective under certain conditions on filters and nonlinearity, leading to Lipschitz stability estimates.