BLiE optimizes hyperparameters with theoretical guarantees and superior performance.
problem Hyperparameter optimization in machine learning.
method Lipschitz bandit approach exploiting Lipschitz continuity.
result BLiE finds ε-optimal hyperparameters with theoretical complexity.
Adaptive online learning algorithms without manual tuning of a Lipschitz hyperparameter.
problem Designing adaptive online learning algorithms that require minimal user input and automatically adjust hyperparameters.
method Developing new versions of MetaGrad and Squint algorithms that dynamically update learning rates and adapt to the optimal Lipschitz hyperparameter.
result Automatic adaptation of the Lipschitz hyperparameter, improving performance and efficiency of online learning algorithms.
Paper presents adaptive Lipschitz bandit framework for efficient optimization.
problem Optimizing rewards and minimizing regret in stochastic Lipschitz bandit problems.
method Adaptive learning of partitions in context- and arm-space using hierarchical Bayesian models.
result Achieves state-of-the-art performance in real-world tasks like neural network hyperparameter tuning.
HALO uses local Lipschitz constants to optimize functions efficiently.
problem Efficiently solving global optimization problems with complex objective functions.
method Hybrid Adaptive Lipschizian Optimization (HALO) algorithm that estimates local Lipschitz constants and balances global and local information.
result HALO outperforms other global optimization algorithms on numerous test functions.
We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant---for multiple p-norms---of a feed forward neural network composed of commonly used layer types.…
Improved variational inequality algorithms using adaptive step sizes.
problem Solving monotone variational inequalities and convex-concave min-max problems efficiently.
method Adaptive step sizes that eliminate hyperparameters and global Lipschitz continuity requirements.
result Eliminated the need for the golden ratio in the algorithm and improved complexity bounds.
Proves bounds on deep CNN generalization error.
problem Understanding how well deep CNNs perform on unseen data.
method Bounds on generalization error based on training loss, parameters, Lipschitz constant, and weight distance.
result Bounds are independent of input size and feature map dimensions.
ECPv2 optimizes Lipschitz functions efficiently and scalably.
problem Global optimization of Lipschitz-continuous functions with unknown Lipschitz constants.
method Adapting the Every Call is Precious (ECP) framework, ECPv2 introduces adaptive lower bounds, Worst-m memory, and random projections to reduce computational cost and improve acceptance regions.
result ECPv2 retains ECP's no-regret guarantees with optimal finite-time bounds and expands the acceptance region with high probability.
Meta-learning improves performance across similar tasks in adversarial bandit settings.
problem Improving performance across multiple similar tasks in adversarial bandit scenarios.
method Designing meta-algorithms that combine outer learners to tune hyperparameters of inner learners for MAB and BLO.
result Meta-algorithms improve task-averaged regret for MAB and BLO, showing direct relationship with action space-dependent measures.
Introduces gradient decay in Softmax for better generalization.
problem Improving generalization performance in neural networks.
method Gradient decay hyperparameter in Softmax for varying gradient rates based on probability.
result Gradient decay rate affects generalization performance and can be tuned for better optimization.
Paper optimizes diffusion models for denoising tasks with theoretical guarantees.
problem Lack of theoretical understanding of MSE optimality in diffusion models.
method Inspired by MSE-optimal CME, proposes a novel denoising strategy for diffusion models.
result Demonstrates polynomial-time convergence to the CME under mild conditions.
A new strategy selects k in k-NN regression without hold-out data.
problem Choosing optimal k in k-NN regression without hold-out data.
method Iterative procedure over k, minimum discrepancy principle.
result Minimax-optimal over smoothness function classes.
Anderson acceleration (or Anderson mixing) is an efficient acceleration method for fixed point iterations xt+1=G(xt), e.g., gradient descent can be viewed as iteratively applying the operation G(x)≜x−α∇f(x). It is known that Anderson acceleration is quite efficient in practice and can be viewed…
Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often du…
Paper tackles online learning with interval regret, achieving adaptive bounds.
problem Non-stationary online learning over time intervals.
method Two-layer online ensemble structure with gradient variation.
result Achieves strong theoretical guarantees with adaptive bounds.
Neural networks are dense among Lipschitz functions with fixed Lipschitz constant.
problem Characterizing neural network approximations to Lipschitz functions.
method Analyzing L-Lipschitz neural networks and their density in L-Lipschitz functions. result One layer neural networks are dense in the set of all L-Lipschitz functions. Constructs a map with prescribed local Lipschitz constants on a subset of a manifold.
problem Creating a Lipschitz map with specific local Lipschitz constants on a subset of a manifold.
method Constructs a Lipschitz map that matches a given map on a subset and has a local Lipschitz constant defined by a continuous function.
result A Lipschitz map can be constructed with a local Lipschitz constant prescribed by a continuous function.
Maps between certain Lipschitz manifolds are isometries if they preserve volume.
problem Volume preservation and isometry conditions for Lipschitz manifolds.
method Volume-preserving 1-Lipschitz maps from integral currents onto infinitesimally Euclidean Lipschitz manifolds.
result Volume-preserving maps are isometries under given conditions.
New algorithms optimize without tuning, matching tuned SGD performance.
problem Optimizing machine learning models without manual hyperparameter tuning.
method Formalizes tuning-free algorithms for matching SGD performance with loose hints.
result Tuning-free algorithms can match SGD performance, but not optimal convergence rates.
This paper analyzes convergence of RMSProp and Adam in non-convex optimization with tight complexity bounds.
problem Analyzing convergence of RMSProp and Adam in non-convex optimization with relaxed assumptions.
method Developed new convergence analyses for RMSProp and Adam, considering adaptive learning rates and affine noise variance.
result RMSProp and Adam converge to ε-stationary points with iteration complexities of O(ε^(-4)) under proper hyperparameters.
Abstract: Lipschitz homeomorphisms are deformed using Perelman's methods.
problem Deformation of Lipschitz homeomorphisms
method Lipschitz analogues of Siebenmann's and Perelman's homeomorphism theory
result Lipschitz stability theorem and gluing theorem
Self-Tuning Networks optimize hyperparameters using bilevel optimization and gated best-response functions.
problem Optimizing hyperparameters for neural networks.
method Bilevel optimization with gated best-response functions to adapt regularization hyperparameters online.
result Self-Tuning Networks outperform fixed hyperparameter values on large-scale deep learning problems.
We compute the local Lipschitz constant of ReLU networks precisely.
problem Estimating the local Lipschitz constant of ReLU networks is hard.
method We use a novel approach involving the generalized Jacobian and backpropagation.
result We provide an algorithm to compute the exact Lipschitz constant of ReLU networks.
Study shows infinite distinct outer metric Lipschitz classes for knots in S3.
problem Lipschitz classification of surface singularities in R4. method Outer metric Lipschitz equivalence and topological equivalence of knots.
result Infinitely many distinct outer metric Lipschitz classes for knots in S3. The study examines the limitations of bi-Lipschitz Normalizing Flows in approximating certain distributions.
problem The expressivity of bi-Lipschitz Normalizing Flows in approximating specific target distributions.
method Characterization of expressivity through lower bounds on Total Variation distance and discussion of potential remedies.
result Several target distributions are difficult to approximate using bi-Lipschitz Normalizing Flows, and lower bounds on their approximation are provided.
The paper analyzes Random Search and introduces BLiN-MOS for bandit learning.
problem Understanding and optimizing hyperparameter tuning in metric measure spaces.
method Introducing scattering dimension to quantify performance, and developing BLiN-MOS for bandit learning.
result Random Search converges to optimal values with specific rates in noise-free and noisy environments.
Novikov conjecture reduced to Lipschitz cohomology of groups.
problem Novikov higher signature conjecture for groups.
method Introducing Lipschitz cohomology classes and reducing the conjecture.
result Reduced Novikov conjecture to Lipschitz cohomology.
We examine the impact of learning Lipschitz continuous models in the context of model-based reinforcement learning. We provide a novel bound on multi-step prediction error of Lipschitz models where we quantify the error using the Wasserstein metric. We go on to prove an error bound for the value-function estimate arisi…
New MIP formulations for neural network Lipschitz constant estimation.
problem Ensuring robustness of neural networks by calculating their Lipschitz constant.
method Reformulating the neural network Lipschitz estimation problem as a Quadratically Constrained MIP (MIQCQP) problem.
result Solutions of the MIQCQP formulations provide bounds on the Lipschitz constant, with conditions for exactness.
Adversarial Lipschitz Regularization improves Wasserstein GANs without gradient norm penalties.
problem Training stability and sample quality issues in Wasserstein GANs.
method Explicit Lipschitz penalty using adversarial training.
result Explicit Lipschitz penalty leads to competitive performance in Wasserstein GANs.
Lipschitz-volume rigidity holds for smooth manifolds but fails for singular spaces.
problem Lipschitz-volume rigidity on singular spaces with lower curvature bounds.
method Survey of Lipschitz-volume rigidity theorems on singular spaces.
result Lipschitz-volume rigidity doesn't hold for all singular spaces.
New method for differentially private optimization with general Lipschitz conditions.
problem Differentially private optimization under general Lipschitz conditions.
method Generalized Lipschitz condition for per-sample gradients, tuning clip norm based on minimum per-sample Lipschitz constant.
result Efficacy of the recommended clip norm tuning method verified on 8 datasets.
The paper studies Lipschitz bounds for integral kernels under differentiability assumptions.
problem Understanding the Lipschitz continuity of feature maps associated with integral kernels.
method Analyzes differentiability assumptions to derive explicit formulas for Lipschitz constants and conditions for non-Lipschitz continuity.
result Explicit formulas and conditions for Lipschitz continuity of feature maps associated with various kernels.
New scalable Lipschitz bounds improve neural network robustness analysis.
problem Computing tight Lipschitz bounds for deep neural networks is challenging and computationally expensive.
method Derived new closed-form Lipschitz bounds using more general feasible points of LipSDP, avoiding SDP solvers.
result Improved scalability and precision of Lipschitz estimation for large neural networks.
The paper studies properties of intrinsically Lipschitz constants in metric spaces.
problem Investigating properties of intrinsically Lipschitz constants.
method Introduced Leibniz and product formulas for intrinsic slope.
result Formulated Leibniz and product formulas for intrinsic slope.
We characterize locally Lipschitz mappings and existence of Lipschitz extensions through a first order nonlinear system of PDEs. We extend this study to graded group-valued Lipschitz mappings defined on compact Riemannian manifolds. Through a simple application, we emphasize the connection between these PDEs and the Ru…
Geodesically complete spaces with curvature bounded above have maps with finite energy that are Lipschitz.
problem Analyzing the properties of geodesically complete spaces with curvature constraints.
method Geometric perturbations of geodesics to curves with zero length on singular sets.
result Every Sobolev map in W1,∞ space has a Lipschitz representative with the same Lipschitz constant as its infinity energy. We construct a smooth compact n-dimensional manifold Y with one point singularity such that all its Lipschitz homotopy groups are trivial, but Lipschitz mappings Lip(S^n,Y) are not dense in the Sobolev space W^{1,n}(S^n,Y). On the other hand we show that if a metric space Y is Lipschitz (n-1)-connected, then Lipschitz …
Paper investigates Lipschitz constants of self-attention modules in neural networks.
problem Lipschitz constants of self-attention modules in neural networks.
method Proved standard dot-product self-attention is not Lipschitz for unbounded input domain. Proposed L2 self-attention that is Lipschitz. Derived upper bound on L2 self-attention's Lipschitz constant.
result Proved standard self-attention is not Lipschitz for unbounded input domain and proposed an alternative L2 self-attention that is Lipschitz.
Proves rigidity for maps between manifolds using degree theory and current developments.
problem Lipschitz-volume rigidity for maps between metric manifolds and Riemannian manifolds.
method Degree theory and recent developments of Lipschitz-volume rigidity for integral currents.
result Proves a Lipschitz-volume rigidity result for 1-Lipschitz maps.
Bayesian optimization finds best hyperparameters for deep learning models.
problem Finding optimal hyperparameters in high-dimensional space.
method Bayesian optimization, Gaussian processes, and adaptive experimentation.
result Ax, BoTorch, and GPyTorch provide a powerful and simple framework for hyperparameter optimization.
CCE improves anomaly detection metrics by measuring both confidence and consistency.
problem Existing anomaly detection metrics lack discriminative power, hyperparameter dependency, and robustness to perturbations.
method CCE uses Bayesian estimation to quantify uncertainty and constructs global and event-level confidence and consistency scores.
result CCE demonstrates strict boundedness, robustness, and linear time complexity.
The thesis defines and proves invariants for manifolds of bounded geometry.
problem Lipschitz-homotopy invariants for manifolds of bounded geometry.
method Definition of a controvariant functor and invariance of the Roe index and ρ-class.
result Lipschitz-homotopy invariants are defined and proven for manifolds of bounded geometry.
Neural networks with low Lipschitz constants can approximate any Lipschitz function.
problem Adversarial attacks on machine learning models.
method Three-layer neural networks with FullSort activation function.
result Three-layer neural networks using FullSort are Universal Lipschitz Approximators.
Defines a new modulus for Lipschitz surfaces and proves a homological duality theorem.
problem Lipschitz homology classes and their moduli.
method Defining a new modulus dModp and proving a homological duality theorem. result Every relative Lipschitz k-homology class has a unique dual class satisfying a specific modulus product equality. A new method uses reinforcement learning for hyperparameter optimization.
problem Optimizing hyperparameters in machine learning models.
method Modeling hyperparameter optimization as a sequential decision problem and using reinforcement learning.
result The method outperforms state-of-the-art approaches for hyperparameter learning.
Bi-Lipschitz mappings can embed certain algebraic sets into high-dimensional spaces.
problem Embedding algebraic sets into high-dimensional spaces while preserving distances.
method Developed a bi-Lipschitz embedding for semialgebraic sets into Rn. result Embedding is possible for n≥2k+1 and unique for n≥2k+2. Investigates Lipschitz continuity in neural networks across various settings.
problem Understanding the Lipschitz behavior of neural networks.
method Empirical investigation of Lipschitz bounds in different neural network architectures and datasets.
result Remarkable fidelity of the lower Lipschitz bound and a Double Descent trend in both upper and lower bounds.