Paper proposes a new method to stabilize noisy gradient algorithms.
problem Stochastic-gradient Langevin algorithms can introduce bias when taming denominators depend on stochastic-gradient realizations.
method Proposes a structure-preserving framework for designing tamed denominators that avoid unnecessary taming and maintain the stabilizing effect of taming.
result The method avoids stationary bias and explains the stationary error split into bias and remaining error.
A new tamed stochastic gradient Hamiltonian Monte Carlo algorithm for superlinearly growing stochastic gradients.
problem Sampling and stochastic optimization problems with superlinearly growing stochastic gradients.
method Tamed Stochastic Gradient Hamiltonian Monte Carlo (tSGHMC) algorithm.
result Established a non-asymptotic error bound in Wasserstein-2 distance with a convergence rate of 1/4. RELTA-SGLD stabilizes nonconvex SGLD updates with a lighter taming scheme.
problem Stabilizing superlinear stochastic-gradient updates in nonconvex optimization.
method Threshold-based taming with relative-growth principle for stability.
result Polynomial moment stability and first-order stationary accuracy in nonconvex SGLD.
Deep learning models viewed through tame geometry for convergence guarantees.
problem Understanding convergence guarantees in deep learning models.
method Introducing tame geometry concepts and tools for nonsmooth nonconvex settings.
result Illustrates tame geometry as a natural framework for AI systems, especially deep learning.
New algorithm TUSLA improves learning of non-convex neural networks.
problem Optimizing non-convex loss functions in neural networks with superlinear gradient growth.
method Tamed Unadjusted Stochastic Langevin Algorithm (TUSLA) based on SGLD with taming technology.
result Finite-time guarantees for TUSLA to find approximate minimizers of empirical and population risks.
New algorithm tames non-linear growth in stochastic optimization.
problem Computational challenges in E-step of EM framework.
method Employing interacting particle systems and taming techniques to create tIPLA.
result Non-asymptotic convergence error estimates in Wasserstein-2 distance for tIPLA.
Designing deterministic denominators for SGLD stabilizes large drifts.
problem Stabilizing large drifts in SGLD
method Using state-dependent envelopes and empirical quantiles for activation thresholds
result Proxy-quantile denominators are close to oracle-score behavior and improve deterministic taming choices
In this paper, we propose a new adaptive stochastic gradient Langevin dynamics (ASGLD) algorithmic framework and its two specialized versions, namely adaptive stochastic gradient (ASG) and adaptive gradient Langevin dynamics(AGLD), for non-convex optimization problems. All proposed algorithms can escape from saddle poi…
The tame flows are ``nice'' flows on ``nice'' spaces. The nice (tame) sets are the pfaffian sets introduced by Khovanski, and a flow Φ:R×X→X on pfaffian set X is tame if the graph of Φ is a pfaffian subset of R×X×X. Any compact tame set admits plenty tame flows. We prove …
Understanding the convergence performance of asynchronous stochastic gradient descent method (Async-SGD) has received increasing attention in recent years due to their foundational role in machine learning. To date, however, most of the existing works are restricted to either bounded gradient delays or convex settings.…
This paper shows how path spaces on two-level manifolds can be Hilbert manifold structures.
problem Addressing the structure of path spaces on two-level manifolds.
method Introducing the notion of tameness and constructing charts on path spaces of two-level manifolds.
result Path spaces on tame two-level manifolds have the structure of a Hilbert manifold.
Although distributed computing can significantly reduce the training time of deep neural networks, scaling the training process while maintaining high efficiency and final accuracy is challenging. Distributed asynchronous training enjoys near-linear speedup, but asynchrony causes gradient staleness - the main difficult…
New algorithm samples superlinearly growing log-gradient distributions.
problem Sampling from distributions with superlinearly growing log-gradient.
method Proposes a novel taming Langevin-based scheme called sTULA.
result Derives non-asymptotic convergence bounds in KL, TV, and W2 distances.
TUSLA algorithm solves non-convex optimization problems with ReLU activations.
problem Non-convex stochastic optimization with super-linearly growing and discontinuous gradients.
method Non-asymptotic analysis of TUSLA algorithm for non-convex learning.
result TUSLA provides non-asymptotic error bounds in Wasserstein distances for non-convex learning.
We introduce a new second-order inertial optimization method for machine learning called INNA. It exploits the geometry of the loss function while only requiring stochastic approximations of the function values and the generalized gradients. This makes INNA fully implementable and adapted to large-scale optimization pr…
Graph manifolds are manifolds that decompose along tori into pieces with a tame S1-structure. In this paper, we prove that the simplicial volume of graph manifolds (which is known to be zero) can be approximated by integral simplicial volumes of their finite coverings. This gives a uniform proof of the vanishing of …
We introduce the new class of submanifolds of co-Banach type in tame Fréchet manifolds and construct tame Fréchet submanifolds as inverse images of regular values of certain tame maps. Our method furnishes an easy way to construct tame Fréchet manifolds. The results presented are key ingredients in the construction of …
Paper characterizes tamed and weakened tamed four-manifolds using a new technique.
problem Characterizing tamed and weakened tamed four-manifolds.
method Using weakly \widetilde{\mathcal{D}}^+_J -closed technique.
result Characterization of tamed and weakened tamed four-manifolds.
New schemes improve error estimates for sampling from non-log-concave distributions.
problem Improving sampling from non-log-concave distributions with super-linear drift growth.
method Developed tamed Euler and randomized Euler schemes with error estimates.
result Near-optimal error bounds for sampling and optimization problems.
New algorithm improves stability and efficiency of neural network training.
problem Stability and efficiency issues in adaptive optimization for neural networks.
method Polygonal approximations for SDEs with monotone coefficients, providing stability and addressing vanishing gradients.
result TheoPouLa algorithm shows superior performance over popular adaptive optimizers.
Stochastic gradient descent (SGD) is a ubiquitous algorithm for a variety of machine learning problems. Researchers and industry have developed several techniques to optimize SGD's runtime performance, including asynchronous execution and reduced precision. Our main result is a martingale-based analysis that enables us…
The study confirms Gromov's speculation and provides bounds for taming symplectic structures.
problem Understanding the relationship between taming symplectic structures and the area of pseudoholomorphic curves.
method Analyzes the numerical cone of taming symplectic structures and characterizes coarsely holomorphic curves.
result An almost complex manifold with an area bound admits a taming symplectic structure, confirming Gromov's speculation.
Solenoids are ``inverse limits'' of the circle, and the classical knot theory is the theory of tame embeddings of the circle into the 3-space. We give some general study, including certain classification results, of tame embeddings of solenoids into the 3-space as the ``inverse limits'' of the tame embeddings of the ci…
New algorithms sample from log concave distributions without gradient Lipschitz continuity.
problem Sampling from log concave distributions without gradient Lipschitz continuity.
method Two algorithms based on monotone polygonal (tamed) Euler schemes.
result Non-asymptotic 2-Wasserstein distance bounds between the process and target measure.
We prove that the group STame(k3) of special tame automorphisms of the affine 3-space is not simple, over any base field of characteristic zero. Our proof is based on the study of the geometry of a 2-dimensional simply-connected simplicial complex C on which the tame automorphism group acts naturally. We prove that …
A closed discrete subset A⊂C is called tame if C∖A is quasiconformally equivalent to C∖Z. By giving several criteria for A to be tame, we shall show that Z+iZ is not tame.
Positivity of intersections in 4-manifolds leads to taming symplectic structures.
problem Taming symplectic structures in almost complex 4-manifolds.
method Proof of positivity of intersections of pseudoholomorphic curves.
result Positivity of intersections is stable and leads to taming symplectic structures.
Whitehead doubles have matching meridional rank and bridge number.
problem Determining the meridional rank of Whitehead doubles.
method Analyzing algebraically tame knots and their Whitehead doubles.
result Meridional rank and bridge number coincide for Whitehead doubles of prime knots.
Paper proposes new Langevin samplers for sampling from log-concave distributions with superlinear gradient growth.
problem Sampling from log-concave distributions with superlinear gradient growth.
method Proposes two novel discretizations of kinetic Langevin SDEs, showing contractivity and log-Sobolev inequality.
result Establishes non-asymptotic bounds in 2-Wasserstein distance between sampled distributions and target measures.
Every countable compact subset of sphere is tame.
problem Characterizing compact subsets of spheres.
method Proving homeomorphic complements imply homeomorphic subsets.
result Wild subspaces like Antoine contain Cantor sets.
Four-dimensional, oriented Lie algebras g which satisfy the tame-compatible question of Donaldson for all almost complex structures J on g are completely described. As a consequence, examples are given of (non-unimodular) four-dimensional Lie algebras with almost complex structures which are…
We give several versions of local and global inverse mapping theorem for tame non necessarily smooth, mappings. Here tame mapping means a mapping which is subanalytic or, more generally, definable in some o-minimal structure. Our sufficient conditions are formulated in terms of various properties (convexity, positivity…
Develops calculus for tamed Dirichlet spaces using measure theory.
problem Defines calculus for measure spaces with Dirichlet forms.
method Introduces first and second order calculus on tamed Dirichlet spaces.
result Defines various geometric objects on tamed Dirichlet spaces.
In this paper we show that a complete and non-compact surface immersed in the Euclidean space with quadratic extrinsic area growth has finite total curvature provided the surface has tamed second fundamental form and admits total curvature. In such a case we obtain as well a generalized Chern-Osserman inequality. In th…
New learning scheme solves high-dimensional semi-linear PDEs using sparse grids and Picard approximations.
problem Solving high-dimensional semi-linear parabolic PDEs.
method Probabilistic learning scheme based on Picard iteration with SGD, employing sparse grid approximation.
result Convergence proof and polynomial complexity in ε−1 for high-dimensional PDEs. Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.
problem Suboptimal performance of Asynchronous SGD under heterogeneous worker computation times.
method Ringmaster ASGD, a novel Asynchronous SGD method with optimal time complexity.
result Ringmaster ASGD achieves optimal time complexity under arbitrary worker heterogeneity.
We classify Veech groups of tame non-compact flat surfaces. In particular we prove that all countable subgroups of GL+(2,R) avoiding the set of mappings of norm less than 1 appear as Veech groups of tame non-compact flat surfaces which are Loch Ness monsters. Conversely, a Veech group of any tame flat surf…
Symplectic forms taming complex structures on compact manifolds are strictly related to Hermitian metrics having the fundamental form ∂∂ˉ-closed, i.e. to strong Kähler with torsion (SKT) metrics. It is still an open problem to exhibit a compact example of a complex manifold having a tamed …
We study the existence of strong Kähler with torsion (SKT) metrics and of symplectic forms taming invariant complex structures J on solvmanifolds G/Γ providing some negative results for some classes of solvmanifolds. In particular, we show that if either J is invariant under the action of a nilpotent complement o…
A new algorithm improves GAN training stability and performance.
problem Training GANs is difficult due to variance and rotational dynamics.
method Proposes Lookahead-Minmax algorithm for minmax optimization.
result Significant improvement in GAN performance and stability.
In this short note, we give a new proof of a theorem of Arezzo-Tian on the existence of smooth geodesic rays tamed by a special degeneration.
We study Nakai-Moishezon type question and Donaldson's "tamed to compatible" question for almost complex structures on rational four manifolds. By extending Taubes' subvarieties--current--form technique to J−nef genus 0 classes, we give affirmative answers of these two questions for all tamed almost complex structu…
Combines techniques to remove tameness condition in Morse-Smale flows.
problem Tameness condition in Morse-Smale flows.
method Combines Shilnikov's ODE techniques with Latschev's ideas.
result Removes tameness hypothesis from homotopy formula.
We show that a complete submanifold M with tamed second fundamental form in a complete Riemannian manifold N with sectional curvature KN≤κ≤0 are proper, (compact if N is compact). In addition, if N is Hadamard then M has finite topology. We also show that the fundamental tone is an obstruction fo…
Neural networks estimate SDEs with jump noise using a Tamed-Milstein scheme.
problem Estimating drift and diffusion functions in SDEs with jump noise.
method Tamed-Milstein scheme with neural networks as non-parametric approximators.
result Flexible estimation of complex nonlinear dynamics in systems with state-dependent noise.
We introduce a new technique for finding CAT(-1) surfaces in hyperbolic 3-manifolds. We use this to show that a complete hyperbolic 3-manifold with finitely generated fundamental group is geometrically and topologically tame.
New method samples from non-log-concave distributions with weak dissipativity.
problem Sampling from distributions that are not log-concave and weakly dissipative.
method Taming scheme tailored to growth and decay properties of the target distribution.
result Explicit non-asymptotic guarantees for KL, TV, and Wasserstein distances.
Product of shellable complexes yields shellable triangulations under tameness conditions.
problem Understanding shellability in products of simplicial complexes.
method Definition and proof of shellability properties for products of complexes under tameness conditions.
result Product of shellable complexes yields shellable triangulations under certain conditions.