Study on test risk dynamics in learning theory with stochastic gradient flow.
problem Understanding test risk in stochastic gradient flow dynamics.
method Path integral formulation for small learning rates, explicit computation for weak features.
result Explicit corrections due to stochastic term in dynamics, good agreement with simulations.
Riemannian stochastic gradient descent approximates a diffusion process called Riemannian stochastic modified flow.
problem Improving convergence rate of Riemannian stochastic gradient descent.
method Using stochastic differential geometry, the paper shows RSGD can be approximated by the Riemannian stochastic modified flow (RSMF).
result RSGD can be approximated by the solution to the RSMF driven by an infinite-dimensional Wiener process, increasing the order of approximation.
New dynamics for SGD in small learning rate regime.
problem Improving stochastic gradient descent in small learning rate regime.
method Introducing stochastic modified flows and distribution dependent stochastic modified flows.
result Captures fluctuating dynamics of SGD in small learning rate - infinite width scaling regime.
Study shows how SGD's implicit regularization relates to ridge regression.
problem Least squares regression optimization with mini-batch SGD.
method Analyzes stochastic gradient flow as a continuous-time model of SGD.
result Bound on excess risk of SGD flow over ridge regression, revealing how parameters drive risk.
We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion is given by an averaged ordinary differential equation. We then demonstrate that…
Paper introduces a differentially private generative model using gradient flow and sliced Wasserstein distance.
problem Protecting privacy in sensitive training data for generative models.
method Gradient flow in the space of probability measures, Gaussian-smoothed Sliced Wasserstein Distance, and numerical scheme for SDE.
result Demonstrates higher-fidelity data generation at low privacy budget compared to existing methods.
Paper explores stability, regularization, and gradient flows for stochastic inverse problems.
problem Recovering random probability distributions from measurements.
method Direct inversion, variational formulation with regularization, and optimization via gradient flows.
result The choice of metric impacts stability and properties of the optimizer.
Introduces a new stochastic optimization method for deep learning.
problem Minimizing loss functions in deep neural networks.
method Introduces a second-order stochastic Runge-Kutta method and an adaptive SGD-G2.
result The method yields consistent minimization of loss functions and automatically adjusts learning rates.
In general, gradient estimates are very important and necessary for deriving convergence results in different geometric flows, and most of them are obtained by analytic methods. In this paper, we will apply a stochastic approach to systematically give gradient estimates for some important geometric quantities under the…
This paper investigates asymptotic behaviors of gradient descent algorithms (particularly accelerated gradient descent and stochastic gradient descent) in the context of stochastic optimization arising in statistics and machine learning where objective functions are estimated from available data. We show that these alg…
New gradient estimator improves training for normalizing flows.
problem Training normalizing flows for complex models.
method Developed a new gradient estimator for Stochastic Gradient Descent.
result Significantly faster and more precise training for φ^4 model.
Self-test loss functions improve data-driven modeling of weak-form operators and gradient flows.
problem Challenges in selecting test functions for data-driven modeling involving weak-form operators and gradient flows.
method Introducing self-test loss functions that depend on unknown parameters and are quadratic.
result Self-test loss functions conserve energy for gradient flows and coincide with log-likelihood ratios for stochastic differential equations.
Analyzes high-dimensional SGD dynamics using DMFT.
problem Understanding the high-dimensional behavior of multi-pass SGD with small batch sizes.
method Derives DMFT equations for high-dimensional SGD dynamics.
result Proves DMFT equations characterize the asymptotic distribution of SGF parameters.
SGD with large learning rates can achieve better test accuracy than expected.
problem SGD with large learning rates often outperforms expected convergence bounds.
method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.
result Explicitly adding an implicit regularizer to the loss improves test accuracy.
Analysis of SGD for Gaussian mixture classification using dynamical mean-field theory.
problem Learning dynamics of SGD for a neural network classifying Gaussian mixture.
method Applying dynamical mean-field theory to track SGD dynamics in high dimensions.
result Reveals how SGD navigates the non-convex loss landscape.
A new gradient flow framework for distributionally robust optimization.
problem Optimizing under uncertainty with worst-case distributional constraints.
method Gradient flow theory applied to distributionally robust optimization.
result Practical algorithms for sampling from worst-case distributions.
The paper analyzes reg-SGD for convex problems, proving convergence and quantifying the rate of convergence.
problem Minimizing convex, L-smooth functions in a Hilbert space.
method Regularized stochastic gradient descent with decaying regularization.
result Strong convergence to the minimum-norm solution without boundedness assumptions.
Novel approach finds implicit regularisation in two-player games using BEA.
problem Understanding implicit regularisation in two-player games.
method Using backward error analysis to construct continuous-time flows with gradient-eligible vector fields.
result Identifies new implicit regularisation effects in two-player games.
Stochastic gradient methods converge for training wide PINNs.
problem Convergence of stochastic gradient descent in training over-parameterized PINNs.
method Established linear convergence of stochastic gradient descent/flow in training over-parameterized two-layer PINNs.
result Linear convergence with high probability for general activation functions.
New algorithms for sampling and optimization without tuning.
problem Efficient sampling and optimization over probability measures.
method Optimization on the space of probability measures, using gradient flows.
result Strong theoretical guarantees and similar performance to optimally tuned algorithms.
New framework for Bayesian inference using neural Schrödinger-Föllmer flows.
problem Approximate Bayesian inference in large datasets.
method Stochastic control, Schrödinger bridges, SDE-based models.
result Advocates stochastic control as a finite time and low variance alternative to SGLD.
Stable training of deep normalizing flows for high-dimensional variational inference.
problem Training deep normalizing flows for high-dimensional posterior distributions is infeasible due to high stochastic gradient variance.
method Proposed a combination of soft-thresholding of scale and bijective soft log transformation to stabilize training.
result Stable training of Real NVPs for posterior distributions with thousands of dimensions is possible.
Mathematical analysis of SNE and t-SNE for dimension reduction.
problem Optimal mapping of high-dimensional data to low dimensions.
method Gradient flow of relative entropy to minimize the distance between points.
result The diameter of the evolving sets remains bounded for SNE but may blow up for t-SNE.
Functional central limit theorem for kernel gradient flow and infinitesimal gradient boosting
problem Fluctuations of boosting processes around their deterministic limit
method Stochastic perturbation analysis of ODEs in Banach spaces
result Rescaled deviations converge to a Gaussian process
New method reveals insights about stochastic optimization methods using modified equations.
problem Understanding the qualitative behavior of stochastic optimization algorithms.
method Developed a class of stochastic differential equations to approximate the dynamics of stochastic optimization methods.
result Mean-square stability of the modified equation provides qualitative insights about stochastic coordinate descent.
Optimizes convex functions in finite vs infinite dimensions, revealing slow convergence rates.
problem Analyzing gradient flows in finite and infinite-dimensional Hilbert spaces.
method Proves convergence rates and optimality conditions for gradient flows and related methods.
result Gradient flow convergence rates in finite dimensions are slower than in infinite dimensions, with optimal rates achievable in Hilbert spaces.
Variational inference has experienced a recent surge in popularity owing to stochastic approaches, which have yielded practical tools for a wide range of model classes. A key benefit is that stochastic variational inference obviates the tedious process of deriving analytical expressions for closed-form variable updates…
CoSMIC extends flow-based SVI to transdimensional problems.
problem Bayesian structure learning and model selection with multi-model parameter spaces.
method Normalizing flows with a combined stochastic variational transdimensional inference approach.
result Improved performance on high-cardinality model spaces.
SGD handles label noise with bounds improving over SGLD.
problem Label noise in non-convex optimization.
method Stochastic gradient descent with uniform dissipativity and smoothness conditions, using Wasserstein distance and algorithmic stability.
result Generalization error bounds with a rate of n−2/3, better than SGLD's n−1/2. This paper analyzes neural networks for solving complex optimization problems.
problem Minimax optimization problems in infinite-dimensional function spaces.
method Mean-field analysis of stochastic gradient descent-ascent in neural networks.
result The algorithm converges to a stationary point at a sublinear rate.
Proves equations for high-dimensional gradient-based methods from Gaussian data.
problem High-dimensional asymptotics of gradient-based learning algorithms.
method Closed-form equations derived from dynamical mean-field theory.
result Equations match those from discretized DMFT for gradient flow.
In this paper, we investigate the geometry of a general class of gradient flows with multiple local maxima. we decompose the underlying space into disjoint regions of attraction and establish the adjacency criterion. The criterion states a necessary and sufficient condition for two regions of attraction of stable equil…
Proves convergence of neural networks in a two-timescale regime.
problem Training dynamics of shallow neural networks.
method Two-timescale regime analysis of gradient flow.
result Gradient flow converges to global optimum in non-convex optimization.
Jeffreys Flow improves robustness of Boltzmann generators for rare event sampling.
problem Rare events and metastable trapping in sampling physical systems with rough energy landscapes.
method Introduces Jeffreys Flow, a robust generative framework using Parallel Tempering distillation and symmetric Jeffreys divergence to mitigate mode collapse and improve mode coverage.
result Minimizing Jeffreys divergence suppresses mode collapse and corrects inaccuracies in multi-modal distributions.
A new method for discrete data normalizing flows using latent transformations.
problem Challenges in parameterizing bijective transformations for discrete data.
method Predict a distribution over latent transformations to make the marginal likelihood differentiable.
result Discrete-data normalizing flows can be trained using gradient-based learning with unbiased score function estimation.
A new stochastic algorithm approximates optimal distributions without requiring propagation of chaos.
problem Optimizing functionals over probability distributions using finite particle systems.
method Virtual particle stochastic approximation, viewed as a form of stochastic gradient descent in the Wasserstein space.
result The algorithm's output converges to the optimal distribution and produces i.i.d. samples.
Study SGD dynamics in high-dimensional models, revealing consistent behavior across different batch sizes and learning rates.
problem Understanding SGD dynamics in high-dimensional multi-index models.
method Asymptotic analysis of SGD, developing mean-field equations and Gaussian diffusion approximations.
result Consistent SGD dynamics across different batch sizes and learning rates, distinct from gradient flow and online SGD.
This paper introduces a new probabilistic model for online learning which dynamically incorporates information from stochastic gradients of an arbitrary loss function. Similar to probabilistic filtering, the model maintains a Gaussian belief over the optimal weight parameters. Unlike traditional Bayesian updates, the m…
The paper extends gradient flow and relaxation studies to non-flat Riemannian manifolds.
problem Understanding gradient flows and relaxation in non-flat Riemannian manifolds.
method Developed a criterion for comparing relaxation along gradient descent curves using non-metricity tensor.
result Revealed a universal asymmetry: warming up is faster than cooling down.
The paper explores how symmetries and noise in SGD influence parameter dynamics.
problem Understanding the dynamics of parameter updates in SGD with symmetries.
method Proved the existence of noise equilibria and showed their role in balancing gradient noise.
result Gradient noise creates a systematic motion of parameters to a unique fixed point, called noise equilibria.
A scalable algorithm for sampling and fine-tuning models using Tilt Matching.
problem Efficient sampling and fine-tuning of generative models.
method Tilt Matching, arising from a dynamical equation, minimizes variance and inherits regularity from stochastic interpolants.
result Empirically verified to be efficient and highly scalable, providing state-of-the-art results.
Stochastic particle-optimization sampling (SPOS) is a recently-developed scalable Bayesian sampling framework that unifies stochastic gradient MCMC (SG-MCMC) and Stein variational gradient descent (SVGD) algorithms based on Wasserstein gradient flows. With a rigorous non-asymptotic convergence theory developed recently…
Analyzes symmetries in neural networks to predict learning dynamics.
problem Understanding the dynamics of neural network parameters during training.
method Unified theoretical framework based on symmetries and conservation laws.
result Symmetries impose geometric constraints on gradients and Hessians, leading to conservation laws.
Most deep learning models are based on deep neural networks with multiple layers between input and output. The parameters defining these layers are initialized using random values and are "learned" from data, typically using stochastic gradient descent based algorithms. These algorithms rely on data being randomly shuf…
This work classifies SOC loss functions based on their gradient properties.
problem Optimizing noisy systems in stochastic optimal control.
method Grouping loss functions into classes with the same gradient expectation.
result Different loss functions have the same optimization landscape but differ in gradient variance.
Generative model improved using Liouville PDE-based sliced-Wasserstein flow.
problem Improving generative models for fair regression.
method Transformed sliced-Wasserstein flow into Liouville PDE-based formalism, handling density estimation with normalizing flows of neural ODE.
result Outperforms in convergence and fairness with reduced variance.
Noise balance theory explains SGD's behavior in neural networks.
problem Understanding SGD's navigation in neural network loss landscapes.
method Analyzes minibatch noise and loss function symmetries.
result Derives the stationary distribution of SGD for deep networks.
SGD in DLNs reveals feature learning dynamics.
problem Understanding SGD dynamics in DLNs during saddle-to-saddle training.
method Stochastic Langevin dynamics with anisotropic, state-dependent noise; one-dimensional per-mode SDEs; Boltzmann distribution approximation.
result SGD noise encodes feature learning progression but does not alter saddle-to-saddle dynamics.