HF-opt uses Hamiltonian dynamics to optimize functions, achieving accelerated rates with randomized integration time.
problem Optimizing functions efficiently and accelerating convergence rates.
method Randomized Hamiltonian flow (RHF) with accelerated convergence rates.
result RHGD achieves accelerated convergence rates similar to Nesterov's AGD.
New methods accelerate gradient descent for convex and strongly convex functions.
problem Improving convergence rates of gradient-based optimization methods.
method Formulated two classes of first-order algorithms with Lyapunov analyses and Hamiltonian assisted gradient method.
result Achieved accelerated convergence rates matching Nesterov's methods in strongly and general convex settings.
New bounds for model generalization under deterministic gradient descent.
problem Establishing generalization bounds for models trained with gradient descent methods.
method PAC-Bayesian bounds for deterministic optimisation algorithms.
result Fully computable bounds that depend on initial distribution and Hessian.
Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) is a momentum version of stochastic gradient descent with properly injected Gaussian noise to find a global minimum. In this paper, non-asymptotic convergence analysis of SGHMC is given in the context of non-convex optimization, where subsampling techniques are used o…
Study on Langevin dynamics convergence rates and their application to GAN training.
problem Understanding the long-term behavior of Langevin dynamics equations.
method Analytical and numerical methods to study convergence rates of underdamped mean-field Langevin dynamics.
result Exponential convergence rate results for the Langevin dynamics under various conditions.
ECD algorithm speeds up non-convex optimization, offering quantum and stochastic enhancements.
problem Non-convex optimization challenges in machine learning.
method Energy Conserving Descent (ECD) algorithm, stochastic ECD dynamics (sECD), quantum ECD Hamiltonian (qECD).
result ECD and its quantum version achieve exponential speedup over gradient descent.
We take a Hamiltonian-based perspective to generalize Nesterov's accelerated gradient descent and Polyak's heavy ball method to a broad class of momentum methods in the setting of (possibly) constrained minimization in Euclidean and non-Euclidean normed vector spaces. Our perspective leads to a generic and unifying non…
Hamiltonian Monte Carlo (HMC) is a popular Markov chain Monte Carlo (MCMC) algorithm that generates proposals for a Metropolis-Hastings algorithm by simulating the dynamics of a Hamiltonian system. However, HMC is sensitive to large time discretizations and performs poorly if there is a mismatch between the spatial geo…
A method for efficient approximate inference on discrete distributions.
problem Applying SVGD to discrete distributions.
method Transforming discrete distributions to piecewise continuous distributions for SVGD application.
result Outperforms traditional algorithms and ensemble methods on discrete graphical models.
New method improves convergence for smooth games.
problem Improving convergence for smooth games.
method Stochastic Hamiltonian Gradient Methods (SHGD).
result SHGD converges linearly to the neighbourhood of a stationary point.
Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting. However, whether these methods are superior to GD in the nonconvex setting remains open. This paper studies a simple variant…
While classic work in convex-concave min-max optimization relies on average-iterate convergence results, the emergence of nonconvex applications such as training Generative Adversarial Networks has led to renewed interest in last-iterate convergence guarantees. Proving last-iterate convergence is challenging because ma…
Neural networks are discrete entities: subdivided into discrete layers and parametrized by weights which are iteratively optimized via difference equations. Recent work proposes networks with layer outputs which are no longer quantized but are solutions of an ordinary differential equation (ODE); however, these network…
Deep learning is built on the foundational guarantee that gradient descent on an objective function converges to local minima. Unfortunately, this guarantee fails in settings, such as generative adversarial nets, that exhibit multiple interacting losses. The behavior of gradient-based methods in games is not well under…
New algorithm reduces overfitting in neural networks.
problem Overfitting in neural networks.
method Integrates SMC with SGHMC for mini-batch sampling.
result SMCSGHMC outperforms SGD and deep ensembles.
Policy gradient methods are powerful reinforcement learning algorithms and have been demonstrated to solve many complex tasks. However, these methods are also data-inefficient, afflicted with high variance gradient estimates, and frequently get stuck in local optima. This work addresses these weaknesses by combining re…
New algorithm improves efficiency of quantum system modeling.
problem Intractable complexities in quantum Hamiltonian learning and Gibbs sampling.
method Generalized quantum natural gradient descent and Quantum-Probabilistic Mirror Descent.
result Data sample efficiency proven using information geometry and quantum metrology.
We propose a family of optimization methods that achieve linear convergence using first-order gradient information and constant step sizes on a class of convex functions much larger than the smooth and strongly convex ones. This larger class includes functions whose second derivatives may be singular or unbounded at th…
Gradient-based Monte Carlo sampling algorithms, like Langevin dynamics and Hamiltonian Monte Carlo, are important methods for Bayesian inference. In large-scale settings, full-gradients are not affordable and thus stochastic gradients evaluated on mini-batches are used as a replacement. In order to reduce the high vari…
New method combines neural networks with Monte Carlo for complex system reliability.
problem Estimating small failure probabilities in complex systems.
method Subset Simulation with Hamiltonian Neural Networks.
result High acceptance rates and computational efficiency in low-probability regions.
Recurrent neural networks (RNNs) have gained a great deal of attention in solving sequential learning problems. The learning of long-term dependencies, however, remains challenging due to the problem of a vanishing or exploding hidden states gradient. By exploring further the recently established connections between RN…
Blind Descent avoids gradient issues, using a different learning approach.
problem Gradient issues like exploding and vanishing gradients.
method Does not use gradients to guide learning; instead, it is a more fundamental learning process.
result Gradient descent is a specific case of Blind Descent.
We propose Kernel Hamiltonian Monte Carlo (KMC), a gradient-free adaptive MCMC algorithm based on Hamiltonian Monte Carlo (HMC). On target densities where classical HMC is not an option due to intractable gradients, KMC adaptively learns the target's gradient structure by fitting an exponential family model in a Reprod…
A new tamed stochastic gradient Hamiltonian Monte Carlo algorithm for superlinearly growing stochastic gradients.
problem Sampling and stochastic optimization problems with superlinearly growing stochastic gradients.
method Tamed Stochastic Gradient Hamiltonian Monte Carlo (tSGHMC) algorithm.
result Established a non-asymptotic error bound in Wasserstein-2 distance with a convergence rate of 1/4. In this note, we observe the behavior of gradient flow and discrete and noisy gradient descent in some simple settings. It is commonly noted that addition of noise to gradient descent can affect the trajectory of gradient descent. Here, we run some computer experiments for gradient descent on some simple functions, and…
Stochastic gradient descent on manifolds improves low-rank approximation.
problem Efficiently approximate large matrices with lower rank.
method Stochastic gradient descent on a manifold.
result Algorithm outperforms Euclidean space methods on Netflix Prize data.
The paper introduces a geometric flow for Lagrangian submanifolds that preserves Hamiltonian isotopy.
problem Finding stationary solutions for Hamiltonian stationary Lagrangian submanifolds.
method Introducing a geometric flow that is a gradient flow for volume and corresponds to a fourth order strictly parabolic scalar equation.
result Established short-time existence, uniqueness, and higher order estimates for compact initial Lagrangian immersions with uniformly bounded second fundamental forms.
New method improves performance of Hamiltonian MCMC for log Z estimation.
problem Estimating tight bounds on log Z for unnormalized distributions.
method Uncorrected Hamiltonian Annealing (UHA) using reparameterization gradients.
result Better performance and easier parameter tuning compared to existing methods.
A new method improves stochastic gradient descent for faster and more efficient estimation.
problem Efficient and fast parametric estimation methods.
method Projected stochastic gradient descent corrected by Fisher scoring.
result The method is faster and more efficient than traditional methods.
New method samples from piecewise smooth distributions using Hamiltonian Monte Carlo.
problem Sampling from distributions with discontinuous gradients.
method Generalized Randomized Hamiltonian Monte Carlo (GRHMC) for piecewise smooth targets.
result GRHMC processes sample from piecewise smooth target distributions with the desired distribution as the invariant distribution.
Approximate Bayesian computation (ABC) is a powerful and elegant framework for performing inference in simulation-based models. However, due to the difficulty in scaling likelihood estimates, ABC remains useful for relatively low-dimensional problems. We introduce Hamiltonian ABC (HABC), a set of likelihood-free algori…
Gradient descent implicitly regularizes neural networks by penalizing large loss gradients.
problem How to optimize deep neural networks without explicit regularization.
method Backward error analysis to calculate implicit gradient regularization and demonstrate its effectiveness empirically.
result Implicit gradient regularization biases gradient descent toward flat minima, improving model robustness and test errors.
Derives Mirror Descent from gradient flow on a Riemannian manifold.
problem No specific problem stated; focuses on derivation.
method Derives Mirror Descent from gradient flow on a Riemannian manifold with a natural discretization.
result Generalizes Mirror Descent to non-Hessian metrics.
We consider the behavior of gradient flow and of discrete and noisy gradient descent. It is commonly noted that the addition of noise to the process of discrete gradient descent can affect the trajectory of gradient descent. In previous work, we observed such effects. There, we considered the case where the minima had …
Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
Reparameterizes mirror descent as gradient descent for efficient sparse learning.
problem Efficiently training small sparse networks with mirror descent.
method Develops a framework to convert mirror descent updates into gradient descent updates on different parameters.
result Mirror descent can be reparameterized as gradient descent on modified parameters, facilitating standard backpropagation.
Gradient descent dynamics in nonconvex models explained with universality.
problem Understanding long-time behavior of nonconvex gradient descent.
method Developed a state evolution system for tracking gradient descent iterates.
result Gradient descent iterates are approximately independent of data and strongly incoherent with feature vectors.
The paper provides a geometric framework for understanding non-equilibrium thermodynamics.
problem Unclear geometric structure of GENERIC in non-equilibrium thermodynamics.
method Cotangent lifts of dynamics, splitting into holonomic and vertical representatives, and formulation within contact geometry.
result Physical meaning and explicit formulation of the second law of thermodynamics within evolution equations.
Gravilon improves gradient descent for neural networks.
problem Improving efficiency and accuracy of gradient descent methods.
method Uses geometric modification of gradient step lengths.
result Promising experimental results on MNIST classification.
L-HNNs improve Bayesian inference by reducing gradient requirements and improving ESS.
problem Efficient Bayesian inference with complex target densities.
method Latent Hamiltonian Neural Networks (L-HNNs) with NUTS, incorporating online error monitoring.
result L-HNNs in NUTS with online error monitoring required 1--2 orders of magnitude fewer numerical gradients and improved ESS by an order of magnitude.
A toy model shows how locality can emerge in the universe's Hamiltonian and initial state.
problem Understanding the emergence of locality in the universe's Hamiltonian and initial state.
method A loss functional is minimized by gradient descent to find a tensor product structure.
result Local structure emerges in the universe's Hamiltonian and initial state through spontaneous symmetry breaking.
New method improves sampling efficiency in complex stochastic systems.
problem Sampling efficiency in nonconvex stochastic gradient cases.
method Reflection coupling for unadjusted generalized Hamiltonian Monte Carlo.
result Quantitative Gaussian concentration bounds and convergence rates established.
SGD reduces test error by decorrelating updates.
problem Improving generalization error in machine learning models.
method Derive a formula for generalization gap change due to SGD updates, compare to GD, and show decorrelation effect.
result SGD implicitly regularizes generalization error by decorrelating updates.
New adaptive step-size method for convex optimization without tuning.
problem Optimizing convex functions efficiently with stochastic gradients.
method Adapted Adaptive Gradient Descent Without Descent to stochastic setting.
result Stochastic gradient descent converges under various assumptions.
Natural gradient descent avoids the magic of model parametrization, leading to different optimization outcomes.
problem Understanding the impact of model parametrization on optimization and generalization in deep learning.
method Characterization of natural gradient flow in deep linear networks and nonlinear neural networks.
result Natural gradient descent fails to generalize in some cases, while gradient descent with the right architecture performs well.
The paper studies a gradient system on a beta statistical manifold, proving integrability and deriving explicit expressions.
problem Investigating the geometry and integrability of a gradient system on a bivariate beta statistical manifold.
method Proving the system is Hamiltonian and admitting a Lax pair representation, deriving explicit expressions using Stirling's approximation, and identifying the Hamiltonian function.
result The gradient flow is linearizable in dual affine coordinates, and the system is completely integrable.
Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…
Gradient descent biases towards stable rank networks for nearly-orthogonal data.
problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.
method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.
result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.