Framework for accelerated gradient flows in Bayesian inverse problems.
problem Design efficient MCMC algorithms for Bayesian inverse problems.
method Nesterov's accelerated gradient flows in probability space, considering various information metrics.
result Proved convergence properties and proposed sampling-efficient algorithms for different metrics.
The paper accelerates gradient flows on probability distributions using optimal control theory.
problem Optimizing probability distributions efficiently.
method Variational formulation and Hamilton's equations for accelerated gradient flows.
result The method achieves accelerated density transport from any initial distribution to a target distribution.
HF-opt uses Hamiltonian dynamics to optimize functions, achieving accelerated rates with randomized integration time.
problem Optimizing functions efficiently and accelerating convergence rates.
method Randomized Hamiltonian flow (RHF) with accelerated convergence rates.
result RHGD achieves accelerated convergence rates similar to Nesterov's AGD.
ASVGD accelerates SVGD for efficient sampling.
problem Slow SVGD in high-dimensional sampling.
method Accelerated gradient flow in a metric space of probability densities, using Nesterov's method and momentum-based updates.
result ASVGD outperforms SVGD and other methods in sampling efficiency.
SympFormer accelerates attention blocks using inertial dynamics on density spaces.
problem Improving the efficiency of self-attention blocks in Transformers.
method Introduced accelerated attention blocks derived from inertial Nesterov dynamics on density spaces.
result Accelerated attention blocks converge faster than classical blocks while preserving oracle calls.
ASVGD accelerates SVGD for efficient sampling from Gaussian targets.
problem Efficient sampling from Gaussian distributions using SVGD.
method Accelerated gradient flow in a metric space of probability densities, including momentum and Wasserstein regularization.
result ASVGD achieves optimal convergence rate for Gaussian targets, independent of covariance.
Unified view of accelerated and stochastic optimization methods.
problem Optimization challenges in machine learning and physics.
method Unified gradient flow approach to proximal algorithms and their accelerated variants.
result Unified framework for accelerated and stochastic optimization methods.
New method accelerates optimization in fixed time, improving convergence rates.
problem Optimization in large-scale data-driven problems.
method Gradient-based optimization framework with fixed-time stable dynamical systems.
result Achieves convergence to the optimizer in a fixed number of iterations, independent of initialization.
Study accelerates gradient methods in machine learning, revealing risk and stability connections.
problem Understanding the statistical risk of accelerated gradient methods in machine learning.
method Continuous-time analysis of Nesterov's accelerated gradient method and Polyak's heavy ball method for least squares regression.
result Connections between early stopping, stability, and curvature of loss function are revealed.
GFM models neural network training as a dynamical system to forecast final weights.
problem Computational intensity and inefficiency in training deep neural networks.
method Gradient Flow Matching (GFM) treats training as a dynamical system with learned vector fields.
result GFM achieves forecasting accuracy competitive with Transformer-based models and significantly outperforms classical baselines.
As one of standard approaches to train deep neural networks, dropout has been applied to regularize large models to avoid overfitting, and the improvement in performance by dropout has been explained as avoiding co-adaptation between nodes. However, when correlations between nodes are compared after training the networ…
A new ParVI framework improves particle-based variational inference methods.
problem Non-trivial kernel design in particle-based variational inference methods.
method Proposes a generalized Wasserstein gradient descent (GWG) framework with broader regularizers.
result Demonstrates strong convergence guarantees and effectiveness on simulated and real data.
Accelerates Riemannian gradient methods with extrapolation.
problem Optimizing functions on manifolds efficiently.
method Extrapolating iterates in Riemannian gradient descent.
result Achieves optimal convergence rate and computational advantage.
Unified geometric flows improve deep learning efficiency and simplify neural network topologies.
problem Improving deep learning performance and simplifying neural network structures.
method Proposes a thermodynamically coupled Ricci flow that dynamically adapts parameter space geometry to loss landscape topology, enabling automated singularity resolution and providing entanglement entropy bounds.
result Demonstrates 2.1× convergence acceleration and 63% topological simplification while maintaining O(NlogN) complexity, outperforming Riemannian baselines by 15.2% in few-shot accuracy. Flow-VQE uses generative flows to optimize VQE efficiently.
problem Complex objective functions and expensive optimization in VQE.
method Generative normalizing flows with parameterized quantum circuits.
result Flow-VQE accelerates convergence and reduces circuit evaluations.
AGNES accelerates gradient descent with noisy gradients.
problem Minimizing smooth convex and strongly convex functions with noisy gradients.
method Generalization of Nesterov's accelerated gradient descent algorithm for noisy conditions.
result AGNES achieves acceleration for noisy gradients with a constant of proportionality up to 1.
Develops accelerated methods for optimization using low-dimensional projected-gradient information.
problem Optimization with low-dimensional projected-gradient information and Nesterov acceleration.
method Randomized-subspace Nesterov accelerated gradient methods for smooth convex and strongly convex optimization.
result Established accelerated oracle-complexity guarantees and unified basis for comparing sketch families.
New method accelerates energetic variational inference using particle dynamics.
problem Efficiently solving variational inference problems with reduced computational cost.
method Particle-based variational inference with implicit scheme, inspired by energy quadratization and operator splitting.
result Significantly reduces computational cost compared to existing methods.
Super-acceleration of gradient descent with momentum improves loss function minimization.
problem Minimizing loss functions in machine learning.
method Extending Nesterov acceleration by using gradients at multiple steps ahead.
result Super-acceleration of the momentum algorithm is beneficial for various loss landscapes and tasks.
Gradient flow autoencoder improves data efficiency over traditional autoencoders.
problem Sub-optimal latent space representations in autoencoders.
method Gradient flow through ODE with adaptive step size for optimization.
result Gradient flow autoencoder achieves higher data efficiency.
Accelerated gradient method's stability deteriorates exponentially with steps.
problem Algorithmic stability of Nesterov's accelerated gradient method.
method Analysis of two notions of algorithmic stability for Nesterov's accelerated gradient method.
result Stability of Nesterov's accelerated method deteriorates exponentially with the number of gradient steps.
Locally Accelerated Conditional Gradients improve convergence rates for smooth convex optimization problems.
problem Achieving optimal convergence rates for smooth convex optimization problems over polytopes.
method Locally Accelerated Conditional Gradients, coupling accelerated steps with conditional gradient steps.
result Achieves optimal accelerated local convergence for smooth strongly convex problems.
FAKI improves gradient-free inference for inverse problems.
problem Expensive forward models without gradients.
method Temperature annealing with normalizing flows.
result Dramatic improvements in accuracy over EKI.
Particle-based variational inference methods (ParVIs) have gained attention in the Bayesian inference literature, for their capacity to yield flexible and accurate approximations. We explore ParVIs from the perspective of Wasserstein gradient flows, and make both theoretical and practical contributions. We unify variou…
A new method directly encodes data into latent space using gradient flow.
problem Suboptimal representations in physical sciences due to encoder inversion.
method Decoder-only approach using gradient flow and ODEs, avoiding integrals.
result Superior data efficiency and explicit encoding compared to traditional autoencoders.
Accelerates coordinate descent methods for machine learning problems.
problem Slowness of coordinate descent methods in machine learning.
method Extrapolation-based accelerated coordinate descent.
result Significant speed-up in practice compared to existing methods.
Continuized Nesterov acceleration accelerates stochastic gradient descent and gossip algorithms.
problem Improving the convergence rate of stochastic gradient descent and gossip algorithms.
method Introducing a continuized variant of Nesterov acceleration, which mixes variables continuously and takes gradient steps at random times.
result The continuized Nesterov acceleration achieves convergence rates similar to Nesterov's original acceleration but with random parameters.
We analyze Riemannian accelerated methods using a new framework.
problem Understanding Riemannian accelerated gradient methods.
method Riemannian A-HPE framework, focusing on Euclidean A-HPE insights and metric distortion control.
result Characterization of acceleration for various Riemannian methods.
There is widespread sentiment that it is not possible to effectively utilize fast gradient methods (e.g. Nesterov's acceleration, conjugate gradient, heavy ball) for the purposes of stochastic optimization due to their instability and error accumulation, a notion made precise in d'Aspremont 2008 and Devolder, Glineur, …
New method accelerates gradient descent on curved spaces.
problem Optimizing functions on curved Riemannian manifolds.
method Developed a novel geometric inequality to control metric distortion, enabling a Riemannian accelerated gradient method.
result Proposed the first global accelerated gradient method for Riemannian manifolds.
Acceleration in Hilbert spaces reduces computations but not accuracy.
problem Improving learning accuracy with fewer computations.
method Analysis of Nesterov acceleration and heavy-ball methods in Hilbert spaces.
result Acceleration can reduce computations but not improve accuracy with respect to gradient descent.
This work accelerates gradient descent with anytime convergence guarantees.
problem Improving the convergence rate of gradient descent methods.
method Proposes a stepsize schedule for gradient descent that achieves anytime convergence rates.
result Gradient descent can achieve convergence rates of O(T−1.119) for any stopping time T. Unified analysis of conjugate gradients and accelerated methods using duality gap.
problem Minimizing convex quadratic functions efficiently.
method Approximate Duality Gap Technique to unify conjugate gradients and accelerated methods.
result Unified and self-contained proof of conjugate gradients without relying on Chebyshev polynomials.
AGBM accelerates GBM with theoretical guarantees.
problem Accumulation of errors in GBM's momentum term.
method Incorporates Nesterov's acceleration techniques and a corrected pseudo residual.
result First GBM type with theoretically-justified accelerated convergence rate.
Optimizes convex functions in finite vs infinite dimensions, revealing slow convergence rates.
problem Analyzing gradient flows in finite and infinite-dimensional Hilbert spaces.
method Proves convergence rates and optimality conditions for gradient flows and related methods.
result Gradient flow convergence rates in finite dimensions are slower than in infinite dimensions, with optimal rates achievable in Hilbert spaces.
FedAc accelerates Federated Averaging for distributed optimization.
problem Efficiently optimizing distributed machine learning models.
method Federated Accelerated Stochastic Gradient Descent (FedAc) using a potential-based perturbed iterate analysis.
result FedAc achieves faster convergence and lower communication costs than previous methods.
Momentum speeds up evolutionary processes in machine learning.
problem Accelerating convergence in evolutionary dynamics.
method Combining momentum from machine learning with evolutionary dynamics using information divergences as Lyapunov functions.
result Momentum accelerates convergence of evolutionary dynamics, including the replicator equation and Euclidean gradient descent.
Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. While many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelera…
Jeffreys Flow improves robustness of Boltzmann generators for rare event sampling.
problem Rare events and metastable trapping in sampling physical systems with rough energy landscapes.
method Introduces Jeffreys Flow, a robust generative framework using Parallel Tempering distillation and symmetric Jeffreys divergence to mitigate mode collapse and improve mode coverage.
result Minimizing Jeffreys divergence suppresses mode collapse and corrects inaccuracies in multi-modal distributions.
PF-LaCG removes the need for knowing smoothness and strong convexity parameters for locally accelerated CG.
problem Locally accelerated CG requires knowledge of smoothness and strong convexity parameters.
method Parameter-Free Locally Accelerated CG (PF-LaCG) algorithm.
result PF-LaCG achieves local acceleration without requiring knowledge of smoothness and strong convexity parameters.
New methods accelerate distributed optimization in noisy networks.
problem Optimizing distributed stochastic gradient methods for noisy, connected networks.
method Developed a framework for choosing stepsize and momentum parameters, proving acceleration and providing performance bounds.
result Distributed accelerated methods achieve acceleration with optimal complexity, reducing bias and variance.
Novel method improves training RNNs by accelerating gradient descent.
problem Vanishing and exploding gradient problems in RNNs training.
method Adaptive stochastic Nesterov accelerated quasi-Newton method.
result Improved performance in training RNNs with low per-iteration cost.
Two new differentially private optimization algorithms derived from accelerated methods.
problem Improving privacy in optimization algorithms while maintaining convergence rates.
method Polyak's heavy ball method and Nesterov's accelerated gradient method with differential privacy.
result The proposed algorithms outperform existing differentially private optimization methods.
No accelerated gradient method for hyperbolic convex functions.
problem Existence of accelerated gradient methods for geodesically convex functions on hyperbolic spaces.
method Analysis of volume growth in negatively curved spaces.
result No-go theorem for accelerated gradient methods on hyperbolic plane.
New methods accelerate gradient descent for convex and strongly convex functions.
problem Improving convergence rates of gradient-based optimization methods.
method Formulated two classes of first-order algorithms with Lyapunov analyses and Hamiltonian assisted gradient method.
result Achieved accelerated convergence rates matching Nesterov's methods in strongly and general convex settings.
This paper investigates asymptotic behaviors of gradient descent algorithms (particularly accelerated gradient descent and stochastic gradient descent) in the context of stochastic optimization arising in statistics and machine learning where objective functions are estimated from available data. We show that these alg…
Improved symbolic regression finds optimal formulas robust to noise.
problem Finding accurate formulas for noisy data.
method Exploits graph modularity, uses normalizing flows, and statistical hypothesis testing.
result Discoveres many formulas previously unattainable.
A new method prunes activation gradients to speed up CNN training.
problem Challenges in accelerating CNN training using sparsity.
method Randomly prunes small activation gradients in back-propagation.
result Substantial speedups (up to 5.92x) with minimal accuracy loss.