Straight lines are a basin of attraction for the elastic flow at least to level 1.9615π.
problem Understanding the basin of attraction for the free boundary free elastic flow.
method Steepest descent gradient flow for elastic energy, numerical evidence.
result Straight lines have a basin of attraction at least to level 1.9615π.
Study shows how steepest descent algorithms' geometric margin increases during training.
problem Understanding implicit bias in steepest descent algorithms for neural networks.
method Analysis of steepest descent algorithms with infinitesimal learning rates in homogeneous neural networks.
result Limit points of training trajectories correspond to KKT points of margin-maximization problems.
We develop a progressive training approach for neural networks which adaptively grows the network structure by splitting existing neurons to multiple off-springs. By leveraging a functional steepest descent idea, we derive a simple criterion for deciding the best subset of neurons to split and a splitting gradient for …
Study identifies stable configurations of intertwined threads with repulsive interactions.
problem Stable configurations of entangled systems with repulsive interactions.
method Analysis of steepest descent flow of an energy functional.
result Existence and uniqueness of stable configuration of two layers drifting apart at t1/3 rate. This work studies the implicit bias of mini-batch SGD in classification.
problem Understanding the implicit bias of mini-batch SGD in multi-class classification.
method Characterizes how batch size, momentum, and variance reduction affect convergence and max-margin behavior under different norms.
result Momentum enables small-batch convergence to an approximate max-margin solution, while variance reduction recovers the exact full-batch bias.
Let S be a complete surface of constant curvature K = + 1 or -1, i.e. the sphere S^2 or the Lobachevskij plane L^2, and D a bounded convex subset of S. If S = S^2, assume also diameter (D) < pi/2. It is proved that the length of any steepest descent curve of a quasi-convex function in D is less than or equal to the per…
New method escapes local optima in neural architecture optimization.
problem Escaping local optima in neural architecture optimization.
method Signed neural splitting in steepest descent framework.
result Escapes local optima, leading to better performance.
This letter proposes a sparse diffusion steepest-descent algorithm for one bit compressed sensing in wireless sensor networks. The approach exploits the diffusion strategy from distributed learning in the one bit compressed sensing framework. To estimate a common sparse vector cooperatively from only the sign of measur…
Gradient flow expands curves to round shapes.
problem Expanding curves to round shapes.
method Steepest descent L2-gradient flow of entropy.
result Flow converges to a round expanding circle for various initial curves.
Study shows momentum-based optimizers like Muon and MomentumGD bias towards KKT points in smooth homogeneous models.
problem Understanding the implicit bias of momentum-based optimizers on smooth homogeneous models.
method Analysis of Muon, MomentumGD, Signum, and Adam optimizers under decaying learning rate schedules.
result Momentum-based optimizers approximate steepest descent trajectories and bias towards KKT points of margin maximization problems.
New method accelerates steepest descent for convex optimization.
problem Achieving acceleration for general ℓp smooth functions. method Primal-dual iterate sequences with differing norms, implicitly determined interpolation parameter.
result Improves iteration complexity to O(d1−p2) for ℓp norm smooth problems. Improved greedy 2-coordinate updates for optimization problems with constraints.
problem Minimizing smooth functions subject to constraints.
method Exploiting a connection to steepest descent in the 1-norm, we give faster convergence rates and efficient computation.
result Greedy selection converges faster than random selection and can be computed in O(nlogn) time. Characterizes corridors in loss surfaces for gradient-based optimization.
problem Understanding and mitigating training instabilities in gradient-based optimization.
method Characterizes corridors as regions where gradient descent and gradient flow trajectories are linearly related.
result Corridors indicate regions without implicit regularization effects, leading to better learning rate adaptation schemes.
This paper analyzes Stein variational gradient descent for Bayesian inference.
problem Sampling or approximating high-dimensional probability distributions.
method Iterated steepest descent steps with a reproducing kernel Hilbert space norm.
result Performance gains of certain nondifferentiable kernels with adjusted tails.
Two popular examples of first-order optimization methods over linear spaces are coordinate descent and matching pursuit algorithms, with their randomized variants. While the former targets the optimization by moving along coordinates, the latter considers a generalized notion of directions. Exploiting the connection be…
GANs excel at learning high dimensional distributions, but they can update generator parameters in directions that do not correspond to the steepest descent direction of the objective. Prominent examples of problematic update directions include those used in both Goodfellow's original GAN and the WGAN-GP. To formally d…
Method approximates Riemannian barycenter on manifolds.
problem Computing the exact Riemannian barycenter is computationally expensive.
method Uses under- and over-approximations of Riemannian distance to compute an approximate barycenter.
result Approximation method is more efficient than exact methods and steepest descent.
The paper optimizes regret using covariance between costs and decisions.
problem Optimizing expected regret in decision-making problems.
method Developed derivative theory of covariance regret functional, derived Gâteaux derivative, and extended to constrained optimization.
result Gradient of covariance regret is the cost covariance matrix, with implications for portfolio optimization.
In this paper we study the steepest descent L2-gradient flow of the functional $\SW_{λ_1,λ_2}$, which is the the sum of the Willmore energy, λ1-weighted surface area, and λ2-weighted enclosed volume, for surfaces immersed in R3. This coincides with the Helfrich functional with zero `spontaneous curvature'.…
Real flag manifolds are the isotropy orbits of noncompact symmetric spaces G/K. Any such manifold M enjoys two very peculiar geometric properties: It carries a transitive action of the (noncompact) Lie group G, and it is embedded in euclidean space as a taut submanifold. The aim of the paper is to link these two …
Unified signSGD and gradient descent analysis for neural networks.
problem Performance of sign-based optimization methods in neural networks.
method Unified analysis of separable smoothness and ℓ∞-smoothness, isolating geometric properties affecting performance. result Sign-based methods are preferable over gradient descent under specific Hessian properties in deep networks.
In this paper we will discuss how one may be able to use mean curvature flow to tackle some of the central problems in topology in 4-dimensions. We will be concerned with smooth closed 4-manifolds that can be smoothly embedded as a hypersurface in R^5. We begin with explaining why all closed smooth homotopy spheres can…
In optimization, the negative gradient of a function denotes the direction of steepest descent. Furthermore, traveling in any direction orthogonal to the gradient maintains the value of the function. In this work, we show that these orthogonal directions that are ignored by gradient descent can be critical in equilibri…
Adaptive step-size improves optimization in complex geometries.
problem Optimizing functions with non-Euclidean geometries.
method Adaptive step-size strategy for optimization algorithms.
result Guaranteed convergence for Adaptive Conditional Gradient Descent.
Logistic regression is one of the most popular methods in binary classification, wherein estimation of model parameters is carried out by solving the maximum likelihood (ML) optimization problem, and the ML estimator is defined to be the optimal solution of this problem. It is well known that the ML estimator exists wh…
Study uses outer metrics for PDE-constrained shape optimization over diffeomorphism group.
problem Optimizing shapes governed by PDEs over the diffeomorphism group.
method Outer metrics on diffeomorphism group, Riemannian steepest descent method.
result Riemannian approach outperforms other metrics in solving PDE-constrained shape optimization problems.
We study the implicit bias of generic optimization methods, such as mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing underdetermined linear regression or separable linear classification problems. We explore the question of whether the specifi…
SONIA optimizes machine learning problems with a novel algorithm.
problem Empirical risk minimization in machine learning.
method Symmetric Blockwise Truncated Optimization (SONIA) algorithm combining second-order and steepest descent steps.
result SONIA converges to stationary points in both convex and nonconvex cases.
In this paper we consider the steepest descent H−1-gradient flow of the length functional for immersed plane curves, known as the curve diffusion flow. It is known that under this flow there exist both initially immersed curves which develop at least one singularity in finite time and initially embedded curves whi…
A quantum generalization of Natural Gradient Descent is presented as part of a general-purpose optimization framework for variational quantum circuits. The optimization dynamics is interpreted as moving in the steepest descent direction with respect to the Quantum Information Geometry, corresponding to the real part of…
Analyzed a generative model framework through Wasserstein Gradient Flow.
problem Generative modeling challenges.
method Wasserstein Gradient Flow (WGF) interpretation of Drifting Models (GMD).
result Different algorithms correspond to specific limiting points of WGFs on various divergences.
New method optimizes SDE models using continuous-time gradient descent.
problem Optimizing over the stationary distribution of SDE models.
method Continuous-time stochastic gradient descent for SDE models.
result Asymptotic convergence to the direction of steepest descent.
We present here a new model and algorithm which performs an efficient Natural gradient descent for Multilayer Perceptrons. Natural gradient descent was originally proposed from a point of view of information geometry, and it performs the steepest descent updates on manifolds in a Riemannian space. In particular, we ext…
New data-driven Cartan connection tracks complex vascular structures.
problem Tracking complex vascular structures in multi-orientation images.
method Formulated a data-driven Cartan connection on M2 for geodesic tracking. result Improved geodesic tracking of vascular trees with globally optimal curves.
Paper accelerates Bayesian few-shot classification using mirror descent.
problem Non-conjugate inference in Bayesian few-shot classification.
method Integrates mirror descent-based variational inference into Gaussian process-based few-shot classification.
result Accelerated convergence and improved uncertainty quantification.
New optimizers control network width scaling, improving stability and transfer across different model sizes.
problem Designing stable optimizers for networks of varying widths.
method Interpreting optimizers as steepest descent under mean-normalized operator norms, enabling layerwise composability and width-independent bounds.
result New optimizers like row normalization and column normalization provide stable learning-rate transfer across different model widths.
Scalable and robust TR decomposition for large-scale data with missing entries and outliers.
problem Handling large-scale tensor data with missing entries and outliers.
method Auto-weighted steepest descent method for missing entries and outliers identification, FGMC and RStS strategies.
result Outperforms existing TR decomposition methods in the presence of outliers and runs faster than robust tensor completion algorithms.
This paper presents a stochastic behavior analysis of a kernel-based stochastic restricted-gradient descent method. The restricted gradient gives a steepest ascent direction within the so-called dictionary subspace. The analysis provides the transient and steady state performance in the mean squared error criterion. It…
A new algorithm for training generative models using Sinkhorn divergence.
problem Training generative adversarial networks (GANs).
method Sinkhorn Natural Gradient (SiNG) algorithm for steepest descent on probability space.
result Explicit expression and efficient evaluation of the Sinkhorn information matrix (SIM).
A new algorithm for minimizing functions on Wasserstein space.
problem Discretization of continuous Wasserstein gradient flows in machine learning.
method Forward-Backward discretization scheme for minimizing functions with smooth and nonsmooth components.
result The FB scheme converges similarly to proximal gradient algorithms in Euclidean spaces.
New algorithm optimizes nonlinear SDEs online with convergence guarantees.
problem Optimizing nonlinear stochastic differential equations (SDEs) is computationally challenging.
method Forward propagation algorithm that solves an SDE derived using forward differentiation.
result Convergence theorem for nonlinear dissipative SDEs with bounds on stochastic fluctuations.
New RL algorithms learn policies competitive with best in class without assuming optimal policy.
problem Agnostic RL where best policy not necessarily optimal.
method First-order optimization in non-Euclidean space, reducing to policy learning.
result Sample complexity upper bounds for three algorithms under VGD condition.
A new method improves Bayesian filtering in nonlinear systems.
problem Bayesian filtering in nonlinear dynamical systems with non-Gaussian posteriors.
method Transport maps with block-triangular structure and gradient flows for MMD minimization.
result Accurate approximation of non-Gaussian posteriors without particle collapse.
AdamW optimizes a constrained loss with ℓ∞ norm constraint.
problem Understanding the optimization behavior of AdamW with ℓ∞ norm constraint. method Analyzing AdamW as a smoothed version of SignGD and connecting it to Frank-Wolfe optimization.
result AdamW implicitly performs constrained optimization with ℓ∞ norm constraint. The paper studies ideal flows of closed curves, classifying critical points and proving flow behavior.
problem Analyzing the generalised ideal flow of closed planar curves.
method Completely classifies critical points and proves properties of the m-ideal flow. result For m>1, the m-ideal flow of closed curves converges to a round multiply-covered circle. We revisit the choice of SGD for training deep neural networks by reconsidering the appropriate geometry in which to optimize the weights. We argue for a geometry invariant to rescaling of weights that does not affect the output of the network, and suggest Path-SGD, which is an approximate steepest descent method with …
The affine Grassmannian is a noncompact smooth manifold that parameterizes all affine subspaces of a fixed dimension. It is a natural generalization of Euclidean space, points being zero-dimensional affine subspaces. We will realize the affine Grassmannian as a matrix manifold and extend Riemannian optimization algorit…
A new method de-randomizes MCMC dynamics using the Stein operator.
problem Estimating complex target distributions in Bayesian inference.
method De-randomized kernel-based particle samplers that discretize the fiber-gradient Hamiltonian flow.
result GSVGD de-randomizes complex MCMC dynamics, maintaining high sample quality.