A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Two major momentum-based techniques that have achieved tremendous success in optimization are Polyak's heavy ball method and Nesterov's accelerated gradient. A crucial step in all momentum-based methods is the choice of the momentum parameter m which is always suggested to be set to less than 1. Although the choice…
We take a Hamiltonian-based perspective to generalize Nesterov's accelerated gradient descent and Polyak's heavy ball method to a broad class of momentum methods in the setting of (possibly) constrained minimization in Euclidean and non-Euclidean normed vector spaces. Our perspective leads to a generic and unifying non…
Study on stochastic approximation with Polyak-Ruppert averaging for linear systems.
problem Understanding the asymptotic and non-asymptotic properties of stochastic approximation procedures.
method Detailed analysis of linear stochastic approximation with Polyak-Ruppert averaging, focusing on asymptotic and non-asymptotic properties.
result Proves CLT and non-asymptotic concentration inequality for averaged iterates, providing refined understanding of linear stochastic approximation.
SGDM accelerates faster than SGD with large batch sizes and permits broader learning rates.
problem Understanding the role of momentum in SGDM and its convergence rates.
method Analysis of SGDM convergence rates under strongly convex settings, including finite-sample rates and asymptotic normality of the averaged estimator.
result SGDM converges faster than SGD with large batch sizes and permits broader learning rates.
Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dyna…
Gradient descent-based optimization methods underpin the parameter training of neural networks, and hence comprise a significant component in the impressive test results found in a number of applications. Introducing stochasticity is key to their success in practical problems, and there is some understanding of the rol…
Momentum methods such as Polyak's heavy ball (HB) method, Nesterov's accelerated gradient (AG) as well as accelerated projected gradient (APG) method have been commonly used in machine learning practice, but their performance is quite sensitive to noise in the gradients. We study these methods under a first-order stoch…
We use matrix iteration theory to characterize acceleration in smooth games. We define the spectral shape of a family of games as the set containing all eigenvalues of the Jacobians of standard gradient dynamics in the family. Shapes restricted to the real line represent well-understood classes of problems, like minimi…
Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution. With this perspective, we derive several new results. (1) We show that constant SGD can be used as an approximate Bayesian posterior inference algorithm. Specifically, we show how to adjust …
New adaptive scheduler improves SAM for better model training.
problem Training machine learning models requires selecting a learning rate, which is often difficult and time-consuming.
method Derive Polyak schedulers tailored to SAM-style updates, proving linear convergence for strongly convex objectives and an O(1/T) rate for convex objectives.
result Polyak schedulers achieve comparable or better performance than tuned SAM baselines, reducing the need for learning-rate tuning.
We describe the space of arrow diagram formulas for virtual knot diagrams in the annulus as the kernel of a linear map, inspired from a conjecture due to M. Polyak. As a main application, we slightly improve Grishanov-Vassiliev's theorem for planar chain invariants.
In this paper, it is shown that there are no nonconstant Goussarov-Polyak-Viro finite-type invariants that are invariant under the virtualization move. As an immediate corollary, we obtain the theorem which states none of the Birman coefficients of the Jones-Kauffman polynomial are of GPV finite type.
We describe the Polyak-Viro arrow diagram formulas for the coefficients of the Conway polynomial. As a consequence, we obtain the Conway polynomial as a state sum over some subsets of the crossings of the knot diagram. It turns out to be a simplification of a special case of Jaeger's state model for the HOMFLY polynomi…
We enhance the biquandle counting invariant using elements of truncated biquandle-labeled Polyak algebras. These finite type enhancements reduce to the finite type enhancements defined by Goussarov, Polyak and Viro for the trivial biquandle of one element and determine (but are not determined by) the biquandle counting…
We present a new method to produce simple formulas for 1-cocycles of knots over the integers, inspired by Polyak-Viro's formulas for finite-type knot invariants. We conjecture that these formulas always represent finite-type cohomology classes in the sense of Vassiliev. An example of degree 3 is studied, and shown to c…
Adaptive gradient methods (AGMs) have become popular in optimizing the nonconvex problems in deep learning area. We revisit AGMs and identify that the adaptive learning rate (A-LR) used by AGMs varies significantly across the dimensions of the problem over epochs (i.e., anisotropic scale), which may lead to issues in c…