On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,657 papers · 148 categories

Trend · papers per month

count % of corpus

4 results for “SignGD”

On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descentcs.LG

Study on SignGD optimization of two-layer transformer on noisy data.

problem Understanding how SignGD optimizes transformers and its generalization.

method Analysis of a two-layer transformer with SignGD on a linearly separable noisy dataset.

result SignGD converges fast but has poor generalization on noisy data.

Transformer Optimization SignGD Adam Generalization Two-Layer Model

2024-10-07abs ↗pdf ↗

A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscs.LG

The study examines the dynamic behavior of RMSprop and Adam algorithms.

problem Understanding the training loss curve patterns of adaptive gradient algorithms.

method Careful numerical experiments and theoretical explanations using the signGD flow.

result Adam converges smoother and faster when momentum factors are close to each other.

adaptive gradient algorithms RMSprop Adam training loss curve signGD flow

2020-09-14abs ↗pdf ↗

Implicit Bias of AdamW:

\ell_\infty

Norm Constrained Optimizationcs.LG

AdamW optimizes a constrained loss with $\ell_\infty$ norm constraint.

problem Understanding the optimization behavior of AdamW with

\ell_\infty

norm constraint.

method Analyzing AdamW as a smoothed version of SignGD and connecting it to Frank-Wolfe optimization.

result AdamW implicitly performs constrained optimization with

\ell_\infty

norm constraint.

AdamW Implicit Bias Constrained Optimization Frank-Wolfe SignGD

2024-04-05abs ↗pdf ↗

Muon in Associative Memory Learning: Training Dynamics and Scaling Lawscs.LG

Muon outperforms GD in associative memory learning by balancing frequency components.

problem Training dynamics and scaling behavior of Muon in associative memory learning.

method Study of Muon in a linear associative memory model with softmax retrieval and hierarchical frequency spectrum over query-answer pairs.

result Muon achieves exponential speedup over GD in noiseless case and superior scaling efficiency in noisy case.

Muon Gradient Descent Associative Memory Matrix Preconditioner Frequency Spectrum

2026-02-05abs ↗pdf ↗