On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count % of corpus

21 results for “SignSGD”

Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effectsstat.ML

SignSGD analysis quantifies its effects in high dimensions.

problem Understanding signSGD's effects in high-dimensional settings.

method High-dimensional analysis of signSGD, deriving SDE and ODE for risk.

result Quantification of signSGD's effects: effective learning rate, noise compression, diagonal preconditioning, gradient noise reshaping.

signSGD high dimensions risk curves preconditioning noise compression

2024-11-19abs ↗pdf ↗

SignSGD: Fault-Tolerance to Blind and Byzantine Adversariescs.LG

SignSGD improves distributed learning by tolerating faulty devices, including Byzantine adversaries.

problem Faulty devices, including Byzantine adversaries, disrupt convergence in distributed learning.

method Developed SignSGD from SGD, proving its robustness to Byzantine adversaries with upper bound convergence rate.

result SignSGD converges in the presence of Byzantine adversaries, providing a fault-tolerant solution.

distributed learning fault tolerance SGD Byzantine adversaries convergence rate

2022-02-04abs ↗pdf ↗

Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?cs.LG

SignSGD outperforms SGD in linear regression with optimal scaling laws under PLRF model.

problem Improving linear regression performance with signSGD under power-law random features.

method Analysis of signSGD risk under PLRF model, comparison with SGD, identification of unique effects.

result SignSGD can have a steeper compute-optimal slope than SGD in noisy regimes, especially with WSD schedule.

signSGD linear regression scaling laws power-law random features noise-reshaping

2026-03-02abs ↗pdf ↗

SignSVRG: fixing SignSGD via variance reductionmath.OC

SignSVRG improves SignSGD by reducing variance, achieving similar convergence rates.

problem Minimizing finite sums of convex and Lipschitz functions.

method Incorporates variance reduction techniques into SignSGD.

result Achieves convergence rates of

\mathcal{O}(1 / \sqrt{T})

for expected norm of the gradient and

\mathcal{O}(1/T)

for smooth convex functions.

Variance Reduction SignSGD Convergence Rates Finite Sums Convex Functions

2023-05-22abs ↗pdf ↗

Phases of Muon: When Muon Eclipses SignSGDmath.OC

New analysis of Muon and SignSGD on matrix-valued least squares problems.

problem Understanding the behavior of Muon and SignSGD on matrix-valued least squares problems.

method Derive explicit deterministic dynamics to study learning behavior of Muon and SignSGD.

result Muon and SignSGD exhibit different optimal learning rates and convergence characteristics based on batch size and data covariance.

Muon SignSGD Matrix-valued least squares Learning dynamics Batch size

2026-05-10abs ↗pdf ↗

Error Feedback Fixes SignSGD and other Gradient Compression Schemescs.LG

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD does not converge to the optimum. Further, even when it does conver…

2019-01-28abs ↗pdf ↗

Sparse-SignSGD with Majority Vote for Communication-Efficient Distributed Learningcs.LG

New algorithm reduces communication costs in distributed deep learning.

problem High communication costs in distributed deep learning.

method Sparse-SignSGD with Majority Vote (S3GD-MV).

result Significantly reduces communication costs while maintaining accuracy.

distributed learning communication efficiency sparse-SignSGD majority vote deep learning

2023-02-15abs ↗pdf ↗

Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimizationcs.LG

New analysis of signSGD with random reshuffling shows faster convergence rates.

problem Understanding the convergence of signSGD with random reshuffling in nonconvex optimization.

method Developed new sign-based algorithms (SignRVR, SignRVM) and analyzed convergence rates.

result Achieved faster convergence rates for signSGD with random reshuffling.

signSGD random reshuffling nonconvex optimization convergence analysis variance-reduced gradients

2023-10-24abs ↗pdf ↗

The Geometry of Sign Gradient Descentcs.LG

Unified signSGD and gradient descent analysis for neural networks.

problem Performance of sign-based optimization methods in neural networks.

method Unified analysis of separable smoothness and

\ell_\infty

-smoothness, isolating geometric properties affecting performance.

result Sign-based methods are preferable over gradient descent under specific Hessian properties in deep networks.

signSGD gradient descent neural networks smoothness steepest descent

2020-02-19abs ↗pdf ↗

Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithmscs.LG

Recently, there is a growing interest in the study of median-based algorithms for distributed non-convex optimization. Two prominent such algorithms include signSGD with majority vote, an effective approach for communication reduction via 1-bit compression on the local gradients, and medianSGD, an algorithm recently pr…

2019-06-04abs ↗pdf ↗

Stochastic-Sign SGD for Federated Learning with Theoretical Guaranteescs.LG

Proposes Stochastic-Sign SGD for federated learning with theoretical guarantees.

problem Developing efficient, private, and resilient parameter estimation methods for federated learning.

method Introduces Stochastic-Sign SGD, a novel method based on SIGNSGD, which addresses convergence and communication efficiency.

result Demonstrates the effectiveness of Stochastic-Sign SGD through experiments on MNIST and CIFAR-10 datasets.

federated learning SGD quantization convergence privacy

2020-02-25abs ↗pdf ↗

Communication Efficient Federated Learning with Energy Awareness over Wireless Networkscs.LG

This work improves communication efficiency in federated learning over wireless networks by optimizing energy consumption.

problem Optimizing energy consumption in federated learning over wireless networks.

method Adopting SignSGD for gradient sign exchange, considering channel capacity with outage, and proposing a stochastic sign-based algorithm for uneven data distribution.

result Proposed methods achieve a balance between learning performance and energy consumption.

Federated Learning Wireless Networks Energy Efficiency SignSGD Channel Capacity

2020-04-15abs ↗pdf ↗

CLion: Efficient Cautious Lion Optimizer with Enhanced Generalizationcs.LG

Enhanced Lion optimizer CLion improves generalization with lower error.

problem Lion optimizer's generalization analysis is lacking.

method Algorithmic stability analysis and cautious sign function use.

result CLion has a lower generalization error of

O(\frac{1}{N})

Lion optimizer generalization algorithmic stability Cautious Lion SignSGD

2026-04-16abs ↗pdf ↗

Adaptive Strategies in Non-convex Optimizationcs.LG

Adaptive algorithms improve performance in non-convex optimization across various scenarios.

problem Improper handling of noise scales, gradient magnitudes, and smoothness in non-convex optimization.

method Design and analysis of noise-adaptive, scale-free, and generalized algorithms.

result Adaptive algorithms achieve optimal rates and performance in diverse optimization settings.

Adaptive Algorithms Non-convex Optimization Gradient Noise Deep Learning BatchNorm

2023-06-17abs ↗pdf ↗

Distributed Learning with Compressed Gradient Differencescs.LG

Training large machine learning models requires a distributed computing approach, with communication of the model updates being the bottleneck. For this reason, several methods based on the compression (e.g., sparsification and/or quantization) of updates were recently proposed, including QSGD (Alistarh et al., 2017), …

2019-01-26abs ↗pdf ↗

Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedbackcs.LG

Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote …

2019-05-27abs ↗pdf ↗

Fundamental Limits of Communication Efficiency for Model Aggregation in Distributed Learning: A Rate-Distortion Approachcs.IT

Paper studies fundamental limits of communication in distributed learning.

problem Communication efficiency in model aggregation for distributed learning.

method Rate-Distortion approach to model aggregation as a vector Gaussian CEO problem.

result Derives rate region bound and sum-rate-distortion function for model aggregation.

distributed learning model aggregation communication efficiency rate-distortion gradient compression

2022-06-28abs ↗pdf ↗

Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcs.LG

The paper analyzes and improves a deep learning optimization technique using matrix gradient orthogonality.

problem Improving deep learning training through more effective optimization methods.

method Develops a stochastic non-Euclidean trust-region gradient method for deep learning optimization.

result Proves state-of-the-art convergence results for the proposed algorithm in various scenarios.

Deep Learning Optimization Trust-Region Matrix Gradient Orthogonalization Non-Euclidean Norms

2025-03-16abs ↗pdf ↗

The Implicit Bias of Steepest Descent with Mini-batch Stochastic Gradientcs.LG

This work studies the implicit bias of mini-batch SGD in classification.

problem Understanding the implicit bias of mini-batch SGD in multi-class classification.

method Characterizes how batch size, momentum, and variance reduction affect convergence and max-margin behavior under different norms.

result Momentum enables small-batch convergence to an approximate max-margin solution, while variance reduction recovers the exact full-batch bias.

stochastic gradient descent mini-batch steepest descent classification momentum

2026-02-12abs ↗pdf ↗

z

-SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learningcs.LG

Unified sign-based compression for federated learning with faster convergence.

problem High communication cost in federated learning with large-scale models.

method Unified noisy perturbation scheme for sign-based compression.

result Achieves faster convergence rate than existing sign-based methods.

Federated Learning Sign-based Compression Gradient Compression Convergence Rate Communication Cost

2023-02-06abs ↗pdf ↗

On the Interaction of Batch Noise, Adaptivity, and Compression, under

(L_0,L_1)

-Smoothness: An SDE Approachcs.LG

Unified framework for distributed compressed SGD under $(L_0, L_1)$ -smoothness.

problem Understanding the joint effect of batch noise, adaptivity, and compression in distributed stochastic optimization.

method Developed a unified theoretical framework using SDEs that incorporate curvature-dependent terms.

result Normalizing updates in DCSGD stabilizes convergence, with normalization degree determined by noise structure and landscape regularity.

distributed optimization stochastic gradient descent compression SDE $(L_0

2025-05-30abs ↗pdf ↗