On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

169,051 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “SGD-based algorithms”

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscs.LG

Cluster-GCN efficiently trains deep GCNs on large graphs without memory or computational constraints.

problem Training large-scale GCNs is computationally and memory-intensive.

method Cluster-GCN exploits graph clustering to restrict neighborhood search to dense subgraphs, reducing memory and computational requirements.

result Cluster-GCN achieves comparable test accuracy to previous algorithms while being faster and using less memory.

graph convolutional networks graph clustering memory efficiency large-scale training SGD-based algorithms

2019-05-20abs ↗pdf ↗

OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterationscs.LG

OptEx accelerates first-order optimization with parallelized iterations.

problem Inefficiencies in first-order optimization algorithms for complex tasks.

method Approximately parallelized iterations using kernelized gradient estimation.

result OptEx achieves substantial efficiency improvements with an effective acceleration rate of

Ω(\sqrt{N})

first-order optimization parallel computing gradient estimation machine learning neural networks

2024-02-18abs ↗pdf ↗

Dual SVM Training on a Budgetcs.LG

We present a dual subspace ascent algorithm for support vector machine training that respects a budget constraint limiting the number of support vectors. Budget methods are effective for reducing the training time of kernel SVM while retaining high accuracy. To date, budget training is available only for primal (SGD-ba…

2018-06-26abs ↗pdf ↗

Provably Faster Algorithms for Bilevel Optimizationcs.LG

New algorithms reduce bilevel optimization complexity to ε^(-1.5).

problem Efficiently solving bilevel optimization problems in machine learning.

method Proposed two new algorithms: one using momentum-based recursive iterations, the other using recursive gradient estimations.

result Achieved computational complexity of ε^(-1.5), significantly faster than previous methods.

bilevel optimization machine learning hyperparameter optimization momentum recursive iterations

2021-06-08abs ↗pdf ↗

Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscs.LG

Continuous-time analysis shows SGD with noise prefers flat minima.

problem Optimizing neural networks using SGD with noise.

method Continuous-time model for SGD with noise analysis.

result Optimization prefers flat minima in certain noise regimes.

neural networks stochastic gradient descent continuous time flat minima machine learning

2021-06-04abs ↗pdf ↗

Poly-time universality and limitations of deep learningcs.LG

This paper explores the limits of deep learning in poly-time.

problem Characterizing function distributions that deep learning can or cannot learn efficiently.

method Analysis of SGD and GD-based deep learning approaches, proving universality and non-universality results.

result SGD-based deep learning is efficiently universal, while GD-based is not, especially with large batches.

deep learning poly-time universality non-universality SGD

2020-01-07abs ↗pdf ↗

Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networkscs.LG

ESGD optimizes deep neural networks using evolutionary and stochastic gradient descent methods.

problem Optimizing deep neural networks efficiently and effectively.

method Combines SGD and evolutionary algorithms, alternating between steps to improve population fitness.

result Guarantees that the best fitness in the population never degrades and optimizes across various deep architectures.

Evolutionary Algorithms Deep Neural Networks Optimization Stochastic Gradient Descent Population-Based

2018-10-16abs ↗pdf ↗

Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithmcs.LG

New analysis shows D-SGD can generalize well regardless of graph connectivity.

problem Improving generalization of D-SGD in decentralized settings.

method Algorithmic stability analysis and optimization-dependent generalization bounds.

result D-SGD can achieve generalization bounds similar to classical SGD, independent of graph connectivity.

Decentralized SGD Generalization Stability Graph Connectivity Convex Optimization

2023-06-05abs ↗pdf ↗

Replicable Learning of Large-Margin Halfspacescs.LG

Efficient algorithms improve learning of large-margin halfspaces.

problem Learning large-margin halfspaces efficiently and reproducibly.

method Design of efficient, dimension-independent, polynomial-time algorithms; SGD-based approach; DP-to-Replicability reduction.

result Improved sample complexity compared to previous algorithms, with optimal sample complexity for one algorithm.

learning halfspaces replicability polynomial-time sample complexity

2024-02-21abs ↗pdf ↗

An ADRC-Incorporated Stochastic Gradient Descent Algorithm for Latent Factor Analysiscs.LG

An ADRC-incorporated SGD algorithm improves latent factor analysis speed and accuracy.

problem Slow convergence in standard SGD for HDI matrix analysis.

method Incorporates ADRC principles to refine historical and future learning error states.

result Empirically outperforms state-of-the-art LFA models in HDI matrix prediction.

latent factor analysis stochastic gradient descent ADRC HDI matrix fast convergence

2024-01-13abs ↗pdf ↗

Online Covariance Matrix Estimation in Stochastic Gradient Descentstat.ML

Paper proposes an online estimator for covariance matrix of SGD iterates.

problem Quantifying variability and randomness of SGD-based estimates in online learning.

method Proposes a fully online estimator for covariance matrix of ASGD using SGD iterates.

result Establishes consistency of the online estimator and shows comparable convergence rate to offline methods.

stochastic gradient descent covariance matrix online learning statistical inference SGD iterates

2020-02-10abs ↗pdf ↗

ROOT-SGD: Sharp Nonasymptotics and Near-Optimal Asymptotics in a Single Algorithmmath.OC

ROOT-SGD solves convex optimization problems with optimal nonasymptotic and near-optimal asymptotic performance.

problem Solving strongly convex and smooth unconstrained optimization problems using stochastic first-order algorithms.

method ROOT-SGD: Recursive One-Over-T SGD, averaging past stochastic gradients.

result Achieves state-of-the-art performance in both nonasymptotic and asymptotic senses.

ROOT-SGD convex optimization stochastic algorithms sharp nonasymptotics near-optimal asymptotics

2020-08-28abs ↗pdf ↗

Stagewise Enlargement of Batch Size for SGD-based Learningstat.ML

A new method SEBS optimizes SGD batch size for better performance.

problem Optimizing batch size for SGD to balance training speed and generalization.

method SEBS method uses a multi-stage geometric batch size enlargement scheme.

result SEBS reduces parameter updates without increasing generalization error.

batch size SGD generalization optimization multi-stage

2020-02-26abs ↗pdf ↗

AdamNX: An Adam improvement algorithm based on a novel exponential decay mechanism for the second-order moment estimatecs.LG

AdamNX improves Adam's stability by adjusting its learning rate.

problem Adam's tendency to converge to non-flat minima in large-scale models.

method Proposes a novel exponential decay mechanism for Adam's second-order moment estimate.

result AdamNX outperforms Adam and its variants in stability and performance.

Adam optimization stability momentum exponential decay

2025-11-17abs ↗pdf ↗

Analysis of Biased Stochastic Gradient Descent Using Sequential Semidefinite Programsmath.OC

We present a convergence rate analysis for biased stochastic gradient descent (SGD), where individual gradient updates are corrupted by computation errors. We develop stochastic quadratic constraints to formulate a small linear matrix inequality (LMI) whose feasible points lead to convergence bounds of biased SGD. Base…

2017-11-03abs ↗pdf ↗

Approximate Random Dropoutcs.LG

The training phases of Deep neural network~(DNN) consumes enormous processing time and energy. Compression techniques utilizing the sparsity of DNNs can effectively accelerate the inference phase of DNNs. However, it can be hardly used in the training phase because the training phase involves dense matrix-multiplicatio…

2018-05-23abs ↗pdf ↗

Statistical inference using SGDcs.LG

We present a novel method for frequentist statistical inference in $M$ -estimation problems, based on stochastic gradient descent (SGD) with a fixed step size: we demonstrate that the average of such SGD sequences can be used for statistical inference, after proper scaling. An intuitive analysis using the Ornstein-Uhlen…

2017-05-21abs ↗pdf ↗

Bolt-on Differential Privacy for Scalable Stochastic Gradient Descent-based Analyticscs.LG

While significant progress has been made separately on analytics systems for scalable stochastic gradient descent (SGD) and private SGD, none of the major scalable analytics frameworks have incorporated differentially private SGD. There are two inter-related issues for this disconnect between research and practice: (1)…

2016-06-15abs ↗pdf ↗

Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysisstat.ML

SGD with machine learning noise converges to global minimum exponentially fast.

problem Optimizing machine learning models with stochastic gradient descent.

method Analysis of SGD with machine learning noise, focusing on energy landscapes and gradient noise.

result SGD converges to the global minimum exponentially fast under certain conditions.

stochastic gradient descent machine learning energy landscapes Lojasiewicz inequality exponential convergence

2021-05-04abs ↗pdf ↗

Stochastic Optimization in Semi-Discrete Optimal Transport: Convergence Analysis and Minimax Ratemath.ST

Stochastic optimization improves semi-discrete OT map estimation with a minimax rate.

problem Empirical success of SGD in semi-discrete OT, but lack of theoretical guarantees.

method Averaged projected SGD with a minimax convergence rate of O(1/√n).

result SGD methods can estimate the OT map with a minimax convergence rate of O(1/√n).

Stochastic Optimization Optimal Transport Semi-Discrete Minimax Rate Convergence Analysis

2025-10-29abs ↗pdf ↗

Sampling Approximately Low-Rank Ising Models: MCMC meets Variational Methodscs.DS

New algorithm samples from Ising models efficiently, even with outliers.

problem Sampling from Ising models with general interaction matrices.

method Combines MCMC and variational inference techniques.

result First polynomial time sampling algorithms for low-rank Ising models.

Ising models MCMC Variational inference Polynomial time sampling Low-rank models

2022-02-17abs ↗pdf ↗

What Information Does a ResNet Compress?cs.LG

ResNet learns to compress information during training.

problem Applying information bottleneck principle to deep learning models.

method Trained PixelCNN++ models to measure mutual information between ResNet hidden layers and input data.

result Compression occurs in both classification and autoencoding training regimes.

ResNet information bottleneck deep learning mutual information compression

2020-03-13abs ↗pdf ↗

Statistical Inference with Stochastic Gradient Methods under

φ

-mixing Datastat.ME

A new method for statistical inference using SGD under $φ$ -mixing data.

problem Valid statistical inference for time series data with general correlation.

method Proposes a mini-batch SGD estimator and associated mini-batch bootstrap procedure for

φ

-mixing data.

result The proposed method constructs valid confidence intervals for

φ

-mixing data.

stochastic gradient descent statistical inference φ-mixing mini-batch bootstrap

2023-02-24abs ↗pdf ↗

Rennala MVR: Improved Time Complexity for Parallel Stochastic Optimization via Momentum-Based Variance Reductionmath.OC

Improved time complexity for parallel stochastic optimization in heterogeneous systems.

problem Time complexity in parallel stochastic optimization for large-scale machine learning models.

method Proposes Rennala MVR, a variance-reduced extension of Rennala SGD based on momentum-based variance reduction.

result Variance reduction improves time complexity in relevant parameter regimes for parallel stochastic optimization in heterogeneous systems.

parallel optimization time complexity heterogeneous systems variance reduction momentum

2026-05-09abs ↗pdf ↗

Near-optimal Offline and Streaming Algorithms for Learning Non-Linear Dynamical Systemscs.LG

We develop algorithms to learn non-linear dynamical systems without mixing assumptions.

problem Learning non-linear dynamical systems from dependent data.

method We introduce an offline algorithm and a one-pass streaming method with SGD-RER.

result Our methods achieve optimal or near-optimal performance for learning non-linear systems.

non-linear dynamical systems learning algorithms dependent data mixing assumption SGD-RER

2021-05-24abs ↗pdf ↗

A theory of learning with constrained weight-distributionq-bio.NC

Theory of learning with weight-distribution constraints.

problem Understanding how structure influences function in neural networks.

method Statistical mechanical theory and optimal transport.

result Reduction in capacity due to constrained weight-distribution is related to Wasserstein distance.

neural networks learning weight distribution statistical mechanics optimal transport

2022-06-14abs ↗pdf ↗

Efficient Distributed Semi-Supervised Learning using Stochastic Regularization over Affinity Graphsstat.ML

We describe a computationally efficient, stochastic graph-regularization technique that can be utilized for the semi-supervised training of deep neural networks in a parallel or distributed setting. We utilize a technique, first described in [13] for the construction of mini-batches for stochastic gradient descent (SGD…

2016-12-15abs ↗pdf ↗

Quantized Epoch-SGD for Communication-Efficient Distributed Learningcs.LG

Quantized Epoch-SGD reduces communication in distributed learning.

problem High communication cost in SGD-based distributed learning.

method Quantizes parameters with variance reduction for efficient distributed learning.

result QESGD achieves better performance with less communication compared to other methods.

distributed learning quantized SGD communication efficiency variance reduction

2019-01-10abs ↗pdf ↗

Online Distributionally Robust LLM Alignment via Regression to Relative Rewardcs.LG

DRO-REBEL improves LLM alignment by robustly updating models online.

problem Overfitting and drifting of LLMs during RLHF.

method DRO-REBEL uses type-

p

Wasserstein, KL, and

χ^2

ambiguity sets for robust online updates.

result DRO-REBEL achieves faster convergence and better performance than prior methods.

Reinforcement Learning Human Feedback Large Language Models Distributionally Robust Optimization Online Learning

2025-09-23abs ↗pdf ↗

An Efficient Variant of One-Class SVM with Lifelong Online Learning Guaranteesstat.ML

SONAR improves outlier detection for streaming data with strong theoretical guarantees.

problem Outlier detection for non-stationary streaming data with high Type I/II errors.

method SONAR is an efficient SGD-based OCSVM solver with strong convex regularization and lifelong learning guarantees.

result SONAR outperforms traditional OCSVM in Type I/II error rates under non-stationary data.

outlier detection streaming data SGD OCSVM lifelong learning

2025-12-11abs ↗pdf ↗

Communication-Efficient Adaptive Federated Learningcs.LG

A new federated learning method reduces communication costs and improves adaptivity.

problem Large communication overhead and lack of adaptivity in federated learning.

method FedCAMS: A novel communication-efficient adaptive federated learning method with theoretical guarantees.

result FedCAMS achieves the same convergence rate as non-compressed federated learning methods.

federated learning communication efficiency adaptive optimization federated adaptive convergence analysis

2022-05-05abs ↗pdf ↗

SGD with Dependent Data: Optimal Estimation, Regret, and Inferencemath.ST

SGD performs optimally on dependent data, achieving both estimation and regret.

problem Performance of SGD on temporally dependent data.

method Stochastic gradient descent with martingale-type and sequential dependence.

result SGD achieves optimal estimation and regret under various dependence structures.

SGD dependent data martingale regret estimation

2026-01-04abs ↗pdf ↗

On Scalable Inference with Stochastic Gradient Descentstat.ML

In many applications involving large dataset or online updating, stochastic gradient descent (SGD) provides a scalable way to compute parameter estimates and has gained increasing popularity due to its numerical convenience and memory efficiency. While the asymptotic properties of SGD-based estimators have been establi…

2017-07-01abs ↗pdf ↗

Learning Rate Schedules in the Presence of Distribution Shiftcs.LG

Optimal learning rate schedules for SGD in changing data distributions.

problem Minimizing regret in online learning with changing data distributions.

method Characterized optimal schedules for linear regression, proposed schedules for general convex and non-convex losses, and defined a notion of regret for non-convex losses.

result Upper and lower bounds for regret with constants for convex losses, and an upper bound on total expected regret for non-convex losses.

learning rate schedules regret minimization distribution shift online learning stochastic differential equations

2023-03-27abs ↗pdf ↗

Stability of SGD: Tightness Analysis and Improved Boundscs.LG

Paper improves stability analysis of SGD for various loss functions and data distributions.

problem Improving stability analysis of SGD for non-convex loss functions and data distributions.

method Analyzes stability of SGD for convex and non-convex loss functions, and improves data-dependent bounds.

result Improved stability bounds for non-convex loss functions and convex regularized loss functions.

Stochastic Gradient Descent Generalization Loss Functions Data Distribution Convex Optimization

2021-02-10abs ↗pdf ↗

Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcs.LG

Neural network learns low-dimensional polynomials with SGD near information-theoretic limit.

problem Learning a single-index target function with gradient descent.

method Two-layer neural network optimized by SGD on squared loss.

result Sample and runtime complexity of

n \simeq T = Θ(d\!\cdot\! \mathrm{polylog} d)

for polynomial single-index models, matching information theoretic limit up to polylogarithmic factors.

Neural Networks Gradient Descent Polynomial Models Information Exponent Sample Complexity

2024-06-03abs ↗pdf ↗

HiGrad: Uncertainty Quantification for Online Learning and Stochastic Approximationstat.ML

Stochastic gradient descent (SGD) is an immensely popular approach for online learning in settings where data arrives in a stream or data sizes are very large. However, despite an ever-increasing volume of work on SGD, much less is known about the statistical inferential properties of SGD-based predictions. Taking a fu…

2018-02-13abs ↗pdf ↗

On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthstat.ML

Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly und…

2018-07-13abs ↗pdf ↗

Stochastic Gradient Descent in the Optimal Control of Execution Costsq-fin.TR

Study uses SGD to find near-optimal execution cost policies in dynamic markets.

problem Finding optimal execution cost policies in complex markets.

method Stochastic Gradient Descent (SGD) approach to derive near-optimal policies.

result SGD-based policies offer valuable insights and are implementable in volatile markets.

Stochastic Gradient Descent Optimal Execution Costs Market Dynamics Institutional Investing Price Dynamics

2024-12-14abs ↗pdf ↗

Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structurecs.LG

Centripetal SGD prunes deep CNNs by making filters collapse.

problem Pruning deep CNNs with complex structures.

method Centripetal SGD, a novel optimization method.

result Pruning deep CNNs without performance loss.

CNN pruning filter collapse Centripetal SGD complicated structure redundancy

2019-04-08abs ↗pdf ↗

A Hybrid Approach Towards Two Stage Bengali Question Classification Utilizing Smart Data Balancing Techniquecs.CL

Hybrid QC system for Bengali questions using smart data balancing.

problem Classifying factoid questions in Bengali.

method Two-stage approach with 1D CNN for coarse classification and SGD for fine classification.

result Effectiveness of smart data balancing technique in improving classification accuracy.

Bengali Question Classification Hybrid Approach Smart Data Balancing Convolutional Neural Network

2019-11-30abs ↗pdf ↗

AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networkscs.LG

AngularGrad optimizes CNNs by considering gradient direction, improving convergence.

problem Dying gradient problem and inefficiency in exploiting gradient curvature.

method AngularGrad considers gradient direction/angle, generating a score for step size control.

result AngularGrad outperforms state-of-the-art optimizers in benchmark tests.

Convolutional Neural Networks Optimization Gradient Descent Angular Information Adam

2021-05-21abs ↗pdf ↗

Global Sparse Momentum SGD for Pruning Very Deep Neural Networkscs.LG

Global compression method prunes deep neural networks efficiently.

problem Efficiently compress deep neural networks for resource-constrained devices.

method Global Sparse Momentum SGD for on-the-fly pruning.

result Automatically finds appropriate per-layer sparsity ratios without re-training.

Deep Neural Networks Compression Pruning Momentum SGD Resource-Constrained Devices

2019-09-27abs ↗pdf ↗

Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descentcs.LG

Paper introduces elastic consistency for distributed SGD, enabling convergence analysis.

problem Training large-scale machine learning models in distributed environments.

method Introduces elastic consistency as a general consistency model for distributed SGD.

result Derives convergence bounds for various distributed SGD methods.

distributed SGD consistency model convergence analysis stochastic gradient descent machine learning

2020-01-16abs ↗pdf ↗

Large deviations rates for stochastic gradient descent with strongly convex functionscs.LG

Study large deviations rates for SGD with strongly convex functions.

problem High probability metrics with SGD.

method Large deviations theory, generic gradient noise, strongly convex functions.

result Upper large deviations bound for SGD with strongly convex functions.

stochastic gradient descent large deviations strongly convex functions high probability metrics gradient noise

2022-11-02abs ↗pdf ↗

Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learningcs.IT

This paper studies communication efficiency in federated learning by optimizing the sum-rate-distortion function for indirect multiterminal source coding.

problem Indirect multiterminal source coding in federated learning where edge devices send noisy gradients to the server.

method Analyzes the rate region for the quadratic vector Gaussian CEO problem under unbiased estimator and derives an explicit formula for the sum-rate-distortion function.

result Derives an explicit formula for the sum-rate-distortion function in the special case of identical gradients over edge devices.

federated learning multiterminal source coding CEO problem gradient variance sum-rate-distortion function

2021-01-21abs ↗pdf ↗

Stopping Rules for Stochastic Gradient Descent via Anytime-Valid Confidence Sequencesmath.OC

Develops anytime-valid stopping rules for SGD based on observed trajectory.

problem Stopping stochastic gradient descent (SGD) based on observed trajectory.

method Develops anytime-valid confidence sequences for stochastic gradient methods.

result Statistically valid, time-uniform stopping rules for SGD across convex and nonconvex settings.

stochastic gradient descent confidence sequences stopping rules online monitoring suboptimality

2025-12-15abs ↗pdf ↗

Statistical Inference for Linear Functionals of Online Least-squares SGD when

t \gtrsim d^{1+δ}

cs.LG

New bounds for SGD in high dimensions improve inference efficiency.

problem Quantifying uncertainty in high-dimensional SGD.

method Established non-asymptotic Berry--Esseen bounds for online least-squares SGD.

result Gaussian Central Limit Theorem holds for

t \gtrsim d^{1+δ}

, extending dimensional scaling.

Stochastic Gradient Descent Online Learning High-Dimensional Inference Berry--Esseen Bounds Central Limit Theorem

2025-10-22abs ↗pdf ↗