Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “robust gradient descent”
Paper introduces robust learning methods using coordinate gradient descent.
problem Supervised learning with corrupted features and labels.
method Coordinate gradient descent combined with robust estimators of partial derivatives.
result Robust learning methods with nearly identical numerical complexity to non-robust ones.
Implicit Regularization Makes Overparameterized Asymmetric Matrix Sensing Robust to Perturbationscs.LG
Paper shows robustness of gradient descent in matrix sensing despite perturbations.
problem Understanding robustness of gradient descent in matrix sensing.
method Developed perturbed gradient flow to capture noise and improve robustness.
result Gradient descent is robust to perturbations in matrix sensing.
Gradient descent solves robust mean estimation in high dimensions.
problem High-dimensional robust mean estimation in the presence of adversarial outliers.
method Gradient descent with a structural lemma showing near-optimal solutions.
result Gradient descent can solve the robust mean estimation problem directly.
Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networkscs.LG
Gradient descent biases neural networks to use an average of features, leading to non-robustness.
problem Non-robustness in neural networks due to feature averaging.
method Theoretical analysis and experiments on binary classification tasks.
result Gradient descent trains networks to rely on an average of features, making them vulnerable to adversarial attacks.
New approach improves model generalization through distributionally robust learning.
problem Improving model generalization in machine learning.
method Stochastic gradient descent applied to the outer minimization problem, with gradient estimation through multi-level Monte Carlo randomization.
result Our approach yields significant benefits over previous work in numerical experiments.
New method improves neural network robustness to adversarial attacks.
problem Improving adversarial robustness of neural networks.
method Inspired by adaptive control theory, the approach uses persistency of excitation to constrain gradient descent updates.
result Networks trained with the PoE-motivated learning rate schedule are significantly more robust to adversarial attacks.
Improved linear regression with privacy and robustness guarantees.
problem Private and robust linear regression with adversarial corruption.
method Differentially private stochastic gradient descent with full-batch gradient descent and adaptive clipping.
result Near optimal sample complexity for both private and robust linear regression.
PGD-trained models have a preferential direction in their gradients, which improves robustness.
problem Mathematical lack of clarity in the direction of preferential gradient alignment after adversarial training.
method Proposed a novel definition of preferential direction and evaluated it using a metric based on GANs.
result PGD-trained models have higher alignment with the proposed preferential direction than baseline models.
Paper tackles phase retrieval with robust gradient descent for noisy data.
problem Recover signals from magnitude measurements with noise and corruption.
method Robust gradient descent applied to Wirtinger Flow algorithm.
result Improves algorithm's robustness to heavy-tailed noise and adversarial corruption.
Gradient descent implicitly regularizes neural networks by penalizing large loss gradients.
problem How to optimize deep neural networks without explicit regularization.
method Backward error analysis to calculate implicit gradient regularization and demonstrate its effectiveness empirically.
result Implicit gradient regularization biases gradient descent toward flat minima, improving model robustness and test errors.
Accelerated optimization methods improve robustness and privacy in estimation.
problem Improving robustness and privacy in estimation methods.
method Accelerated gradient methods based on Frank-Wolfe and projected gradient descent, with tailored learning rates and Nesterov's momentum.
result Reduction in iteration complexity, leading to stronger statistical guarantees.
Robust multivariate linear regression methods for online and offline use.
problem Estimating parameters of multivariate Gaussian linear regression models robustly.
method Robust versions of least-square criterion with online and offline algorithms.
result Asymptotic normality of robust estimates proved under weak assumptions.
New Muon and Momo variants improve neural network optimization robustness.
problem Improving neural network optimization methods.
method Systematic exploration of non-Euclidean gradient descent variants.
result Momo variants of Muon are more robust to hyperparameter tuning.
New method speeds up PGD for CV robustness evaluation.
problem Computational inefficiency of PGD for robustness evaluation.
method Early termination of PGD based on cycle detection.
result Large speedup factors without sacrificing robustness.
Framework combines adversarial training and provable robustness for neural networks.
problem Training certifiably robust neural networks with provable robustness guarantees.
method Formulates joint optimization problem with adversarial and provable robustness objectives; develops gradient-descent technique.
result Consistently matches or outperforms prior approaches for provable l infinity robustness on MNIST and CIFAR-10.
This paper solves tensor robust principal component analysis via scaled gradient descent.
problem Extracting useful information from tensor data robust to corruptions and ill-conditioning.
method Directly recovers low-rank tensor factors via scaled gradient descent with adaptive thresholding.
result The proposed algorithm converges linearly to the true low-rank tensor at a constant rate independent of the condition number.
Modern neural networks are typically trained in an over-parameterized regime where the parameters of the model far exceed the size of the training data. Such neural networks in principle have the capacity to (over)fit any set of labels including pure noise. Despite this, somewhat paradoxically, neural network models tr…
We provide a new computationally-efficient class of estimators for risk minimization. We show that these estimators are robust for general statistical models: in the classical Huber epsilon-contamination model and in heavy-tailed settings. Our workhorse is a novel robust variant of gradient descent, and we provide cond…
Paper shows how to use geometric median for robust SGD in high dimensions.
problem Robustifying SGD for high-dimensional optimization problems with gross corruption.
method Applying geometric median to only chosen blocks of coordinates at a time.
result Retains optimal breakdown point of 0.5 for smooth non-convex problems.
A new method for robust training under label noise using weighted gradient descent.
problem Overfitting to noisy examples in machine learning.
method Exponentiated gradient reweighting for flexible handling of noisy data.
result Improved generalization in noisy classification and PCA problems.
Stochastic gradient descent outperforms traditional force-directed methods.
problem Improving graph layout quality and efficiency.
method Applying stochastic gradient descent for stress minimization.
result Stochastic gradient descent is simpler and more robust than traditional methods.
Tuning-free OR-PCA improves scalability for large datasets.
problem Dataset sensitivity of OR-PCA tuning parameters.
method Implicit regularization through modified gradient descents.
result Comparable or better performance on simulated and real-world datasets.
Distributionally robust optimization (DRO) problems are increasingly seen as a viable method to train machine learning models for improved model generalization. These min-max formulations, however, are more difficult to solve. We therefore provide a new stochastic gradient descent algorithm to efficiently solve this DR…
Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterizationcs.LG
Gradient descent recovers low-rank matrices from corrupted measurements with double over-parameterization.
problem Robust recovery of low-rank matrices from grossly corrupted measurements.
method Gradient descent with discrepant learning rates for double over-parameterized models.
result Gradient descent with discrepant learning rates provably recovers the underlying matrix without prior knowledge on rank or sparsity.
We propose Zeno++, a new robust asynchronous Stochastic Gradient Descent~(SGD) procedure which tolerates Byzantine failures of the workers. In contrast to previous work, Zeno++ removes some unrealistic restrictions on worker-server communications, allowing for fully asynchronous updates from anonymous workers, arbitrar…
Two novel algorithms improve distributed machine learning in the presence of Byzantine adversaries.
problem Improving distributed machine learning in the presence of Byzantine adversaries.
method Two novel stochastic gradient descent algorithms, ByGARS and ByGARS++, using reputation scores for gradient aggregation.
result Robust to any number of multiplicative noise Byzantine adversaries and converge for strongly convex loss functions.
A scalable method for heavy-tailed gradients using cheap stochastic sub-processes.
problem Scalability issues with robust gradient descent under heavy-tailed gradients.
method Cheap stochastic sub-processes for a single pass over partitioned data.
result Improved scalability and robustness to perturbations.
Improved SGD for robust linear and ReLU regression with adversarial corruptions.
problem Robust regression with adversarial corruptions in streaming data.
method Stochastic gradient descent (SGD-exp) with exponentially decaying step size.
result Nearly linear convergence to true parameter with up to 50% Massart corruption rate.
Study robustness of early-stopping GD for linear regression attacks.
problem Robustness of gradient-descent methods to adversarial attacks.
method Early-stopping strategies, gradient-descent, Mahalanobis attacks, feature-dependent learning rates, data transformations.
result Early-stopped GD is optimally robust to Euclidean-norm attacks but sub-optimal for Mahalanobis attacks.
New method improves MMD estimation without convexity assumptions.
problem Lack of theoretical guarantees for MMD estimation algorithms.
method Preconditioned gradient descent (PGD) scheme for MMD optimization.
result PGD scheme converges globally under specific conditions.
OGD proves robustness to Catastrophic Forgetting in Continual Learning.
problem Catastrophic Forgetting in Continual Learning with deep neural networks.
method Theoretical framework based on Neural Tangent Kernel for OGD.
result First generalization bound for SGD and OGD in Continual Learning.
RESIST improves decentralized learning resilience against MITM attacks.
problem Decentralized learning's vulnerability to MITM attacks.
method Multistep consensus gradient descent framework with robust statistics-based screening methods.
result Achieves algorithmic and statistical convergence for various ERM problems.
Improved robust regression algorithms with faster runtime and better estimation rates.
problem Statistical regression problems under strong contamination model.
method Nearly-linear time algorithms using robust gradient descent and Sever framework.
result Improved estimation rates and runtime compared to state-of-the-art.
In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations. In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal s…
Stochastic gradient descent's long-term fluctuations are described by a diffusion limit.
problem Long-term behavior of stochastic gradient descent in non-smooth settings.
method Functional central limit theorem applied to rescaled trajectory of SGD.
result Characterization of long-term fluctuations around the minimizer.
Paper explores using bootstrap methods to improve SGD's stability and robustness.
problem Improving the stability and robustness of SGD.
method Investigates empirical bootstrap approaches for SGD from algorithmic stability and statistical robustness perspectives.
result Demonstrates construction of purely distribution-free confidence intervals using bootstrap SGD.
New Stein operator improves robustness in model inference.
problem Improving robustness in inference for unnormalized models.
method Density-power weighted Stein operator (-Stein operator).
result Robust methods for goodness-of-fit testing and posterior approximation.
Study reveals biases in gradient descent for GLNs, improving neural network performance.
problem Understanding and improving the inductive biases of deep neural networks.
method Derive infinite-time training limit of gated linear networks and generalize to other networks.
result Theoretical framework captures key inductive biases of ReLU networks.
In this paper, we propose an implicit gradient descent algorithm for the classic -means problem. The implicit gradient step or backward Euler is solved via stochastic fixed-point iteration, in which we randomly sample a mini-batch gradient in every iteration. It is the average of the fixed-point trajectory that is c…
BadGD identifies gradient descent vulnerabilities through strategic backdoor attacks.
problem Gradient descent vulnerabilities through malicious data manipulation.
method Introduces Max RiskWarp, Max GradWarp, and Max GradDistWarp triggers to exploit gradient descent.
result Demonstrates how malicious triggers can significantly alter loss landscapes and gradient calculations.
New Transformers maintain Lipschitz continuity for robustness.
problem Ensuring robustness in Transformers for safety-sensitive applications.
method Introducing gradient-descent-type in-context Transformers with explicit Euler steps of negative gradient flows.
result Universal approximation theorem for Lipschitz continuous Transformers.
We propose a novel robust aggregation rule for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilience…
PEGR improves deep learning models' robustness against noisy data.
problem Learning signals from noisy data in deep learning models.
method Per-example gradient regularization (PEGR) to suppress noise.
result PEGR enhances test error and robustness against noise perturbations.
New algorithms robustly estimate mean with near-optimal error rates.
problem Outlier robust mean estimation in high-dimensional data.
method Stability condition and iterative filtering algorithms.
result Optimal error rates with subgaussian rates for robust mean estimation.
We consider the minimization of a function defined on a Riemannian manifold accessible only through unbiased estimates of its gradients. We develop a geometric framework to transform a sequence of slowly converging iterates generated from stochastic gradient descent (SGD) on to an averaged i…
Private minimum Hellinger distance estimators maintain robustness and efficiency while ensuring privacy.
problem Ensuring privacy in robust statistical estimation.
method Derive private minimum Hellinger distance estimators satisfying Hellinger differential privacy.
result Private minimum Hellinger distance estimators retain robustness and efficiency under privacy constraints.
Paper studies IPG method's robustness in noisy distributed linear regression.
problem Distributed linear regression with noise.
method Iteratively Pre-conditioned Gradient-descent (IPG) method.
result IPG method's robustness against noise compared favorably to state-of-the-art algorithms.