Optimization with noisy gradients has become ubiquitous in statistics and machine learning. Reparameterization gradients, or gradient estimates computed via the "reparameterization trick," represent a class of noisy gradients often used in Monte Carlo variational inference (MCVI). However, when these gradient estimator…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “noisy gradients”
AGNES accelerates gradient descent with noisy gradients.
problem Minimizing smooth convex and strongly convex functions with noisy gradients.
method Generalization of Nesterov's accelerated gradient descent algorithm for noisy conditions.
result AGNES achieves acceleration for noisy gradients with a constant of proportionality up to 1.
Variational Bayesian neural networks combine the flexibility of deep learning with Bayesian uncertainty estimation. However, inference procedures for flexible variational posteriors are computationally expensive. A recently proposed method, noisy natural gradient, is a surprisingly simple method to fit expressive poste…
Study on privacy leakage in noisy gradient descent algorithms.
problem Information leakage of iterative randomized learning algorithms about training data.
method Analyzes the dynamics of Rényi differential privacy loss in noisy gradient descent algorithms.
result Privacy loss converges exponentially fast for smooth and strongly convex loss functions.
We consider the behavior of gradient flow and of discrete and noisy gradient descent. It is commonly noted that the addition of noise to the process of discrete gradient descent can affect the trajectory of gradient descent. In previous work, we observed such effects. There, we considered the case where the minima had …
Deep learning with noisy gradient descent outperforms linear estimators in high dimensions.
problem Theoretical explanation of deep learning's superiority over linear methods.
method Theoretical analysis of excess risk of a deep learning estimator trained by noisy gradient descent.
result Deep learning achieves a faster learning rate than linear estimators, especially in high dimensions.
Preconditioned non-convex gradient descent improves noisy matrix estimation.
problem Estimating low-rank matrices from noisy measurements.
method Preconditioned non-convex gradient descent for noisy measurements.
result Preconditioned method converges to minimax optimal estimate at a linear rate.
A new method for robust training under label noise using weighted gradient descent.
problem Overfitting to noisy examples in machine learning.
method Exponentiated gradient reweighting for flexible handling of noisy data.
result Improved generalization in noisy classification and PCA problems.
PEGR improves deep learning models' robustness against noisy data.
problem Learning signals from noisy data in deep learning models.
method Per-example gradient regularization (PEGR) to suppress noise.
result PEGR enhances test error and robustness against noise perturbations.
Neural networks can interpolate noisy data and still generalize well.
problem Generalization of neural networks trained on noisy data.
method Two-layer neural networks trained to interpolation by gradient descent on corrupted labels.
result Neural networks can achieve zero training error and optimal test error.
FGD reduces noisy gradient variance in SGD for neural networks.
problem Noisy and unreliable gradient estimation in SGD for deep learning.
method Solves an adaptive filtering problem to consistently estimate the local gradient.
result Significantly reduces gradient variance and accelerates convergence.
New algorithms minimize noisy, irregular functions without gradients.
problem Minimizing noisy, irregular, and algebraically intractable functions.
method Generalized gradient descent recursion with smooth approximations.
result Convergence results under weak assumptions on function regularity.
Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channelsstat.ML
The paper analyzes generalization of noisy iterative algorithms using communication theory.
problem Generalization of models trained by noisy iterative algorithms under different distributions.
method Connecting noisy iterative algorithms to additive noise channels in communication theory.
result Distribution-dependent generalization bounds for noisy iterative algorithms.
Proposes coreset method for robust training of neural networks with noisy labels.
problem Overfitting of neural networks trained with noisy labels.
method Selects weighted subsets (coresets) of clean data points to approximate low-rank Jacobian matrix.
result Gradient descent applied to coreset subsets does not overfit noisy labels.
Unified framework for analyzing batch updating methods with noisy gradients.
problem Analyzing convergence of batch updating methods with noisy gradients and approximations.
method Unified framework using convergence of stochastic processes.
result Establishes a general theorem for most known convergence results.
Resetting from checkpoints improves DNN training with noisy labels.
problem Latent gradient bias induced by noisy labels causes overfitting.
method Stochastic resetting applied to SGD to mitigate latent gradient bias.
result Resetting significantly improves DNN generalization performance.
Saliency Map, the gradient of the score function with respect to the input, is the most basic technique for interpreting deep neural network decisions. However, saliency maps are often visually noisy. Although several hypotheses were proposed to account for this phenomenon, there are few works that provide rigorous ana…
Paper tackles noisy labels in deep learning networks.
problem Learning with noisy labels in deep neural networks.
method Sparse regularization strategy to approximate one-hot constraint.
result Improves performance of commonly-used loss functions in noisy labels and class imbalance.
Paper improves generalization bounds for noisy stochastic algorithms.
problem Improving generalization bounds for noisy stochastic algorithms.
method Introduces Exponential Family Langevin Dynamics (EFLD) and establishes data-dependent expected stability based generalization bounds.
result Sharp generalization bounds with O(1/n) sample dependence and gradient discrepancy.
Variational Bayesian neural nets combine the flexibility of deep learning with Bayesian uncertainty estimation. Unfortunately, there is a tradeoff between cheap but simple variational families (e.g.~fully factorized) or expensive and complicated inference procedures. We show that natural gradient ascent with adaptive w…
Framework improves gradient estimation for faster training convergence.
problem Efficiently estimating noisy gradients in stochastic optimization.
method Dynamic adaptive importance sampling combining multiple distributions.
result Adaptively weighted multiple importance sampling yields superior gradient estimates.
Paper analyzes gradient descent with noisy data copies for linear regression, showing regularization and acceleration effects.
problem Improving generalization in machine learning through data augmentation with noise.
method Gradient descent with on-line noisy copies for linear regression analysis.
result Training with on-line noisy copies is equivalent to ridge regularization with a specific regularization parameter.
Gradient filters track moving parameters under noisy data and misspecification.
problem Tracking multidimensional time-varying parameters under noisy observations and model misspecification.
method Gradient-based filters update parameters using the gradient of a postulated objective function, evaluated at either the predicted or updated parameters.
result Novel sufficient conditions for exponential stability of the filtered parameter path, and finite-sample and asymptotic mean squared error bounds.
Despite the success of deep neural networks (DNNs) in image classification tasks, the human-level performance relies on massive training data with high-quality manual annotations, which are expensive and time-consuming to collect. There exist many inexpensive data sources on the web, but they tend to contain inaccurate…
Noise-corrected Langevin algorithm improves sampling from noisy data.
problem Sampling from noisy data with biased score function.
method Noise-corrected Langevin algorithm using noisy score function.
result Bias due to noisy data is removed, improving sampling accuracy.
APGD algorithm efficiently recovers over-parameterized matrices from noisy measurements.
problem Matrix sensing problem with over-parameterization and noisy measurements.
method Alternating preconditioned gradient descent (APGD) algorithm incorporating preconditioning terms.
result APGD converges to a near-optimal error at a linear rate.
The study examines how many samples are needed to minimize a noisy convex function with inaccurate gradient estimates.
problem Determining the number of samples needed to minimize a noisy convex function with inaccurate gradient estimates.
method Using Stochastic Convex Optimization as a case study, the study analyzes the relationship between the number of samples and the accuracy of gradient estimates.
result The study provides partial answers to the question, showing that a general analyst requires samples and that under certain assumptions, samples are necessary for gradient descent to interact with the oracle.
Given data with noisy labels, over-parameterized deep networks can gradually memorize the data, and fit everything in the end. Although equipped with corrections for noisy labels, many learning methods in this area still suffer overfitting due to undesired memorization. In this paper, to relieve this issue, we propose …
Algorithms for bandit convex optimization and online learning often rely on constructing noisy gradient estimates, which are then used in appropriately adjusted first-order algorithms, replacing actual gradients. Depending on the properties of the function to be optimized and the nature of ``noise'' in the bandit feedb…
In this note, we observe the behavior of gradient flow and discrete and noisy gradient descent in some simple settings. It is commonly noted that addition of noise to gradient descent can affect the trajectory of gradient descent. Here, we run some computer experiments for gradient descent on some simple functions, and…
Mixing Times and Privacy Analysis for the Projected Langevin Algorithm under a Modulus of Continuitystat.ML
New bounds for mixing time and privacy in projected Langevin algorithm and noisy SGD.
problem Analyzing mixing times and privacy in projected Langevin algorithm and noisy SGD.
method New bounds derived using PABI framework and optimization problems.
result New bounds for mixing time and privacy in projected Langevin algorithm and noisy SGD, showing dependency on gradient regularity.
Improved private AdaGrad achieves faster convergence rates for convex functions.
problem Private empirical risk minimization with differential privacy.
method Noisy AdaGrad with knowledge of gradient subspace geometry.
result Faster convergence rates for convex functions, bypassing traditional bounds.
This work focuses on dynamic regret of online convex optimization that compares the performance of online learning to a clairvoyant who knows the sequence of loss functions in advance and hence selects the minimizer of the loss function at each step. By assuming that the clairvoyant moves slowly (i.e., the minimizers c…
FSGLD improves federated data sampling by correcting noisy gradients.
problem Noisy gradients and delayed communication in federated data.
method Conductive gradients to correct noisy gradients in distributed SGLD.
result FSGLD converges to true posterior even with delayed communication.
MSLG generates soft labels to improve DNN performance on noisy datasets.
problem Significant performance degradation of DNNs due to noisy labels.
method Meta-learning techniques to estimate optimal label distribution and iteratively update soft labels.
result MSLG outperforms state-of-the-art methods by a large margin on various datasets.
Study recovers spike order in noisy tensor estimation without SNR assumptions.
problem Estimating multiple signal vectors from noisy tensor observations.
method Gradient flow optimization of a nonconvex function.
result Determines sample complexity for efficient permutation recovery.
Improved reSGLD accelerates convergence in non-convex learning problems.
problem Inefficient swaps due to noisy energy estimators in reSGLD.
method Variance reduction for noisy energy estimators, theoretical analysis, and numerical experiments.
result Exponential acceleration in convergence for non-convex learning problems.
Neural networks learn clean data patterns first, then noisy data, leading to improved performance initially but deteriorating later.
problem Improvement in prediction error on clean data during early training of neural networks with noisy labels.
method Theoretical analysis and experiments to explore the dynamics of gradient descent and the impact of clean and noisy data.
result Neural networks prioritize learning clean data patterns first, leading to improved performance initially but deteriorating later due to diminishing gradient dominance of clean samples over noisy ones.
Analyzes DNNs trained with noisy gradients, finding FWCs negligible for large n.
problem Analyzing DNNs trained with noisy gradients.
method Introduced analytical framework to analyze non-Gaussian stochastic process.
result FWCs negligible for large n, improving CNN performance.
The gradient noise of SGD is considered to play a central role in the observed strong generalization abilities of deep learning. While past studies confirm that the magnitude and the covariance structure of gradient noise are critical for regularization, it remains unclear whether or not the class of noise distribution…
No accelerated gradient method for hyperbolic convex functions.
problem Existence of accelerated gradient methods for geodesically convex functions on hyperbolic spaces.
method Analysis of volume growth in negatively curved spaces.
result No-go theorem for accelerated gradient methods on hyperbolic plane.
The paper analyzes the efficiency of gradient estimation methods in noisy function evaluations.
problem Estimating gradients of smooth functions using noisy function evaluations.
method Information-theoretic lower bounds and finite difference method analysis.
result The finite difference method is not minimax optimal, suggesting room for improvement in gradient estimation.
Improved GSPGS estimators reduce bias in noisy function measurements.
problem Reduced bias in noisy function measurements.
method Generalized Simultaneous Perturbation-based Gradient Search (GSPGS) with various estimators.
result Estimators requiring more function measurements have lower bias.
New bounds found for optimizing non-convex functions with noisy data.
problem Limits of first-order stochastic optimization in non-convex settings.
method Divergence decomposition to construct challenging subclasses.
result Sharp lower bounds on noisy gradient queries for various non-convex classes.
Label noise may affect the generalization of classifiers, and the effective learning of main patterns from samples with noisy labels is an important challenge. Recent studies have shown that deep neural networks tend to prioritize the learning of simple patterns over the memorization of noise patterns. This suggests a …
It has been observed in a variety of contexts that gradient descent methods have great success in solving low-rank matrix factorization problems, despite the relevant problem formulation being non-convex. We tackle a particular instance of this scenario, where we seek the -dimensional subspace spanned by a streaming…
New method improves convergence and reduces variance in noisy optimization problems.
problem Computing exact minimizers with noisy gradient information.
method Stochastic mirror descent with interacting particles.
result Interaction helps improve convergence and reduce variance.
Paper introduces a neural network training algorithm for noisy data that achieves optimal parameters and replicates real-world behaviors.
problem Theoretical gap between universal approximation theorems and practical machine learning with noisy data.
method Randomized training algorithm for neural networks trained on noisy data samples.
result Trained neural networks achieve optimal parameters and exhibit real-world behaviors like sub-linear complexity and interpolation.