U-Clip clips gradients to reduce bias without discarding them.
problem Reduces bias in stochastic gradient descent.
method Maintains a buffer of clipped gradients to add to next iteration.
result U-Clip updates are unbiased on average.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Gradient clipping”
Gradient clipping helps private SGD converge despite potential bias.
problem Gradient clipping in private SGD can bias convergence.
method Theoretical analysis and empirical evaluation of gradient clipping effects.
result Gradient clipping can prevent convergence to stationary points and introduces bias.
A new method improves training speed and robustness for deep learning models.
problem Training deep learning models with heavy-tailed gradient noise.
method Per-sample gradient clipping combined with SGD.
result PS-Clip-SGD achieves optimal convergence rates and high-probability guarantees.
Mitigating Disparate Impact of Differentially Private Learning through Bounded Adaptive Clippingcs.LG
New method improves fairness in DP learning by preventing excessive gradient suppression.
problem Disparate impact on model predictions for minority groups in DP learning.
method Bounded adaptive clipping to prevent excessive gradient suppression.
result Improves worst-class accuracy by over 10 percentage points compared to existing methods.
AutoClip automatically adjusts gradient clipping for better audio separation.
problem Improving generalization in audio source separation networks.
method Adaptive gradient clipping based on historical gradient norms.
result Improves generalization performance in audio source separation networks.
This paper improves convergence guarantees for gradient clipping in deep learning.
problem Improving convergence guarantees for gradient clipping in deep learning models.
method Analyzes and provides precise convergence guarantees for arbitrary clipping thresholds.
result Shows tight convergence guarantees for clipped stochastic gradient descent.
Clip21 improves convergence of gradient-clipped methods in DP settings.
problem Gradient clipping introduces bias in distributed training, causing convergence issues.
method Clip21 designs an error feedback mechanism to mitigate bias in gradient-clipped methods.
result Clip21 converges at the same rate as distributed gradient descent, improving from previous $\mathcal{O}\left(\frac{1}{\sqrt{K}}
ight)$ to $\mathcal{O}\left(\frac{1}{K}
ight)$.
Paper explores whether gradient normalization can replace clipping for SGD in heavy-tailed noise.
problem Ensuring convergence of SGD in heavy-tailed noise.
method Revisits gradient clipping and normalization, proving their sufficiency and effectiveness.
result Gradient normalization alone is sufficient for nonconvex SGD convergence under smoothness assumptions.
Study on gradient clipping in SGD for high-dimensional problems.
problem Understanding and optimizing gradient clipping in high-dimensional machine learning models.
method Theoretical analysis of streaming SGD with gradient clipping in a least squares problem, focusing on large intrinsic dimensionality.
result Developed a deterministic equation to describe the loss evolution in clipped SGD, showing benefits under certain conditions.
Many continuous control tasks have bounded action spaces. When policy gradient methods are applied to such tasks, out-of-bound actions need to be clipped before execution, while policies are usually optimized as if the actions are not clipped. We propose a policy gradient estimator that exploits the knowledge of action…
This paper improves analysis of clipping algorithms for non-convex optimization.
problem Improving convergence analysis of clipping algorithms for non-convex optimization.
method Develops a general framework to study clipping algorithms, including momentum methods, and provides convergence analysis in deterministic and stochastic settings.
result Demonstrates that clipping methods can maintain efficiency even in highly non-smooth regions.
Improved differentially private deep learning with group-wise clipping techniques.
problem Efficiency and privacy trade-offs in deep learning models.
method Group-wise clipping techniques (per-layer and per-device) to reduce compute time and memory overhead.
result Private learning with group-wise clipping achieves similar or better performance than non-private learning with less wall time.
This paper analyzes convergence of DP-SGD with adaptive quantile clipping.
problem Empirical success of adaptive clipping methods lacks theoretical understanding.
method Comprehensive convergence analysis of SGD with quantile clipping (QC-SGD).
result Establishes theoretical guarantees for DP-QC-SGD, revealing relationships between quantile selection, step size, and convergence.
New optimization method combines gradient clipping and non-Euclidean smoothness.
problem Improving optimization in non-Euclidean spaces for machine learning.
method Hybrid of steepest descent and conditional gradient, incorporating weight decay.
result Achieves optimal convergence rate and demonstrates effectiveness in deep learning.
New method improves convergence of SGD for heavy-tailed noise.
problem Heavy-tailed noise in machine learning applications.
method Normalized SGD (NSGD) to handle heavy-tailed noise.
result Improved sample complexity and high-probability convergence.
New algorithm robustly optimizes data streams with heavy-tailed or infinite variance samples.
problem Optimizing data streams with heavy-tailed or infinite variance samples.
method Gradient quantile clipping for SGD, leveraging Markov chain connections.
result Algorithm converges to a concentrated distribution with high probability bounds.
New analysis shows how to balance privacy and accuracy in deep learning.
problem Balancing privacy and accuracy in deep learning models.
method Continuous time analysis through neural tangent kernel (NTK) for arbitrary architectures.
result Large clipping norm improves calibration without sacrificing accuracy.
PPO's gradients are heavy-tailed, affecting learning; a robust estimator improves performance.
problem Heavy-tailedness of PPO gradients causing learning issues.
method Characterized heavy-tailed gradients, identified likelihood ratios and advantages as sources, proposed GMOM as a robust estimator.
result GMOM improves PPO performance without clipping tricks.
Improved optimization technique reduces training complexity for non-convex problems.
problem Training non-convex optimization problems with exploding gradients.
method Employed variance reduction technique (SPIDER) with carefully designed learning rate.
result Improved stochastic gradient complexity to for -stationary solutions.
A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscs.LG
A new algorithm reduces communication in deep learning training.
problem Slow convergence in deep neural networks due to exploding gradients.
method Communication-efficient distributed gradient clipping algorithm.
result Algorithm achieves linear speedup and reduced communication rounds.
Improved Clipped-SGD achieves near-optimal heavy-tailed statistical estimation in streaming settings.
problem High-dimensional heavy-tailed statistical estimation in streaming with memory constraints.
method Stochastic convex optimization with Clipped-SGD, proving near-optimal sub-Gaussian statistical rates.
result Clipped-SGD achieves an error of with probability .
Removing Disparate Impact of Differentially Private Stochastic Gradient Descent on Model Accuracycs.LG
New method reduces privacy impact on model accuracy for underrepresented groups.
problem Privacy mechanisms disproportionately affect underrepresented groups in machine learning models.
method Proposes DPSGD-F, a modified DPSGD that adjusts group contributions based on clipping bias.
result DPSGD-F removes disparate impact of differential privacy on model accuracy for protected groups.
Improved analysis for clipped gradient methods in nonsmooth convex optimization under heavy-tailed noise.
problem Optimization under heavy-tailed noise in nonsmooth convex problems.
method Refined analysis of Clipped Stochastic Gradient Descent (Clipped SGD) with new rates and improved utilization of Freedman's inequality.
result New rates and for nonsmooth convex and strongly convex problems, respectively.
AdaDPIGU improves privacy in deep learning by adaptively clipping and pruning gradients.
problem Privacy in deep learning models, especially in high-dimensional settings.
method Importance-based gradient updates, adaptive clipping, differentially private SGD.
result AdaDPIGU achieves high accuracy while maintaining privacy, outperforming non-private models.
Privacy-preserving SGD with heavy-tailed noise achieves differential privacy guarantees.
problem Privacy preservation in noisy SGD with heavy-tailed noise.
method Differential privacy guarantees for SGD with heavy-tailed noise.
result SGD with heavy-tailed perturbations achieves -DP.
SAD-DPSGD improves model performance on imbalanced medical datasets like HAM10000.
problem Data leakage and imbalanced distribution in medical image classification datasets.
method SAD-DPSGD uses a linear decaying mechanism for noise and clipping thresholds to enhance performance.
result SAD-DPSGD outperforms Auto-DPSGD on HAM10000, improving accuracy by 2.15%.
New methods speed up training of differentially private deep learning models.
problem Training differentially private deep learning models is slower than non-private models.
method Derive and implement new per-example gradient clipping methods compatible with auto-differentiation.
result Significant training speed-ups (54x - 94x) for various models and architectures.
This paper analyzes bias-variance trade-off for clipped SFOMs, improving complexity guarantees for heavy-tailed noise.
problem Improving complexity guarantees for stochastic optimization methods with heavy-tailed noise.
method Novel analysis of bias-variance trade-off in gradient clipping for clipped SFOMs.
result Improved complexity guarantees for clipped SFOMs across various tail indices, including infinite mean noise.
Dp-CLIP preserves privacy in multimodal AI training.
problem Privacy concerns in multimodal AI, especially in vision-language tasks.
method Differentially private adaptation of CLIP model.
result Dp-CLIP retains accuracy while ensuring privacy.
Privacy preserving machine learning algorithms are crucial for learning models over user data to protect sensitive information. Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed. At each step, these algorithms modify the gradie…
New algorithm maintains privacy while improving model performance in selective release.
problem Privacy degradation and slow convergence in DPSGD.
method Differentially Private Selective Release based on Clipped Gradients (DPSR-CG).
result Maintains strict privacy guarantees while achieving exceptional model performance.
Paper proposes DP-SGD and DP-NSGD for differentially private non-convex optimization.
problem Mitigating privacy risks in large model learning.
method Clip or normalize per-sample gradients and add noise for differential privacy.
result Achieved convergence rate of gradient norm for non-convex optimization.
Recently, the study of heavy-tailed noises in first-order nonconvex stochastic optimization has gotten a lot of attention since it was recognized as a more realistic condition as suggested by many empirical observations. Specifically, the stochastic noise (the difference between the stochastic and true gradient) is con…
New theory explains why normalization is preferred in SGD under heavy-tailed noise.
problem Understanding why normalization is preferred in stochastic gradient descent (SGD) under heavy-tailed noise.
method Developed a worst-case complexity theory for stochastically preconditioned SGD and its variants.
result Normalization guarantees convergence at optimal rates, while clipping may fail in the worst case.
We establish the first benchmark for federated learning with differential privacy in ASR, achieving competitive performance.
problem Challenges in training large transformer models for ASR in federated learning with differential privacy.
method Per-layer clipping and layer-wise gradient normalization to mitigate gradient heterogeneity.
result FL with DP is viable in ASR with strong privacy guarantees, achieving competitive performance.
Paper proves convergence for private FL on non-Lipschitz convex objectives using normalization instead of clipping.
problem Lack of convergence results for differentially private federated learning with non-Lipschitz objectives.
method Developed a convergence result for private FL on smooth convex objectives without assuming Lipschitzness, using normalization instead of clipping.
result Normalization-based private FL algorithm converges better than clipping-based counterpart on smooth convex functions.
Theory for RLHF generalization under reward shift and clipped KL.
problem Theoretical understanding of RLHF generalization, especially with reward shift and clipped KL.
method Developed generalization theory for RLHF, accounting for reward shift and clipped KL.
result Presented generalization bounds for RLHF, suggesting generalization error from sampling, reward shift, and KL clipping.
New method for differentially private optimization with general Lipschitz conditions.
problem Differentially private optimization under general Lipschitz conditions.
method Generalized Lipschitz condition for per-sample gradients, tuning clip norm based on minimum per-sample Lipschitz constant.
result Efficacy of the recommended clip norm tuning method verified on 8 datasets.
Generalizes smoothness conditions for optimization methods.
problem Optimization under non-uniform smoothness conditions.
method Develops a new analysis technique for bounding gradients.
result Obtains convergence rates for gradient descent and Nesterov's method.
Proposes an alternative method to train RBMs with binary synapses using Bayesian learning rule.
problem Training RBMs with binary synapses is challenging due to discrete nature of synapses.
method Proposes an alternative optimization method using the Bayesian learning rule, updating natural parameters instead of expectation parameters.
result No additional clipping is needed as natural parameters take values in the entire real domain.
New method makes reinforcement learning robust to heavy-tailed rewards.
problem Heavy-tailed rewards cause statistical outliers in reinforcement learning.
method Dynamic gradient clipping in TD learning and NAC.
result Provably robust TD and NAC achieve optimal sample complexities.
Improved linear regression with privacy and robustness guarantees.
problem Private and robust linear regression with adversarial corruption.
method Differentially private stochastic gradient descent with full-batch gradient descent and adaptive clipping.
result Near optimal sample complexity for both private and robust linear regression.
Parametric t-SNE improves generalization for streaming data.
problem Training neural networks for t-SNE objective function fails due to gradient exploding.
method Applied gradient clipping to solve gradient exploding problem.
result Parametric t-SNE achieves quality compatible with non-parametric t-SNE while generalizing to new data.
Proposes a new policy gradient algorithm to improve reinforcement learning efficiency and stability.
problem Inefficiency and instability of DDPG in practical applications, and difficulty in controlling Q estimation bias and variance.
method Introduces a Regularly Updated Deterministic (RUD) policy gradient algorithm.
result The RUD algorithm makes better use of new data and has lower Q value variance, leading to improved performance.
A new gradient estimator reduces variance near boundaries for binary latent variables.
problem Explosive gradient variance near boundaries in binary latent variable models.
method Introduces a new gradient estimator (bitflip-1) and an aggregated estimator (UGC) that uses either bitflip-1 or DisARM for each coordinate.
result UGC has uniformly lower variance than DisARM and achieves optimal optimization objectives.
Private adaptive methods improve on traditional SGD for convex optimization.
problem Differential privacy constraints in gradient optimization.
method Differentially private variants of SGD and AdaGrad with adaptive stepsizes and non-isotropic clipping.
result Private AdaGrad outperforms private SGD in high-dimensional problems.
Optimal DP-GD achieves high-dimensional linear regression risk.
problem Differentially private linear regression in high-dimensional settings.
method One-pass DP gradient descent (DP-GD) with gradient clipping and decaying learning rate.
result Achieves optimal non-asymptotic risk of for well-conditioned data.
RoNGBa: A Robustly Optimized Natural Gradient Boosting Training Approach with Leaf Number Clippingcs.LG
Natural gradient has been recently introduced to the field of boosting to enable the generic probabilistic predication capability. Natural gradient boosting shows promising performance improvements on small datasets due to better training dynamics, but it suffers from slow training speed overhead especially for large d…