AdaCliP reduces noise in private SGD training.
problem Privacy preserving machine learning over user data.
method Adaptive clipping of gradients to reduce noise in private SGD.
result AdaCliP adds less noise and improves model accuracy.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,878 papers · 148 categories
Trend · papers per month
48 results for “coordinate-wise clipping”
AdaDPIGU improves privacy in deep learning by adaptively clipping and pruning gradients.
problem Privacy in deep learning models, especially in high-dimensional settings.
method Importance-based gradient updates, adaptive clipping, differentially private SGD.
result AdaDPIGU achieves high accuracy while maintaining privacy, outperforming non-private models.
New DP-CD method outperforms DP-SGD in solving composite DP-ERM problems.
problem Privacy-preserving machine learning with differential privacy.
method Differentially Private proximal Coordinate Descent (DP-CD) for composite Empirical Risk Minimization (ERM).
result DP-CD outperforms DP-SGD due to larger step sizes and better gradient exploitation.
We develop and analyze efficient "coordinate-wise" methods for finding the leading eigenvector, where each step involves only a vector-vector product. We establish global convergence with overall runtime guarantees that are at least as good as Lanczos's method and dominate it for slowly decaying spectrum. Our methods a…
Proposes a new Armijo's condition for coordinate-wise functions.
problem Finding optimal step sizes in coordinate-wise optimization.
method Introduces a new Armijo's condition for functions defined on product spaces.
result Shows the advantage of the new condition over the standard Armijo's condition.
The paper introduces a new method for feature selection without explicit sparsification.
problem Feature selection with implicit sparsity-inducing mechanisms.
method Optimization over a family of kernels with gradient descent.
result The method achieves exact sparsity without explicit penalization techniques.
U-Clip clips gradients to reduce bias without discarding them.
problem Reduces bias in stochastic gradient descent.
method Maintains a buffer of clipped gradients to add to next iteration.
result U-Clip updates are unbiased on average.
Improved differentially private deep learning with group-wise clipping techniques.
problem Efficiency and privacy trade-offs in deep learning models.
method Group-wise clipping techniques (per-layer and per-device) to reduce compute time and memory overhead.
result Private learning with group-wise clipping achieves similar or better performance than non-private learning with less wall time.
Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network par…
Study on gradient clipping in SGD for high-dimensional problems.
problem Understanding and optimizing gradient clipping in high-dimensional machine learning models.
method Theoretical analysis of streaming SGD with gradient clipping in a least squares problem, focusing on large intrinsic dimensionality.
result Developed a deterministic equation to describe the loss evolution in clipped SGD, showing benefits under certain conditions.
In this paper, we consider the problem of recovering a sparse signal based on penalized least squares formulations. We develop a novel algorithm of primal-dual active set type for a class of nonconvex sparsity-promoting penalties, including , bridge, smoothly clipped absolute deviation, capped and mini…
Mitigating Disparate Impact of Differentially Private Learning through Bounded Adaptive Clippingcs.LG
New method improves fairness in DP learning by preventing excessive gradient suppression.
problem Disparate impact on model predictions for minority groups in DP learning.
method Bounded adaptive clipping to prevent excessive gradient suppression.
result Improves worst-class accuracy by over 10 percentage points compared to existing methods.
This paper analyzes convergence of DP-SGD with adaptive quantile clipping.
problem Empirical success of adaptive clipping methods lacks theoretical understanding.
method Comprehensive convergence analysis of SGD with quantile clipping (QC-SGD).
result Establishes theoretical guarantees for DP-QC-SGD, revealing relationships between quantile selection, step size, and convergence.
A new method improves training speed and robustness for deep learning models.
problem Training deep learning models with heavy-tailed gradient noise.
method Per-sample gradient clipping combined with SGD.
result PS-Clip-SGD achieves optimal convergence rates and high-probability guarantees.
CLIP models robustness to spurious features is re-evaluated using a new dataset.
problem Existing robustness tests of CLIP models may not fully reflect their performance on spurious features.
method Crafted a new dataset (CounterAnimal) to reveal CLIP models' reliance on realistic spurious features.
result CLIP models are robust to spurious features learned from their training data, not ImageNet.
CLIP learns joint image-text representations for zero-shot learning.
problem Understanding and improving zero-shot transfer performance in CLIP.
method Formal study of transferrable representation learning and analysis of zero-shot transfer performance.
result Proposes a new CLIP-type approach that outperforms existing methods.
Paper introduces Latent-CLIP for efficient text-image comparison in latent space.
problem Efficiently compare text and images in latent space without costly decoding.
method Trains CLIP model in latent space, uses Latent-CLIP rewards for noise optimization, and guides generation away from harmful content.
result Latent-CLIP matches CLIP performance on text-image classification and harmful content detection.
Proposes a new Armijo's condition for general functions and provides an algorithm for its application.
problem Finding suitable step sizes for general functions in optimization.
method Introduces a coordinate-wise Armijo's condition and provides an algorithm to find suitable step sizes.
result Proves convergent results for various functions using the proposed algorithm.
Gradient clipping helps private SGD converge despite potential bias.
problem Gradient clipping in private SGD can bias convergence.
method Theoretical analysis and empirical evaluation of gradient clipping effects.
result Gradient clipping can prevent convergence to stationary points and introduces bias.
This paper improves analysis of clipping algorithms for non-convex optimization.
problem Improving convergence analysis of clipping algorithms for non-convex optimization.
method Develops a general framework to study clipping algorithms, including momentum methods, and provides convergence analysis in deterministic and stochastic settings.
result Demonstrates that clipping methods can maintain efficiency even in highly non-smooth regions.
Enhances CLIP's similarity computation using PMI's linear structure.
problem CLIP's similarity computation misses the optimal linear structure of PMI.
method KME-CLIP, utilizing inner product in a reproducing kernel Hilbert space.
result KME-CLIP approximates PMI with arbitrary accuracy and outperforms CLIP.
AutoClip automatically adjusts gradient clipping for better audio separation.
problem Improving generalization in audio source separation networks.
method Adaptive gradient clipping based on historical gradient norms.
result Improves generalization performance in audio source separation networks.
Dp-CLIP preserves privacy in multimodal AI training.
problem Privacy concerns in multimodal AI, especially in vision-language tasks.
method Differentially private adaptation of CLIP model.
result Dp-CLIP retains accuracy while ensuring privacy.
This paper tackles minimizing clipped convex functions with heuristics and mixed-integer convex programming.
problem Minimizing a sum of clipped convex functions.
method Heuristics and mixed-integer convex programming.
result Heuristics can find good solutions, and the perspective transformation yields tractable lower bounds.
This paper improves convergence guarantees for gradient clipping in deep learning.
problem Improving convergence guarantees for gradient clipping in deep learning models.
method Analyzes and provides precise convergence guarantees for arbitrary clipping thresholds.
result Shows tight convergence guarantees for clipped stochastic gradient descent.
Unified CLIP space manipulations improve GAN adaptation with a single target image.
problem Overfitting or underfitting in fine-tuning a pre-trained generator with a single target image.
method Two-step training strategy: latent optimization in CLIP space followed by generator fine-tuning with CLIP space consistency loss.
result Our model generates diverse outputs with the target texture and outperforms baseline models.
Existing approaches for training neural networks with user-level differential privacy (e.g., DP Federated Averaging) in federated learning (FL) settings involve bounding the contribution of each user's model update by clipping it to some constant value. However there is no good a priori setting of the clipping norm acr…
Clip21 improves convergence of gradient-clipped methods in DP settings.
problem Gradient clipping introduces bias in distributed training, causing convergence issues.
method Clip21 designs an error feedback mechanism to mitigate bias in gradient-clipped methods.
result Clip21 converges at the same rate as distributed gradient descent, improving from previous $\mathcal{O}\left(\frac{1}{\sqrt{K}}
ight)$ to $\mathcal{O}\left(\frac{1}{K}
ight)$.
New method for estimating covariance with robustness to outliers.
problem Estimating covariance from noisy data with outliers.
method Cross-fitted clipped covariance estimator with computable Bernstein certificates.
result The method balances certified stochastic error and robust hold-out proxy for clipping bias.
Paper proves convergence for private FL on non-Lipschitz convex objectives using normalization instead of clipping.
problem Lack of convergence results for differentially private federated learning with non-Lipschitz objectives.
method Developed a convergence result for private FL on smooth convex objectives without assuming Lipschitzness, using normalization instead of clipping.
result Normalization-based private FL algorithm converges better than clipping-based counterpart on smooth convex functions.
INF-clip optimizes heavy-tailed MAB problems with improved performance.
problem Optimizing multi-armed bandit problems with heavy-tailed rewards.
method INF-clip algorithm for adversarial and stochastic heavy-tailed MAB settings.
result INF-clip is optimal for linear and non-linear heavy-tailed stochastic MAB problems.
Paper explores whether gradient normalization can replace clipping for SGD in heavy-tailed noise.
problem Ensuring convergence of SGD in heavy-tailed noise.
method Revisits gradient clipping and normalization, proving their sufficiency and effectiveness.
result Gradient normalization alone is sufficient for nonconvex SGD convergence under smoothness assumptions.
Many continuous control tasks have bounded action spaces. When policy gradient methods are applied to such tasks, out-of-bound actions need to be clipped before execution, while policies are usually optimized as if the actions are not clipped. We propose a policy gradient estimator that exploits the knowledge of action…
In importance sampling (IS)-based reinforcement learning algorithms such as Proximal Policy Optimization (PPO), IS weights are typically clipped to avoid large variance in learning. However, policy update from clipped statistics induces large bias in tasks with high action dimensions, and bias from clipping makes it di…
TROLL improves RL for LLMs by replacing clipping with a trust region projection.
problem Clipping in RL for LLMs causes instability and suboptimal performance.
method TROLL uses a discrete differentiable trust region projection to replace clipping, balancing computational cost and effectiveness.
result TROLL consistently outperforms PPO-like clipping in training speed, stability, and final success rates.
New analysis shows how to balance privacy and accuracy in deep learning.
problem Balancing privacy and accuracy in deep learning models.
method Continuous time analysis through neural tangent kernel (NTK) for arbitrary architectures.
result Large clipping norm improves calibration without sacrificing accuracy.
In this paper we study the fundamental problems of maximizing a continuous non-monotone submodular function over the hypercube, both with and without coordinate-wise concavity. This family of optimization problems has several applications in machine learning, economics, and communication systems. Our main result is the…
Robust CLIP improves vision models' resistance to attacks.
problem Vulnerability of vision models to adversarial attacks.
method Unsupervised adversarial fine-tuning of CLIP.
result Robust CLIP prevents stealth attacks on vision models.
Theory for RLHF generalization under reward shift and clipped KL.
problem Theoretical understanding of RLHF generalization, especially with reward shift and clipped KL.
method Developed generalization theory for RLHF, accounting for reward shift and clipped KL.
result Presented generalization bounds for RLHF, suggesting generalization error from sampling, reward shift, and KL clipping.
Enhanced metrics improve generative model evaluation reliability.
problem Lack of reliable quality metrics for generative models.
method Introduce Clipped Density and Clipped Coverage metrics.
result Metrics prevent out-of-distribution samples from biasing quality scores.
Improved Clipped-SGD achieves near-optimal heavy-tailed statistical estimation in streaming settings.
problem High-dimensional heavy-tailed statistical estimation in streaming with memory constraints.
method Stochastic convex optimization with Clipped-SGD, proving near-optimal sub-Gaussian statistical rates.
result Clipped-SGD achieves an error of with probability .
New algorithm resists up to half of Byzantine workers in distributed learning.
problem Resilience of distributed SGD in the presence of Byzantine attackers.
method Lipschitz-inspired coordinate-wise median approach (LICM-SGD).
result LICM-SGD can resist up to half of Byzantine workers in non-convex settings.
New optimization method combines gradient clipping and non-Euclidean smoothness.
problem Improving optimization in non-Euclidean spaces for machine learning.
method Hybrid of steepest descent and conditional gradient, incorporating weight decay.
result Achieves optimal convergence rate and demonstrates effectiveness in deep learning.
We consider the problem of recovering a low-rank matrix from its clipped observations. Clipping is conceivable in many scientific areas that obstructs statistical analyses. On the other hand, matrix completion (MC) methods can recover a low-rank matrix from various information deficits by using the principle of low-ran…
New method improves convergence of SGD for heavy-tailed noise.
problem Heavy-tailed noise in machine learning applications.
method Normalized SGD (NSGD) to handle heavy-tailed noise.
result Improved sample complexity and high-probability convergence.
Paper compares RL models for finance, finding Reward Clipping best.
problem Optimizing asset allocation in finance.
method Actor-only, actor-critic, and PPO models compared; Reward Clipping introduced.
result Reward Clipping model outperforms others in bull and bear markets.
Fast algorithm for rescaling vectors with clipping, improving training efficiency.
problem Efficiently rescale vectors to a desired length while maintaining them within a domain after clipping.
method Analytical solution for optimal rescaling using fast and differentiable algorithm.
result Optimal rescaling can be found analytically, improving training efficiency for neural networks.
PPOS improves PPO by smoothing the surrogate objective function.
problem Performance instability and optimization inefficiency in PPO.
method Use of a functional clipping method instead of a flat clipping method.
result PPOS conducts more accurate updates and outperforms other PPO variants.