Adversarial training is a training scheme designed to counter adversarial attacks by augmenting the training dataset with adversarial examples. Surprisingly, several studies have observed that loss gradients from adversarially trained DNNs are visually more interpretable than those from standard DNNs. Although this phe…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “gradient training”
Gradient amplification boosts deep learning model performance without increasing training time.
problem Vanishing gradients in deep neural networks.
method Gradient amplification approach to prevent vanishing gradients and training strategy to enable/disable across epochs.
result Improves performance of deep learning models with reduced training time.
Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suff…
We propose gradient adversarial training, an auxiliary deep learning framework applicable to different machine learning problems. In gradient adversarial training, we leverage a prior belief that in many contexts, simultaneous gradient updates should be statistically indistinguishable from each other. We enforce this c…
Study shows gradient variance increases during deep learning training, contrary to common belief.
problem Understanding and minimizing gradient variance in deep learning models.
method Gradient Clustering method using stratified sampling to minimize gradient variance.
result Gradient variance increases during training, and smaller learning rates coincide with higher variance.
Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodsstat.ML
Adaptive batch sizes improve local gradient methods in distributed training.
problem Communication bottlenecks in distributed deep learning.
method Adaptive batch size strategies for local gradient methods.
result Adaptive batch sizes reduce minibatch gradient variance and improve training efficiency.
New techniques improve distributed training with compressed gradients.
problem Gradient mismatch problem in local error feedback.
method Step-ahead error feedback and error averaging techniques.
result Our methods handle gradient mismatch and train faster than full-precision training.
Improves deep learning models by blending gradients from training loss and auxiliary objective.
problem Minimizing a single training loss while encouraging desirable model properties.
method Solves a bilevel optimization problem by combining training loss gradients and orthogonal projections of auxiliary gradients.
result Bloop method leads to better performance than other gradient surgery methods without EMA.
Gradient flossing stabilizes RNN training by controlling Lyapunov exponents.
problem Gradient instability in RNNs leading to exploding and vanishing gradients.
method Regularizing Lyapunov exponents through backpropagation using differentiable linear algebra.
result Gradient flossing improves RNN training success rate and convergence speed.
AB dynamically scales gradients to mitigate asynchronous training delays.
problem Gradient delay in asynchronous training reduces model performance.
method Adaptive Braking (AB) dynamically scales gradients based on alignment.
result AB enables training with up to 32 update steps of delay without accuracy loss.
This paper studies how neural network architecture affects the speed of training. We introduce a simple concept called gradient confusion to help formally analyze this. When gradient confusion is high, stochastic gradients produced by different data samples may be negatively correlated, slowing down convergence. But wh…
Paper reconstructs training data from a single gradient query.
problem Privacy threats in federated learning due to model gradients.
method Provable attack using tensor decomposition.
result Training samples can be fully reconstructed from a single gradient query.
TracIn measures how training examples affect model predictions.
problem Measuring the influence of training examples on model predictions.
method Tracing changes in loss during training when a specific example is used.
result TracIn provides a scalable method to estimate example influence.
Paper introduces a new adaptive gradient method with gradient compression for distributed training.
problem Communication overhead in distributed machine learning systems.
method Adaptive gradient method with gradient compression, scalable system BytePS-Compress.
result Convergence rate of for non-convex problems.
SAPAG attacks distributed learning by reconstructing true training data from gradients.
problem Privacy attacks on distributed learning systems through gradients.
method SAPAG uses a Gaussian kernel-based gradient difference distance measure.
result SAPAG can reconstruct training data on various DNNs and at different training phases.
Paper develops a statistical framework for quantized training of deep neural networks.
problem Lack of theoretical understanding of gradient quantization in FQT.
method Presented a statistical framework for analyzing FQT algorithms, viewing quantized gradient as a stochastic estimator of QAT gradient.
result Developed two novel gradient quantizers with smaller variance than existing per-tensor quantizer.
Improved training of RBMs using mode-assisted gradient updates.
problem Difficulty in training RBMs due to hard-to-compute gradients.
method Combining standard gradient updates with mode-assisted off-gradient direction.
result Mode training improves training speed and stability, reduces KL divergence.
PGD-trained models have a preferential direction in their gradients, which improves robustness.
problem Mathematical lack of clarity in the direction of preferential gradient alignment after adversarial training.
method Proposed a novel definition of preferential direction and evaluated it using a metric based on GANs.
result PGD-trained models have higher alignment with the proposed preferential direction than baseline models.
New method reduces training cost by using approximate gradients.
problem Training neural networks is computationally expensive.
method Uses control variates to approximate gradients without full backward pass.
result Efficacy demonstrated on a vision transformer classification task.
New method reduces communication in deep learning training.
problem Communication overhead in distributed deep learning training.
method Random-block sparsification to reduce gradients communicated.
result Performance close to standard SGD with reduced communication.
The paper analyzes how adding gradient data affects overparameterized models' performance.
problem The impact of Sobolev training on high-dimensional predictive models.
method Combining replica method from statistical physics and operator-valued free probability theory.
result Sobolev training does not universally improve performance for target functions described by single-index models.
Optimal gradient quantization reduces communication costs in distributed deep learning.
problem High communication costs in distributed training of deep neural networks.
method Deduced optimal gradient quantization conditions for binary and multi-level quantization, developed novel schemes for dynamic quantization levels.
result Demonstrated superior performance of proposed quantization schemes on CIFAR and ImageNet datasets.
PLUMAGE improves large model training efficiency and stability.
problem Accelerator memory and networking constraints during large model training.
method Probabilistic Low rank Unbiased Minimum Variance Gradient Estimator (PLUMAGE) that resolves bias and variance issues.
result PLUMAGE reduces training loss by 28% on average across the GLUE benchmark.
AOPU stabilizes NN training by approximating natural gradient, improving stability and convergence.
problem Stability and interpretability in online NN training for industrial soft sensors.
method AOPU truncates gradient backpropagation, optimizing trackable parameters, and approximating natural gradient.
result AOPU achieves stable convergence and superior performance on chemical process datasets.
New GFlowNet training framework using policy gradients for combinatorial object generation.
problem Training efficiency and robustness in GFlowNet models.
method Policy-dependent rewards and coupled training strategy for forward and backward policies.
result Advanced RL perspectives for robust gradient estimation improve GFlowNet performance.
Paper proposes an adaptive gradient method for federated learning.
problem Communication efficiency in federated learning.
method Adaptive gradient method for federated learning.
result Guaranteed convergence and communication efficiency in federated learning.
Monte Carlo method trains deep neural networks without gradients.
problem Vanishing and exploding gradients in backpropagation.
method Randomly mutate parameters, keep if loss decreases.
result Gradient-free method trains deep networks effectively.
What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training. We observe that for batch sizes …
Gradient descent biases towards stable rank networks for nearly-orthogonal data.
problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.
method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.
result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.
Optimal defenses protect FL models from gradient reconstruction attacks.
problem Gradient reconstruction attacks compromise FL models by recovering original data from shared gradients.
method Derive a theoretical lower bound of reconstruction error, customize noise and pruning defenses, and achieve optimal trade-off between leakage and utility.
result Our methods outperform Gradient Noise and Pruning in protecting training data and maintaining model utility.
In this work we revisit gradient regularization for adversarial robustness with some new ingredients. First, we derive new per-image theoretical robustness bounds based on local gradient information. These bounds strongly motivate input gradient regularization. Second, we implement a scaleable version of input gradient…
Stochastic gradient methods converge for training wide PINNs.
problem Convergence of stochastic gradient descent in training over-parameterized PINNs.
method Established linear convergence of stochastic gradient descent/flow in training over-parameterized two-layer PINNs.
result Linear convergence with high probability for general activation functions.
AutoClip automatically adjusts gradient clipping for better audio separation.
problem Improving generalization in audio source separation networks.
method Adaptive gradient clipping based on historical gradient norms.
result Improves generalization performance in audio source separation networks.
Low-rank Gradient Approximation For Memory-Efficient On-device Training of Deep Neural Networkeess.AS
Paper proposes low-rank gradient approximation to save memory for deep neural network training.
problem Memory limitation on mobile devices for deep neural network training.
method Approximating gradient matrices using low-rank parameterization.
result Reduces training memory by about 33.0% for Adam optimization and 4.5% relative lower word error rate on ASR personalization task.
Nonlinear conjugate gradient (NLCG) based optimizers have shown superior loss convergence properties compared to gradient descent based optimizers for traditional optimization problems. However, in Deep Neural Network (DNN) training, the dominant optimization algorithm of choice is still Stochastic Gradient Descent (SG…
Understanding Training-Data Leakage from Gradients in Neural Networks for Image Classificationstat.ML
This paper analyzes how training data can be leaked from gradients in neural networks and proposes a metric for measuring model security.
problem Training data leakage from gradients in neural networks for image classification.
method Formulated the problem as an optimisation problem for each layer, involving weights, gradients, and constraints from preceding layers.
result Attributed training data leakage to the architecture of the deep network and proposed a metric for measuring model security.
CProp scales learning rate based on past gradient direction.
problem Adaptive learning rate scheduling during training.
method Gradient scaling method based on past gradient conformity.
result Significant gain in training speed on SGD and adaptive methods.
Paper proposes a new method for training nonconvex models.
problem Training nonconvex models like neural networks.
method Successive functional gradient optimization using mirror descent in a function space.
result The method leads to better performance than standard training techniques.
Normalization layers are widely used in deep neural networks to stabilize training. In this paper, we consider the training of convolutional neural networks with gradient descent on a single training example. This optimization problem arises in recent approaches for solving inverse problems such as the deep image prior…
Improved DLG extracts accurate labels from gradients, overcoming DLG's convergence issues.
problem Private training data leakage from shared gradients in distributed learning systems.
method Proposes iDLG, a simple approach to synthesize accurate labels from gradients.
result iDLG reliably extracts ground-truth labels from gradients, unlike DLG.
Gradient descent can find better tensor decompositions than lazy training in over-parameterized settings.
problem Finding better tensor decompositions in over-parameterized settings.
method Gradient descent on over-parameterized tensor decomposition problems.
result Gradient descent can find an approximate tensor decomposition with rank , while lazy training requires .
Framework improves gradient estimation for faster training convergence.
problem Efficiently estimating noisy gradients in stochastic optimization.
method Dynamic adaptive importance sampling combining multiple distributions.
result Adaptively weighted multiple importance sampling yields superior gradient estimates.
Exchanging gradients is a widely used method in modern multi-node machine learning system (e.g., distributed training, collaborative learning). For a long time, people believed that gradients are safe to share: i.e., the training data will not be leaked by gradient exchange. However, we show that it is possible to obta…
SGD fails to converge for deep ReLU networks with limited random initializations.
problem SGD convergence in deep neural networks with limited random initializations.
method Analysis of four discretization parameters: network architecture, training data, gradient steps, and random initializations.
result SGD fails to converge for ReLU networks with depth much larger than width.
We apply stochastic average gradient (SAG) algorithms for training conditional random fields (CRFs). We describe a practical implementation that uses structure in the CRF gradient to reduce the memory requirement of this linearly-convergent stochastic gradient method, propose a non-uniform sampling scheme that substant…
Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcs.LG
Unified framework for analyzing neural networks trained by gradient descent.
problem Lack of generalizable guarantees for neural networks trained by gradient descent.
method Proxy convexity and proxy Polyak-Lojasiewicz inequalities.
result Unified guarantees for neural networks trained by gradient descent.
Improved KL divergence estimators for normalizing flows lead to faster convergence and better approximations.
problem Estimating KL divergences for normalizing flows efficiently and accurately.
method Path-gradient estimators for reverse and forward KL divergences.
result Path-gradient estimators lead to faster convergence and better approximation results.
SNGM improves large-batch training accuracy.
problem Improving generalization in large-batch training.
method Stochastic Normalized Gradient Descent with Momentum.
result SNGM achieves better test accuracy than MSGD and other large-batch methods.