Deep learning frameworks leverage GPUs to perform massively-parallel computations over batches of many training examples efficiently. However, for certain tasks, one may be interested in performing per-example computations, for instance using per-example gradients to evaluate a quantity of interest unique to each examp…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Per-Example Gradients”
A method predicts GNS of transformer layers using normalization layer norms.
problem Estimating gradient noise scale with minimal variance.
method Simultaneously compute per-example gradient norms and parameter gradients.
result Total GNS is predicted well by normalization layer GNS.
This technical report describes an efficient technique for computing the norm of the gradient of the loss function for a neural network with respect to its parameters. This gradient norm can be computed efficiently for every example.
New methods speed up training of differentially private deep learning models.
problem Training differentially private deep learning models is slower than non-private models.
method Derive and implement new per-example gradient clipping methods compatible with auto-differentiation.
result Significant training speed-ups (54x - 94x) for various models and architectures.
PEGR improves deep learning models' robustness against noisy data.
problem Learning signals from noisy data in deep learning models.
method Per-example gradient regularization (PEGR) to suppress noise.
result PEGR enhances test error and robustness against noise perturbations.
Proposes efficient Bayesian logistic regression for large sparse datasets.
problem Infeasibility of theoretical Bayesian methods for large sparse feature sets.
method Low complexity analytical approximations for sparse online logistic and probit regressions.
result Empirical results show superior performance compared to more complex methods.
New metric -coherence measures gradient alignment during training, revealing surprising memorization patterns.
problem Measuring and understanding the alignment of per-example gradients during training.
method Introducing -coherence as a metric to study gradient alignment, showing its advantages over existing metrics.
result Training with random labels leads to high -coherence, indicating common patterns even when generalization is not possible.
New methods validate a hypothesis explaining how neural nets generalize well.
problem Why over-parameterized nets generalize well despite memorizing training data.
method Developed new algorithms to suppress weak gradient directions without per-example gradients.
result Validated a hypothesis about gradient directions and their role in generalization.
Theoretical Limits of Pipeline Parallel Optimization and Application to Distributed Deep Learningstat.ML
We investigate the theoretical limits of pipeline parallel learning of deep learning architectures, a distributed setup in which the computation is distributed per layer instead of per example. For smooth convex and non-convex objective functions, we provide matching lower and upper complexity bounds and show that a na…
We develop new stochastic gradient methods for efficiently solving sparse linear regression in a partial attribute observation setting, where learners are only allowed to observe a fixed number of actively chosen attributes per example at training and prediction times. It is shown that the methods achieve essentially a…
Loss-guided training accelerates node embedding methods on graphs.
problem Training efficiency in graph learning methods with implicit positive examples.
method Dynamic adjustment of training distribution based on loss values.
result Significant acceleration in training and computation over static methods.
New method for private learning with fairness constraints.
problem Rate-constrained optimization under differential privacy.
method RaCO-DP, a DP variant of SGDA solving Lagrangian formulation.
result Empirical results show RaCO-DP outperforms existing methods.
Proposes sigmoidF1 loss for multilabel classification, improving performance metrics.
problem Lack of smooth, tractable loss functions for multilabel classification.
method Introduces sigmoidF1, a smooth F1 score surrogate loss function.
result sigmoidF1 outperforms other loss functions on various datasets and metrics.
Deep networks learn clean structure before memorizing corrupted labels, leaving a spectral signature in gradient centered scatter.
problem Deep networks' transition from learning clean structure to memorizing corrupted labels under label noise.
method Analysis of the centered scatter of per-example last-layer gradients to identify Fisher Rank Inflation.
result Fisher Rank Inflation is a spectral signature of memorization under label noise, with effective rank expanding during memorization.
We introduce an efficient method for training the linear ranking support vector machine. The method combines cutting plane optimization with red-black tree based approach to subgradient calculations, and has O(m*s+m*log(m)) time complexity, where m is the number of training examples, and s the average number of non-zer…
New method visualizes tabular feature semantics for better model understanding.
problem Lack of feature interaction interpretation in tabular ML models.
method Feature Vectors method for global tabular dataset interpretability.
result Visualizes semantic relationships among tabular features.
Complex classification performance metrics such as the F-measure and Jaccard index are often used, in order to handle class-imbalanced cases such as information retrieval and image segmentation. These performance metrics are not decomposable, that is, they cannot be expressed in a per-example manner, which hinder…
A method for predicting credal sets in classification tasks using conformal prediction.
problem Designing methods for learning credal set predictors in machine learning.
method Incorporates conformal prediction for predicting credal sets in classification tasks.
result Conformal credal sets are guaranteed to be valid with high probability.
S2M optimizes mining for diverse data subpopulations.
problem Scalability and uniformity in training sets with many labels and diverse data.
method Doubly-stochastic mining (S2M) computes per-example and minibatch losses on hardest labels/examples.
result S2M ensures good performance across all data subpopulations.
Objective: We aim to provide an algorithm for the detection of myocardial infarction that operates directly on ECG data without any preprocessing and to investigate its decision criteria. Approach: We train an ensemble of fully convolutional neural networks on the PTB ECG dataset and apply state-of-the-art attribution …
A new framework for interpretable models using sparse linear layers.
problem Performance degradation and lower interpretability in concept bottleneck models.
method Contrastive Language Image models and a single sparse linear layer with Bayesian inference.
result Our framework outperforms recent CBM approaches in accuracy and concept sparsity.
Paper proposes methods to learn with multiple incorrect labels per example.
problem Learning with a single incorrect label per example limits potential.
method Proposes a novel problem setting allowing multiple incorrect labels per example and two learning methods.
result Demonstrates improved learning with multiple incorrect labels compared to single incorrect labels.
The paper proposes a new method for comparing logistic regression models across different populations.
problem Comparing logistic regression models across sub-populations can lead to misleading results.
method Develops a cascading set of equivalence tests for logistic regression models, addressing coding, predictions, and overall accuracy.
result Equivalence testing incentivizes accurate inference and avoids perverse incentives from significance tests.
Path-independent equilibrium models improve network performance on harder problems.
problem Improving network performance on harder problem instances.
method Investigated path-independent equilibrium models and their impact on network performance.
result Path independence correlates with better performance on harder problem instances.
The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice…
Many state-of-the-art results obtained with deep networks are achieved with the largest models that could be trained, and if more computation power was available, we might be able to exploit much larger datasets in order to improve generalization ability. Whereas in learning algorithms such as decision trees the ratio …
This work makes neural sequence models more efficient by controlling computation.
problem Fixed compute for all examples in neural networks.
method Conditional computation to adapt compute to example complexity.
result Conditional Computation Transformer (CCT) improves efficiency and performance.
End-to-end framework learns from imperfect annotations directly.
problem Training machine learning models on imperfect human annotations.
method End-to-end framework merging aggregation with model training and modeling annotator competencies.
result Accuracy gains of up to 25% over state-of-the-art annotation aggregation methods.
Paper proposes a GPU-based system for training massive deep learning models in ads systems.
problem Training massive deep learning models with terabyte-scale parameters in ads systems.
method Hierarchical GPU parameter server with 3-layer storage (GPU High-Bandwidth Memory, CPU main memory, SSD).
result 4-node hierarchical GPU parameter server trains a model 2X faster than a 150-node in-memory system.
Proposes semi-supervised feature ranking for handling high-dimensional, unlabeled data.
problem Handling high-dimensional, unlabeled data in machine learning.
method Tree ensembles and Relief family algorithms for semi-supervised feature ranking.
result Semi-supervised feature ranking outperforms supervised methods across most datasets.
New approach removes data influence in high dimensions with single step.
problem Efficiently removing data influence in high-dimensional settings with strong convexity and smoothness assumptions.
method Introduces ε-Gaussian certifiability and analyzes Newton method performance.
result Single Newton step followed by Gaussian noise achieves privacy and accuracy.
Study shows gradient variance increases during deep learning training, contrary to common belief.
problem Understanding and minimizing gradient variance in deep learning models.
method Gradient Clustering method using stratified sampling to minimize gradient variance.
result Gradient variance increases during training, and smaller learning rates coincide with higher variance.
Researchers compare different gradient methods for ridge regression, finding conjugate gradients have similar performance.
problem Comparing statistical properties of different gradient methods in ridge regression.
method Explicit non-standard error decomposition to bound prediction error of conjugate gradient iterates.
result Conjugate gradient iterates share optimality properties with gradient flow and ridge regression up to a constant factor.
Gradient descent implicitly regularizes neural networks by penalizing large loss gradients.
problem How to optimize deep neural networks without explicit regularization.
method Backward error analysis to calculate implicit gradient regularization and demonstrate its effectiveness empirically.
result Implicit gradient regularization biases gradient descent toward flat minima, improving model robustness and test errors.
Improved sampling method using regularized Stein Variational Gradient Flow.
problem Improving the accuracy of sampling methods in machine learning.
method Proposed Regularized Stein Variational Gradient Flow to interpolate between SVGD and Wasserstein Gradient Flow.
result Established theoretical properties and provided preliminary numerical evidence of improved performance.
In this work we revisit gradient regularization for adversarial robustness with some new ingredients. First, we derive new per-image theoretical robustness bounds based on local gradient information. These bounds strongly motivate input gradient regularization. Second, we implement a scaleable version of input gradient…
FGD reduces noisy gradient variance in SGD for neural networks.
problem Noisy and unreliable gradient estimation in SGD for deep learning.
method Solves an adaptive filtering problem to consistently estimate the local gradient.
result Significantly reduces gradient variance and accelerates convergence.
Proposes a new adaptive gradient method based on gradient differences.
problem Manual tuning of stepsize in vanilla gradient methods.
method Adaptation driven by cumulative squared norms of gradient differences.
result More robust than AdaGrad in various settings.
New methods using natural gradient for structured optimization.
problem Structured optimization problems.
method Structured second-order methods via natural gradient descent.
result Efficiency demonstrated on non-convex and deep learning problems.
CProp scales learning rate based on past gradient direction.
problem Adaptive learning rate scheduling during training.
method Gradient scaling method based on past gradient conformity.
result Significant gain in training speed on SGD and adaptive methods.
The present paper develops a novel aggregated gradient approach for distributed machine learning that adaptively compresses the gradient communication. The key idea is to first quantize the computed gradients, and then skip less informative quantized gradient communications by reusing outdated gradients. Quantizing and…
Paper examines the structure of stochastic gradients in deep learning.
problem Exploring the structure and heavy tails of stochastic gradients in deep learning.
method Conducted formal statistical tests on stochastic gradients and gradient noise.
result Stochastic gradients and gradient noise do not exhibit power-law heavy tails, but their covariance spectra do.
Faster policy learning via continuous-time gradients.
problem Efficiently estimating policy gradients for continuous-time systems.
method Approximating continuous-time gradients directly, using adaptive discretization.
result More efficient policy gradient estimator leads to faster learning.
Optimization with noisy gradients has become ubiquitous in statistics and machine learning. Reparameterization gradients, or gradient estimates computed via the "reparameterization trick," represent a class of noisy gradients often used in Monte Carlo variational inference (MCVI). However, when these gradient estimator…
Gradient flossing stabilizes RNN training by controlling Lyapunov exponents.
problem Gradient instability in RNNs leading to exploding and vanishing gradients.
method Regularizing Lyapunov exponents through backpropagation using differentiable linear algebra.
result Gradient flossing improves RNN training success rate and convergence speed.
Off-policy stochastic actor-critic methods rely on approximating the stochastic policy gradient in order to derive an optimal policy. One may also derive the optimal policy by approximating the action-value gradient. The use of action-value gradients is desirable as policy improvement occurs along the direction of stee…
Adaptive methods improve gradient descent and proximal gradient for convex optimization.
problem Improving efficiency of gradient descent and proximal gradient methods.
method Adaptive versions of GD and ProxGD using local curvature information.
result Proved convergence with local Lipschitz gradient assumptions.
Stein variational gradient decent (SVGD) has been shown to be a powerful approximate inference algorithm for complex distributions. However, the standard SVGD requires calculating the gradient of the target density and cannot be applied when the gradient is unavailable. In this work, we develop a gradient-free variant …