A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
We present ErasureHead, a new approach for distributed gradient descent (GD) that mitigates system delays by employing approximate gradient coding. Gradient coded distributed GD uses redundancy to exactly recover the gradient at each iteration from a subset of compute nodes. ErasureHead instead uses approximate gradien…
The reparameterization gradient has become a widely used method to obtain Monte Carlo gradients to optimize the variational objective. However, this technique does not easily apply to commonly used distributions such as beta or gamma without further approximations, and most practical applications of the reparameterizat…
Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suff…
DANA mitigates gradient staleness in asynchronous distributed SGD with momentum.
problem Gradient staleness in asynchronous distributed SGD with momentum.
method DANA: a novel technique for asynchronous distributed SGD with momentum that computes the gradient on an estimated future position of the model's parameters.
result DANA fully incorporates momentum in asynchronous training with almost no ramifications to final accuracy.
REGS samples from unnormalized distributions using gradient flow and neural networks.
problem Sampling from unnormalized distributions with high accuracy and efficiency.
method REGS is a particle method that iteratively transforms samples from a reference distribution to match an unnormalized target distribution using Wasserstein gradient flow and neural networks.
result REGS outperforms state-of-the-art methods in sampling from challenging multimodal distributions and real datasets.
Gradient flows on distributions of distributions for machine learning tasks.
problem Designing gradient flows for datasets of probability distributions.
method Representing classes as conditional distributions, modeling datasets as mixture distributions, using Wasserstein over Wasserstein (WoW) distance and gradients.
result Demonstrated gradient flows for dataset transfer and distillation tasks.
Clip21 improves convergence of gradient-clipped methods in DP settings.
problem Gradient clipping introduces bias in distributed training, causing convergence issues.
method Clip21 designs an error feedback mechanism to mitigate bias in gradient-clipped methods.
result Clip21 converges at the same rate as distributed gradient descent, improving from previous $\mathcal{O}\left(\frac{1}{\sqrt{K}}
ight)$ to $\mathcal{O}\left(\frac{1}{K}
ight)$.
Gradient-based methods can be biased by distributional asymmetries in bivariate categorical data.
problem Gradient-based causal discovery methods can be biased by distributional asymmetries in bivariate categorical data.
method Identified and examined two distributional biases: Marginal Distribution Asymmetry and Marginal Distribution Shift Asymmetry. Employed two simple models to demonstrate and control these biases.
result Gradient-based methods can be biased by distributional asymmetries, and these biases can be controlled.
This paper presents a new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation. Simple rules are designed to detect slowly-varying gradients and, therefore, trigger the reuse of outdated gradients. The resultant gra…
Stagewise boosting improves gradient boosting for distributional regression.
problem Vanishing gradient in gradient boosting for distributional regression leads to suboptimal models.
method Proposes a stagewise boosting-type algorithm for distributional regression, combining stagewise regression ideas with gradient boosting and incorporating a novel regularization method, correlation filtering.
result The proposed algorithm provides better results, especially for complex distributions, by reducing the risk of being trapped in a local optimum.
Optimal gradient quantization reduces communication costs in distributed deep learning.
problem High communication costs in distributed training of deep neural networks.
method Deduced optimal gradient quantization conditions for binary and multi-level quantization, developed novel schemes for dynamic quantization levels.
result Demonstrated superior performance of proposed quantization schemes on CIFAR and ImageNet datasets.
Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed gradient methods operating with staled and compressed gradients. Non-asymptotic bo…