arXiv research

Paper introduces a new adaptive gradient method with gradient compression for distributed training.

Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suff…

Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed gradient methods operating with staled and compressed gradients. Non-asymptotic bo…

COMP-AMS optimizes distributed training with compressed gradients, achieving similar accuracy with less communication.

We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on …

Proposes a method to improve Byzantine-robustness in compressed federated learning.

We propose and analyze a new type of stochastic first order method: gradient descent with compressed iterates (GDCI). GDCI in each iteration first compresses the current iterate using a lossy randomized compression technique, and subsequently takes a gradient step. This method is a distillation of a key ingredient in t…

Data compression is a popular technique for improving the efficiency of data processing workloads such as SQL queries and more recently, machine learning (ML) with classical batch gradient methods. But the efficacy of such ideas for mini-batch stochastic gradient descent (MGD), arguably the workhorse algorithm of moder…

Gradient sparsification enhances privacy-preserving machine learning models.

Paper improves compressed SGD to reach second-order stationary points.

Data parallelism can boost the training speed of convolutional neural networks (CNN), but could suffer from significant communication costs caused by gradient aggregation. To alleviate this problem, several scalar quantization techniques have been developed to compress the gradients. But these techniques could perform …

New techniques improve distributed training with compressed gradients.

New method reduces communication costs in distributed nonconvex optimization.

New method reduces communication in deep learning training.

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD does not converge to the optimum. Further, even when it does conver…

Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote …

Recently, researchers proposed various low-precision gradient compression, for efficient communication in large-scale distributed optimization. Based on these work, we try to reduce the communication complexity from a new direction. We pursue an ideal bijective mapping between two spaces of gradient distribution, so th…

In the last few years, various communication compression techniques have emerged as an indispensable tool helping to alleviate the communication bottleneck in distributed learning. However, despite the fact biased compressors often show superior performance in practice when compared to the much more studied and underst…

IntSGD compresses SGD gradients without floats, converging as SGD.

Paper uses SGLD to recover signals from generative models, proving convergence under mild conditions.

Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to …

Unified framework for distributed compressed SGD under $(L_0, L_1)$-smoothness.

PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.

Flexible framework improves communication efficiency across various systems.

This paper aims at achieving a "good" estimator for the gradient of a function on a high-dimensional space. Often such functions are not sensitive in all coordinates and the gradient of the function is almost sparse. We propose a method for gradient estimation that combines ideas from Spall's Simultaneous Perturbation …

Physics-inspired methods optimize SVD compression of LLMs.

Unified analysis of federated learning with compression for various data distributions.

Autoencoders fail to capture sparse structure in 1-bit data compression.

Clapping reduces memory usage in distributed optimization by reusing data samples.

Faster reconstruction of compressed signals using conditional GAN and NPGD.

Paper proposes a compression principle for neural networks using Bayesian optimization.

Paper proposes ADC framework to reduce ViT SL training communication overhead.

A new method learns graph compression from data.

Proposes using equivariant generative models for compressed sensing with unknown orientations.

We develop a communication-efficient distributed learning algorithm that is robust against Byzantine worker machines. We propose and analyze a distributed gradient-descent algorithm that performs a simple thresholding based on gradient norms to mitigate Byzantine failures. We show the (statistical) error-rate of our al…

We extend quantization-aware training to extreme model compression.

Unified sign-based compression for federated learning with faster convergence.

Multiple gossip steps improve decentralized optimization convergence.

Unified analysis of SGD variants for nonconvex federated optimization.

Paper proposes LC-Checkpoint for efficient deep learning model checkpoints.

IDF++ improves integer discrete flows for lossless compression.

LEAD algorithm speeds up decentralized optimization with compression.

Paper studies fundamental limits of communication in distributed learning.

A new method compresses large datasets for gradient descent algorithms efficiently.

Adaptive quantization improves SGD accuracy in data-parallel settings.

New method estimates rate-distortion function using optimal transport.

Analyze SGD with biased gradients, improving convergence rates and accuracy.

Two algorithms improve federated learning efficiency and resilience.