arXiv research

Paper proposes SCALLION and SCAFCOM for compressed FL with reduced communication.

New uncertainty principle limits compression in distributed learning, suggesting optimal methods.

Paper improves communication in distributed optimization, reducing worker-to-server data exchanges.

BICompFL tackles bi-directional compression challenges in stochastic FL, reducing communication costs by an order of magnitude.

Paper proposes ADC framework to reduce ViT SL training communication overhead.

Flexible framework improves communication efficiency across various systems.

New algorithm reduces communication in distributed learning by sharing compressed beliefs.

Paper introduces MoTEF for faster decentralized optimization with compressed communication.

Unified analysis of federated learning with compression for various data distributions.

New method reduces communication costs in distributed nonconvex optimization.

Novel compression method preserves privacy while reducing communication costs.

BEER accelerates decentralized nonconvex optimization to $O(1/T)$ rate.

Distributed sensors compress and send features to a fusion center for linear regression.

DP-REC combines privacy and communication efficiency in federated learning.

New compression schemes save communication in distributed mean estimation.

Paper introduces a new adaptive gradient method with gradient compression for distributed training.

PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.

New method reduces communication in deep learning training.

We consider decentralized stochastic optimization with the objective function (e.g. data samples for machine learning task) being distributed over $n$ machines that can only communicate to their neighbors on a fixed communication graph. To reduce the communication bottleneck, the nodes compress (e.g. quantize or sparsi…

Paper proposes DCT for efficient hybrid parallel training of large recommendation models.

Communication is a key bottleneck in distributed training. Recently, an \emph{error-compensated} compression technology was particularly designed for the \emph{centralized} learning and receives huge successes, by showing significant advantages over state-of-the-art compression based methods in saving the communication…

In this paper, we propose and analyze SPARQ-SGD, which is an event-triggered and compressed algorithm for decentralized training of large-scale machine learning models. Each node can locally compute a condition (event) which triggers a communication where quantized and sparsified local model parameters are sent. In SPA…

Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote …

Clapping reduces memory usage in distributed optimization by reusing data samples.

Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suff…

In distributed optimization and machine learning, multiple nodes coordinate to solve large problems. To do this, the nodes need to compress important algorithm information to bits so that it can be communicated over a digital channel. The communication time of these algorithms follows a complex interplay between a) the…

COMP-AMS optimizes distributed training with compressed gradients, achieving similar accuracy with less communication.

New methods reduce private federated learning communication automatically.

Paper studies fundamental limits of communication in distributed learning.

We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on …

Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. Th…

Currently, progressively larger deep neural networks are trained on ever growing data corpora. As this trend is only going to increase in the future, distributed training schemes are becoming increasingly relevant. A major issue in distributed training is the limited communication bandwidth between contributing nodes o…

This work proposes ACTC for adaptive distributed learning under communication constraints.

Proposes a method to improve Byzantine-robustness in compressed federated learning.

Recently, researchers proposed various low-precision gradient compression, for efficient communication in large-scale distributed optimization. Based on these work, we try to reduce the communication complexity from a new direction. We pursue an ideal bijective mapping between two spaces of gradient distribution, so th…

Optimizing distributed learning systems is an art of balancing between computation and communication. There have been two lines of research that try to deal with slower networks: {\em communication compression} for low bandwidth networks, and {\em decentralization} for high latency networks. In this paper, We explore a…

APMSqueeze improves Adam for faster training with less communication.

Despite recent advances in architectures for mobile devices, deep learning computational requirements remains prohibitive for most embedded devices. To address that issue, we envision sharing the computational costs of inference between local devices and the cloud, taking advantage of the compression performed by the f…

The high cost of communicating gradients is a major bottleneck for federated learning, as the bandwidth of the participating user devices is limited. Existing gradient compression algorithms are mainly designed for data centers with high-speed network and achieve $O(\sqrt{d} \log d)$ per-iteration communication cost at…

Federated Learning allows multiple parties to jointly train a deep learning model on their combined data, without any of the participants having to reveal their local data to a centralized server. This form of privacy-preserving collaborative learning however comes at the cost of a significant communication overhead du…

Motivated by the ever-increasing demands for limited communication bandwidth and low-power consumption, we propose a new methodology, named joint Variational Autoencoders with Bernoulli mixture models (VAB), for performing clustering in the compressed data domain. The idea is to reduce the data dimension by Variational…

Optimal privacy and accuracy in distributed mean estimation with compression.

Network agents solve adaptive regression problems with compressed signals.

Communication and privacy are two critical concerns in distributed learning. Many existing works treat these concerns separately. In this work, we argue that a natural connection exists between methods for communication reduction and privacy preservation in the context of distributed machine learning. In particular, we…

New biased compression methods lead to faster convergence in distributed learning.

New techniques improve distributed training with compressed gradients.

EControl improves fast distributed optimization with compression and error control.

New method reduces FL communication cost by leveraging server's side information.