Paper proposes SCALLION and SCAFCOM for compressed FL with reduced communication.
problem Reducing communication overhead in Federated Learning with data heterogeneity and partial participation.
method Revisit and simplify stochastic controlled averaging, proposing SCALLION and SCAFCOM for unbiased and biased compression.
result SCALLION and SCAFCOM outperform existing methods in communication and computation complexities.
New uncertainty principle limits compression in distributed learning, suggesting optimal methods.
problem Minimizing communication cost while maintaining message quality in distributed learning.
method Formalized uncertainty principle for compression operators, introduced Kashin compression.
result Explicit formula for Kashin compression's variance bound, dimension independent.
Paper improves communication in distributed optimization, reducing worker-to-server data exchanges.
problem Efficiency in server-to-worker communication in distributed optimization.
method MARINA-P, a novel downlink compression method using correlated compressors; M3, combining MARINA-P with uplink compression.
result MARINA-P achieves provably superior server-to-worker communication complexity with increasing number of workers.
BICompFL tackles bi-directional compression challenges in stochastic FL, reducing communication costs by an order of magnitude.
problem Communication bottleneck in federated learning, especially with stochastic updates.
method Introduces BICompFL, a bi-directional compression approach for stochastic federated learning.
result Significantly reduces communication costs (by an order of magnitude) while maintaining accuracy.
Paper proposes ADC framework to reduce ViT SL training communication overhead.
problem Reducing communication overhead in ViT SL training.
method Two parallel compression strategies: class-agnostic merging and token discarding.
result Significantly reduces communication overhead without sacrificing accuracy.
Flexible framework improves communication efficiency across various systems.
problem Reducing communication between nodes in machine learning tasks.
method Adapts compression level to true gradient at each iteration, optimizing per-bit improvement.
result Automatic tuning strategies significantly increase communication efficiency.
SPARQ-SGD optimizes communication in decentralized SGD with event-triggered and compressed updates.
problem Efficient communication in decentralized stochastic optimization for large-scale models.
method Event-triggered and compressed algorithm with quantized and sparsified model parameters.
result SPARQ-SGD converges with efficiency comparable to uncompressed training, demonstrating significant communication savings.
New algorithm reduces communication in distributed learning by sharing compressed beliefs.
problem Efficiently learning from private data in a distributed setting with large hypothesis sets.
method Proposes a belief update rule for distributed cooperative learning with compressed (sparse or quantized) beliefs.
result Beliefs converge almost surely to optimal hypotheses with a linear concentration rate.
Paper introduces MoTEF for faster decentralized optimization with compressed communication.
problem Efficiency bottleneck in decentralized machine learning applications.
method Integrates communication compression with Momentum Tracking and Error Feedback.
result Significantly outperforms existing methods under arbitrary data heterogeneity.
Unified analysis of federated learning with compression for various data distributions.
problem Communication overhead in federated learning with unreliable or limited communication.
method Periodic compressed communication and local gradient tracking schemes.
result Sharp convergence rates for various objective functions and data distributions.
New method reduces communication costs in distributed nonconvex optimization.
problem Large communication costs between central server and local workers in distributed learning.
method Communication-compressed AMSGrad for distributed nonconvex optimization.
result Converges to first-order stationary point with same iteration complexity as vanilla AMSGrad.
Novel compression method preserves privacy while reducing communication costs.
problem Reducing communication costs in differential privacy mechanisms.
method Poisson private representation (PPR) for compressing and simulating local randomizers.
result Achieves compression within a logarithmic gap from theoretical lower bound.
BEER accelerates decentralized nonconvex optimization to O ( 1 / T ) O(1/T) O ( 1/ T ) rate.
problem Communication bottleneck in decentralized machine learning.
method Communication-compressed algorithm with gradient tracking.
result Converges at O ( 1 / T ) O(1/T) O ( 1/ T ) rate, matching uncompressed performance. Distributed sensors compress and send features to a fusion center for linear regression.
problem Efficiently compress and transmit features from distributed sensors to a fusion center under varying communication constraints.
method Designs a distributed and adaptive feature compression scheme using optimal quantizers and simple adaptive strategies.
result Demonstrates improved inference performance through simulated experiments.
DP-REC combines privacy and communication efficiency in federated learning.
problem Combining privacy and communication efficiency in federated learning.
method DP-REC uses Relative Entropy Coding (REC) for compression and a minor modification for differential privacy.
result DP-REC reduces communication costs while maintaining privacy comparable to state-of-the-art methods.
New compression schemes save communication in distributed mean estimation.
problem Efficiently compressing vectors for mean estimation in a limited communication setting.
method Collaborative compression schemes that exploit vector similarities without requiring known correlations.
result Analysis shows varying error types with vector similarity.
Paper introduces a new adaptive gradient method with gradient compression for distributed training.
problem Communication overhead in distributed machine learning systems.
method Adaptive gradient method with gradient compression, scalable system BytePS-Compress.
result Convergence rate of O ( 1 / T ) \mathcal{O}(1/\sqrt{T}) O ( 1/ T ) for non-convex problems. HSQ reduces communication costs in federated learning.
problem High cost of communicating gradients in federated learning.
method Hyper-sphere quantization (HSQ) framework for efficient gradient compression.
result HSQ achieves O ( log d ) O(\log d) O ( log d ) per-iteration communication cost, significantly reducing costs without compromising accuracy. PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.
problem Communication bottleneck in decentralized deep learning models.
method Low-rank linear compressors applied on model differences using power iteration steps.
result Asymptotically independent of network and compression, faster convergence, and comparable performance to tuned compression algorithms.
New method reduces communication in deep learning training.
problem Communication overhead in distributed deep learning training.
method Random-block sparsification to reduce gradients communicated.
result Performance close to standard SGD with reduced communication.
We consider decentralized stochastic optimization with the objective function (e.g. data samples for machine learning task) being distributed over n n n machines that can only communicate to their neighbors on a fixed communication graph. To reduce the communication bottleneck, the nodes compress (e.g. quantize or sparsi…
Paper proposes DCT for efficient hybrid parallel training of large recommendation models.
problem Training large recommendation models at scale with efficient communication.
method Dynamic Communication Thresholding (DCT) for both Data Parallelism and Model Parallelism.
result Reduces communication by 100x and 20x during DP and MP, respectively, improving training time by 37%.
Communication is a key bottleneck in distributed training. Recently, an \emph{error-compensated} compression technology was particularly designed for the \emph{centralized} learning and receives huge successes, by showing significant advantages over state-of-the-art compression based methods in saving the communication…
Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote …
Clapping reduces memory usage in distributed optimization by reusing data samples.
problem Significant communication overhead and impractical memory overhead in pipeline-parallel distributed optimization.
method Lazy sampling strategy to reuse data samples across steps, supporting convergence without unbiased gradient assumptions.
result Clapping achieves convergence in few-epoch or online training regimes without sample-size memory overhead.
Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suff…
In distributed optimization and machine learning, multiple nodes coordinate to solve large problems. To do this, the nodes need to compress important algorithm information to bits so that it can be communicated over a digital channel. The communication time of these algorithms follows a complex interplay between a) the…
DORE reduces communication costs in distributed learning by 95%.
problem High communication costs in distributed learning for large-scale models.
method DORE: Double Residual Compression Algorithm for stochastic gradient descent.
result DORE reduces communication costs by over 95%.
COMP-AMS optimizes distributed training with compressed gradients, achieving similar accuracy with less communication.
problem Efficiently training large-scale models in distributed environments with reduced communication costs.
method Distributed optimization framework using gradient averaging and adaptive AMSGrad, with gradient compression and error feedback.
result COMP-AMS achieves the same convergence rate and linear speedup as standard AMSGrad with less communication.
New methods reduce private federated learning communication automatically.
problem Reducing communication in private federated learning.
method Automatic compression rate adjustment based on training error, using secure aggregation and differential privacy.
result Provable instance-optimal for mean estimation, achieving favorable compression rates.
Paper studies fundamental limits of communication in distributed learning.
problem Communication efficiency in model aggregation for distributed learning.
method Rate-Distortion approach to model aggregation as a vector Gaussian CEO problem.
result Derives rate region bound and sum-rate-distortion function for model aggregation.
We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on …
Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. Th…
Currently, progressively larger deep neural networks are trained on ever growing data corpora. As this trend is only going to increase in the future, distributed training schemes are becoming increasingly relevant. A major issue in distributed training is the limited communication bandwidth between contributing nodes o…
Proposes a new method for clustering compressed data.
problem Limited communication bandwidth and low-power consumption.
method Joint Variational Autoencoders with Bernoulli mixture models (VAB).
result The model can perform clustering in the compressed data domain.
This work proposes ACTC for adaptive distributed learning under communication constraints.
problem Adaptive distributed learning in networks with communication constraints.
method ACTC (Adapt-Compress-Then-Combine) strategy with diffusion exchange of compressed updates.
result ACTC iterates converge to the optimizer with significant bit savings.
Proposes a method to improve Byzantine-robustness in compressed federated learning.
problem Byzantine-robustness in compressed federated learning.
method Gradient difference compression and stochastic average gradient algorithm (SAGA).
result The proposed method reaches a neighborhood of the optimal solution at a linear convergence rate.
Recently, researchers proposed various low-precision gradient compression, for efficient communication in large-scale distributed optimization. Based on these work, we try to reduce the communication complexity from a new direction. We pursue an ideal bijective mapping between two spaces of gradient distribution, so th…
Optimizing distributed learning systems is an art of balancing between computation and communication. There have been two lines of research that try to deal with slower networks: {\em communication compression} for low bandwidth networks, and {\em decentralization} for high latency networks. In this paper, We explore a…
APMSqueeze improves Adam for faster training with less communication.
problem Adam's inefficiency in parallel training due to gradient compression limitations.
method Error-compensated gradient compression for Adam-preconditioned Momentum SGD.
result Significantly reduces communication time per epoch without sacrificing accuracy.
New approach for sharing deep learning costs between devices and cloud.
problem Prohibitive deep learning computational requirements for embedded devices.
method Study of representation compressibility in MobileNetV2 for balancing computation, bandwidth, and accuracy.
result An optimal splitting layer for network can be found with a simple PCA-based compression scheme.
Federated Learning allows multiple parties to jointly train a deep learning model on their combined data, without any of the participants having to reveal their local data to a centralized server. This form of privacy-preserving collaborative learning however comes at the cost of a significant communication overhead du…
Optimal privacy and accuracy in distributed mean estimation with compression.
problem Achieving optimal accuracy under privacy and communication constraints.
method Compression to reduce communication while maintaining privacy and accuracy.
result Achieves optimal error with significantly reduced communication.
Network agents solve adaptive regression problems with compressed signals.
problem Solving regression problems in networks with communication constraints.
method ACTC diffusion strategy with randomized differential compression.
result Optimized resource allocation improves performance.
New biased compression methods lead to faster convergence in distributed learning.
problem Improving convergence rates in distributed learning with biased compression.
method Study of three classes of biased compression operators in distributed learning.
result Biased compressors can lead to linear convergence rates in both single node and distributed settings.
DiffSketch combines privacy and communication efficiency in distributed learning.
problem Privacy and communication efficiency in distributed machine learning.
method DiffSketch uses Count Sketch for data stream summarization to achieve both privacy and efficiency.
result DiffSketch provides strong differential privacy guarantees and significant communication compression.
New techniques improve distributed training with compressed gradients.
problem Gradient mismatch problem in local error feedback.
method Step-ahead error feedback and error averaging techniques.
result Our methods handle gradient mismatch and train faster than full-precision training.
EControl improves fast distributed optimization with compression and error control.
problem Stable convergence issues in distributed training with compression.
method Proposes EControl to regulate error compensation and prove fast convergence.
result Proves fast convergence for EControl in various convex settings without additional assumptions.