Optimal gradient quantization reduces communication costs in distributed deep learning.
problem High communication costs in distributed training of deep neural networks.
method Deduced optimal gradient quantization conditions for binary and multi-level quantization, developed novel schemes for dynamic quantization levels.
result Demonstrated superior performance of proposed quantization schemes on CIFAR and ImageNet datasets.
A new method reduces communication in distributed learning by skipping less informative gradient updates.
problem Efficient communication in distributed machine learning.
method Quantizes and skips less informative gradients to reduce communication overhead.
result Proves linear convergence rate similar to gradient descent with significant communication savings.
AdaQuantFL reduces communication in federated learning by adaptively quantizing model updates.
problem Efficient communication of model updates in federated learning with high-dimensional models and limited bandwidth.
method AdaQuantFL uses adaptive quantization to reduce the number of bits for model updates while maintaining low error floor.
result AdaQuantFL converges in fewer communicated bits compared to fixed quantization levels, with minimal impact on accuracy.
Moniqua improves SGD convergence with quantized communication.
problem Efficiently communicating in decentralized SGD with limited bandwidth.
method Modulo quantized communication in decentralized SGD.
result Moniqua converges at the same rate as full-precision communication with less bits.
Q-GADMM reduces communication in decentralized ML by quantizing model updates.
problem Reducing communication in decentralized ML while maintaining accuracy.
method Quantized group ADMM (Q-GADMM) with adaptive quantization.
result Q-GADMM achieves similar accuracy and convergence to non-quantized methods with less communication.
A new gradient quantization scheme improves communication efficiency in distributed training.
problem Efficiently compressing gradients for parallel training of large models.
method Proposes a new gradient quantization scheme with theoretical guarantees and empirical performance.
result The new scheme matches and exceeds the performance of existing methods.
Quantized-TinyLLaVA reduces communication costs in split learning for multimodal models.
problem High communication costs in split learning for multimodal models.
method Integrates a compression module that quantizes intermediate features into discrete representations before transmission.
result Achieves an approximate 87.5% reduction in communication overhead with 2-bit quantization.
Due to its efficiency and ease to implement, stochastic gradient descent (SGD) has been widely used in machine learning. In particular, SGD is one of the most popular optimization methods for distributed learning. Recently, quantized SGD (QSGD), which adopts quantization to reduce the communication cost in SGD-based di…
Modern distributed training of machine learning models suffers from high communication overhead for synchronizing stochastic gradients and model parameters. In this paper, to reduce the communication complexity, we propose \emph{double quantization}, a general scheme for quantizing both model parameters and gradients. …
Efficient decentralized learning framework reduces communication costs.
problem Efficiently solve optimization problems in distributed learning networks.
method Censored and Quantized Generalized GADMM (CQ-GGADMM) framework.
result Achieves linear convergence rate under strong convexity assumptions.
FedPAQ improves federated learning efficiency by averaging and quantizing updates.
problem Communication bottlenecks and scalability issues in federated learning.
method Periodic averaging, partial device participation, and quantized message-passing.
result FedPAQ achieves near-optimal theoretical guarantees and demonstrates communication-computation tradeoffs.
Quantized Adam reduces communication cost in deep learning training.
problem Reducing communication cost in distributed deep learning training.
method Gradient and weight quantization with error feedback in Adam.
result Proposed methods converge to first-order stationary points.
FTTQ optimizes quantized networks in federated learning, reducing communication costs.
problem Redundant parameters in full-precision models lead to excessive communication costs in federated learning.
method FTTQ algorithm that optimizes quantized networks on clients through self-learning quantization factors.
result FTTQ reduces communication costs and can achieve slightly better performance on non-IID data.
One of the most significant bottleneck in training large scale machine learning models on parameter server (PS) is the communication overhead, because it needs to frequently exchange the model gradients between the workers and servers during the training iterations. Gradient quantization has been proposed as an effecti…
vqSGD reduces communication in distributed optimization with convergence guarantees.
problem Reduction of communication cost in distributed optimization.
method Vector quantization schemes based on convex hull of a point set.
result Asymptotic reduction in communication cost with convergence guarantees.
Global-QSGD accelerates distributed training by up to 3.51%.
problem High communication overhead in distributed deep learning.
method Allreduce-compatible gradient quantization with theoretical guarantees.
result Global-QSGD accelerates distributed training by up to 3.51%.
Paper shows how to integrate quantization into neural compression models.
problem Integrating quantization into neural compression models.
method Integrates uniform noise channel at test time using universal quantization.
result Eliminates mismatch between training and test phases while maintaining differentiability.
Adaptive quantization improves SGD accuracy in data-parallel settings.
problem Fixed gradient quantization schemes lead to suboptimal performance in deep learning.
method Developed adaptive quantization schemes ALQ and AMQ that update compression schemes based on gradient statistics.
result Improved validation accuracy on CIFAR-10 and ImageNet datasets by 2% and 1% respectively.
As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed to perform parallel model training. One popular communication-compression method for data-parallel SGD is QSGD (Alistarh et al., 2017), which quantizes and en…
How can we efficiently mitigate the overhead of gradient communications in distributed optimization? This problem is at the heart of training scalable machine learning models and has been mainly studied in the unconstrained setting. In this paper, we propose Quantized-Frank-Wolfe (QFW), the first projection-free and co…
Asynchronous decentralized SGD with quantized and local updates converges in gossip model.
problem Scalable distributed machine learning challenges in decentralized optimization.
method Asynchronous decentralized SGD with quantization and local steps in gossip model.
result Decentralized optimization with quantization and local steps converges in asynchronous gossip model.
Study hypothesis testing under quantized samples with communication constraints, achieving near-optimal sample complexity.
problem Optimizing hypothesis testing with quantized samples and communication constraints.
method Developed a polynomial-time algorithm achieving near-optimal sample complexity under communication constraints.
result Achieved near-optimal sample complexity under communication constraints, with a logarithmic factor increase over unconstrained setting.
SPARQ-SGD optimizes communication in decentralized SGD with event-triggered and compressed updates.
problem Efficient communication in decentralized stochastic optimization for large-scale models.
method Event-triggered and compressed algorithm with quantized and sparsified model parameters.
result SPARQ-SGD converges with efficiency comparable to uncompressed training, demonstrating significant communication savings.
We consider the problem of decentralized consensus optimization, where the sum of n smooth and strongly convex functions are minimized over n distributed agents that form a connected network. In particular, we consider the case that the communicated local decision variables among nodes are quantized in order to all…
Relationships that exist between the classical, Shannon-type, and geometric-based approaches to sampling are investigated. Some aspects of coding and communication through a Gaussian channel are considered. In particular, a constructive method to determine the quantizing dimension in Zador's theorem is provided. A geom…
HSQ reduces communication costs in federated learning.
problem High cost of communicating gradients in federated learning.
method Hyper-sphere quantization (HSQ) framework for efficient gradient compression.
result HSQ achieves O(logd) per-iteration communication cost, significantly reducing costs without compromising accuracy. Differentially quantized gradient methods improve convergence in noisy communication channels.
problem Optimizing distributed learning with limited communication bandwidth and noise.
method Introduces Differential Quantization (DQ) to compensate for quantization errors in gradient descent.
result DQ-GD achieves the same contraction factor as unquantized GD at high bitrates, proving asymptotic optimality.
Distributed quantization improves classification accuracy with less data.
problem Efficiently classify features from distributed nodes with limited communication.
method Designs tailored quantization schemes for classification, proving NP-hardness and proposing polynomial-time algorithms.
result Tailored quantizers can reduce bit communication by more than a factor of two for the same accuracy.
Paper reduces communication in distributed machine learning.
problem Reduces burdensome communication in distributed machine learning.
method Introduces communication-censoring technique to reduce transmissions of variables.
result CSGD algorithm achieves same convergence rate as SGD but with significant communication reduction.
This paper improves federated learning efficiency by auto-tuning secure aggregation parameters.
problem Efficient communication in federated learning with quantized updates.
method Auto-tuning secure aggregation parameters based on random rotation properties.
result Improved communication efficiency in federated learning with secure aggregation.
DFedAvgM is a decentralized FedAvg with momentum for privacy and communication efficiency.
problem Efficiently train models with privacy and communication efficiency in federated learning.
method Decentralized Federated Averaging with Momentum (DFedAvgM) on clients connected by an undirected graph, using stochastic gradient descent with momentum and quantization.
result DFedAvgM converges under trivial assumptions and can be improved with the PŁ property, numerically verified.
In this paper, we introduce the problem of decision-oriented communications, that is, the goal of the source is to send the right amount of information in order for the intended destination to execute a task. More specifically, we restrict our attention to how the source should quantize information so that the destinat…
Compressed Federated Distillation reduces communication in federated learning.
problem Communication constraints in Federated Learning.
method Compressed Federated Distillation (CFD) leverages soft labels and quantization techniques.
result Reduces communication by more than 4 orders of magnitude compared to Federated Averaging.
Quantized Stochastic Primal-Dual Methods for Distributed Optimization
problem Distributed optimization with stochastic gradients and finite-bit communication
method q-PDGD, a quantized stochastic primal-dual method
result Linear contraction to an explicit neighborhood under RSI, O(1/k) convergence under PL inequality
It is of fundamental importance to find algorithms obtaining optimal performance for learning of statistical models in distributed and communication limited systems. Aiming at characterizing the optimal strategies, we consider learning of Gaussian Processes (GPs) in distributed systems as a pivotal example. We first ad…
Data parallelism can boost the training speed of convolutional neural networks (CNN), but could suffer from significant communication costs caused by gradient aggregation. To alleviate this problem, several scalar quantization techniques have been developed to compress the gradients. But these techniques could perform …
New algorithm reduces communication in distributed learning by sharing compressed beliefs.
problem Efficiently learning from private data in a distributed setting with large hypothesis sets.
method Proposes a belief update rule for distributed cooperative learning with compressed (sparse or quantized) beliefs.
result Beliefs converge almost surely to optimal hypotheses with a linear concentration rate.
SQuARM-SGD improves decentralized SGD efficiency with momentum.
problem Efficient decentralized training of large-scale models over networks.
method Fixed local SGD steps with Nesterov's momentum, sparsified and quantized updates, locally computed triggering criterion.
result Convergence rate matches vanilla SGD, momentum improves test performance.
Decentralized detection avoids sharing data, controls false discoveries.
problem Global false discovery rate control in decentralized novelty detection.
method Quantized surrogate models for low-precision sharing, preserving exchangeability.
result Quantized composite scores maintain competitive statistical power with reduced communication.
Distributed sensors compress and send features to a fusion center for linear regression.
problem Efficiently compress and transmit features from distributed sensors to a fusion center under varying communication constraints.
method Designs a distributed and adaptive feature compression scheme using optimal quantizers and simple adaptive strategies.
result Demonstrates improved inference performance through simulated experiments.
FedNew improves federated learning efficiency and privacy.
problem Low communication efficiency and privacy issues in Newton-type methods for federated learning.
method Introduces a two-level framework using ADMM for inverse Hessian-gradient approximation and Newton's method for global model updates, reducing communication overhead.
result FedNew achieves superior communication efficiency and privacy compared to existing methods.
Recent research implies that training and inference of deep neural networks (DNN) can be computed with low precision numerical representations of the training/test data, weights and gradients without a general loss in accuracy. The benefit of such compact representations is twofold: they allow a significant reduction o…
We formulate the notion of minimax estimation under storage or communication constraints, and prove an extension to Pinsker's theorem for nonparametric estimation over Sobolev ellipsoids. Placing limits on the number of bits used to encode any estimator, we give tight lower and upper bounds on the excess risk due to qu…
In this work, a deep learning-based method for log-likelihood ratio (LLR) lossy compression and quantization is proposed, with emphasis on a single-input single-output uncorrelated fading communication setting. A deep autoencoder network is trained to compress, quantize and reconstruct the bit log-likelihood ratios cor…
New methods improve distributed optimization on non-iid data.
problem Communication bottleneck in distributed machine learning models.
method Two types of distributed gradient compression methods (D-QSGD and D-EF-SGD) analyzed for non-iid data.
result D-EF-SGD performs better than D-QSGD on non-iid data but can still slow down with high data skewness.
Two methods reduce BN and DNN complexity, balancing size and accuracy.
problem Balancing model size and prediction accuracy in Bayesian networks and deep neural networks.
method Quantization-aware training and tree-augmented naive Bayes structure learning extension.
result Pareto optimal models found for small-scale scenarios.
Optimizes binary regression models with gradient ascent-descent methods.
problem Regression problems with binary weights in quantized learning and digital communication.
method Maximin optimization using gradient ascent-descent methods.
result The approach is optimal in linear regression with low noise and robust regression with few outliers.
Quantizers play a critical role in digital signal processing systems. Recent works have shown that the performance of quantization systems acquiring multiple analog signals using scalar analog-to-digital converters (ADCs) can be significantly improved by properly processing the analog signals prior to quantization. How…