This article improves communication efficiency in distributed ML over wireless networks.
problem Achieving high ML inference accuracy at scale with zero communication latency.
method Optimizing communication payload types, techniques, scheduling, and ML architectures.
result Communication-efficient and distributed learning frameworks are presented.
Efficient decentralized learning framework reduces communication costs.
problem Efficiently solve optimization problems in distributed learning networks.
method Censored and Quantized Generalized GADMM (CQ-GGADMM) framework.
result Achieves linear convergence rate under strong convexity assumptions.
DP-REC combines privacy and communication efficiency in federated learning.
problem Combining privacy and communication efficiency in federated learning.
method DP-REC uses Relative Entropy Coding (REC) for compression and a minor modification for differential privacy.
result DP-REC reduces communication costs while maintaining privacy comparable to state-of-the-art methods.
Distributed stochastic gradient descent is an important subroutine in distributed learning. A setting of particular interest is when the clients are mobile devices, where two important concerns are communication efficiency and the privacy of the clients. Several recent works have focused on reducing the communication c…
Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies rem…
Efficient algorithm reduces communication costs in sparse regression.
problem Sparse linear regression with massive data.
method CESDAR algorithm, communication-efficient surrogate likelihood.
result Achieves same statistical accuracy as global estimator with reduced communication.
This paper improves hierarchical community detection efficiency using local structural properties.
problem Efficiency of hierarchical community detection methods in large networks.
method Use of local structural network properties as proxies to improve efficiency.
result Achieves competitive results in modularity with improved efficiency.
New algorithms reduce communication costs in collaborative learning.
problem Reducing communication costs in collaborative learning.
method Distributed boosting and adaptation to classification noise.
result Communication-efficient algorithms for collaborative PAC learning robust to noise.
Flexible framework improves communication efficiency across various systems.
problem Reducing communication between nodes in machine learning tasks.
method Adapts compression level to true gradient at each iteration, optimizing per-bit improvement.
result Automatic tuning strategies significantly increase communication efficiency.
Paper studies fundamental limits of communication in distributed learning.
problem Communication efficiency in model aggregation for distributed learning.
method Rate-Distortion approach to model aggregation as a vector Gaussian CEO problem.
result Derives rate region bound and sum-rate-distortion function for model aggregation.
We present a Communication-efficient Surrogate Likelihood (CSL) framework for solving distributed statistical inference problems. CSL provides a communication-efficient surrogate to the global likelihood that can be used for low-dimensional estimation, high-dimensional regularized estimation and Bayesian inference. For…
Efficient method for vertex embedding and community detection.
problem Vertex embedding and community detection.
method Normalized one-hot graph encoder and rank-based cluster size measure.
result Excellent numerical performance of graph encoder ensemble algorithm.
Paper proposes AggITD for efficient federated hypergradient computation.
problem Computing hypergradient in federated settings is challenging due to distributed and nonlinear construction of global Hessian matrices.
method AggITD: a novel communication-efficient federated hypergradient estimator via aggregated iterative differentiation.
result AggITD achieves the same sample complexity as AID-based approaches but with fewer communication rounds, especially in heterogeneous data environments.
DESTRESS optimizes decentralized nonconvex optimization with optimal IFO complexity and efficient communication.
problem Decentralized nonconvex finite-sum optimization in multi-agent systems.
method DESTRESS uses stochastic recursive gradient updates, gradient tracking, and careful hyper-parameter choices to achieve optimal IFO complexity with efficient communication.
result DESTRESS matches the optimal IFO complexity of centralized algorithms while maintaining communication efficiency.
Compressed Federated Distillation reduces communication in federated learning.
problem Communication constraints in Federated Learning.
method Compressed Federated Distillation (CFD) leverages soft labels and quantization techniques.
result Reduces communication by more than 4 orders of magnitude compared to Federated Averaging.
The paper examines efficient algorithms for linear regression over resource-limited networks.
problem Efficient communication in distributed learning over resource-limited networks.
method Developed algorithms for communication-efficient learning of linear regression tasks.
result The algorithms enable a tradeoff between communication and learning with theoretical performance guarantees.
This paper tackles energy-efficient machine learning on low-power devices.
problem Energy consumption in machine learning due to data communication.
method Dynamic averaging for integer exponential families on low-power processors.
result Achieves comparable model quality with significantly less communication and energy.
Scaff-PD improves fairness and robustness in federated learning with reduced communication.
problem Improving fairness and robustness in federated learning with limited communication.
method Scaff-PD uses a family of distributionally robust objectives and an accelerated primal dual algorithm with bias-corrected steps.
result Scaff-PD achieves significant gains in communication efficiency and convergence speed while maintaining fairness and robustness.
D2P-Fed improves privacy and communication in federated learning.
problem Achieving both differential privacy and communication efficiency in federated learning.
method Applying discrete Gaussian noise to private data transmission.
result D2P-Fed outperforms state-of-the-art by 4.7% to 13.0% in model accuracy with one-third less communication cost.
A new gradient quantization scheme improves communication efficiency in distributed training.
problem Efficiently compressing gradients for parallel training of large models.
method Proposes a new gradient quantization scheme with theoretical guarantees and empirical performance.
result The new scheme matches and exceeds the performance of existing methods.
Paper develops efficient algorithms for robust distributed learning with statistical guarantees.
problem Limited communication power and adversarial node behaviors in distributed learning.
method Surrogate likelihood framework and median/trimmed mean operations.
result Provable robustness against Byzantine failures and optimal statistical rates.
New method improves FL efficiency by shuffling data, balancing privacy and accuracy.
problem Balancing privacy, communication, and accuracy in federated learning.
method Developed communication-efficient schemes for private mean estimation, combining privacy amplification and shuffled data.
result Achieved same privacy, optimization performance with lower communication cost.
Recently, the technique of local updates is a powerful tool in centralized settings to improve communication efficiency via periodical communication. For decentralized settings, it is still unclear how to efficiently combine local updates and decentralized communication. In this work, we propose an algorithm named as L…
Paper proposes DCT for efficient hybrid parallel training of large recommendation models.
problem Training large recommendation models at scale with efficient communication.
method Dynamic Communication Thresholding (DCT) for both Data Parallelism and Model Parallelism.
result Reduces communication by 100x and 20x during DP and MP, respectively, improving training time by 37%.
Efficiently addresses federated learning challenges with reduced communication and sample complexity.
problem Heterogeneity in data volumes and distributions at different clients compromises model generalization ability.
method Introduces algorithms for communication-efficient Federated Group Distributionally Robust Optimization (FGDRO).
result Communication complexity reduced to O(1/ε4) for FGDRO-CVaR and O(1/ε3) for FGDRO-KL. A new method for efficient online federated learning reduces communication overhead.
problem Real-world limitations in online federated learning, such as heterogeneous client participation and communication delays.
method Proposes a communication-efficient asynchronous online federated learning (PAO-Fed) strategy.
result Achieves the same convergence properties as online federated stochastic gradient while reducing communication overhead by 98 percent.
AdaQuantFL reduces communication in federated learning by adaptively quantizing model updates.
problem Efficient communication of model updates in federated learning with high-dimensional models and limited bandwidth.
method AdaQuantFL uses adaptive quantization to reduce the number of bits for model updates while maintaining low error floor.
result AdaQuantFL converges in fewer communicated bits compared to fixed quantization levels, with minimal impact on accuracy.
We consider the problem of learning from distributed data in the agnostic setting, i.e., in the presence of arbitrary forms of noise. Our main contribution is a general distributed boosting-based procedure for learning an arbitrary concept space, that is simultaneously noise tolerant, communication efficient, and compu…
This work explores the large-scale multi-agent communication mechanism under a multi-agent reinforcement learning (MARL) setting. We summarize the general categories of topology for communication structures in MARL literature, which are often manually specified. Then we propose a novel framework termed as Learning Stru…
New algorithm reduces communication costs in distributed deep learning.
problem High communication costs in distributed deep learning.
method Sparse-SignSGD with Majority Vote (S3GD-MV).
result Significantly reduces communication costs while maintaining accuracy.
This paper tackles efficient federated learning for generalized linear bandits.
problem Limited communication efficiency restricts existing federated learning solutions to linear models.
method Proposes a communication-efficient solution framework using online and offline regression.
result Proves sub-linear regret and communication cost for generalized linear bandits.
A new FL algorithm reduces communication overhead by selectively updating model parameters.
problem Data heterogeneity and communication overhead in federated learning.
method Uses age of information metric to selectively update model parameters and group clients with similar data.
result Our method can expedite training and surpass other communication-efficient strategies in efficiency.
Identifying overlapping communities in networks is a challenging task. In this work we present a novel approach to community detection that utilises the Bayesian non-negative matrix factorisation (NMF) model to produce a probabilistic output for node memberships. The scheme has the advantage of computational efficiency…
Recently, reducing communication time between machines becomes the main focus of distributed data mining. Previous methods propose to make workers do more computation locally before aggregating local solutions in the server such that fewer communication rounds between server and workers are required. However, these met…
Paper proposes an adaptive gradient method for federated learning.
problem Communication efficiency in federated learning.
method Adaptive gradient method for federated learning.
result Guaranteed convergence and communication efficiency in federated learning.
FLIX simplifies federated learning with efficient communication.
problem Handling constraints specific to federated learning.
method Introduces FLIX, a new framework for federated learning that addresses communication and personalization challenges.
result FLIX achieves dissimilarity regularization similar to local methods without requiring local steps.
DFedAvgM is a decentralized FedAvg with momentum for privacy and communication efficiency.
problem Efficiently train models with privacy and communication efficiency in federated learning.
method Decentralized Federated Averaging with Momentum (DFedAvgM) on clients connected by an undirected graph, using stochastic gradient descent with momentum and quantization.
result DFedAvgM converges under trivial assumptions and can be improved with the PŁ property, numerically verified.
Developed efficient distributed logistic regression for large datasets.
problem Communication inefficiency and sparsity issues in distributed training of large-scale models.
method Iterative local optimization of a surrogate likelihood to improve initial solutions, handling sparsity and diverging updates.
result Learned a communication-efficient distributed logistic regression model for millions of features.
Optimal client sampling reduces communication in federated learning.
problem Efficiently aggregate model updates from distributed clients in federated learning.
method Model weights as an Ornstein-Uhlenbeck process to estimate uncommunicated updates; optimal client sampling strategy.
result Significant reduction in communication with competitive or superior performance.
New method reduces communication costs in distributed nonconvex optimization.
problem Large communication costs between central server and local workers in distributed learning.
method Communication-compressed AMSGrad for distributed nonconvex optimization.
result Converges to first-order stationary point with same iteration complexity as vanilla AMSGrad.
The paper analyzes distributed Bayesian inference and its Frequentist guarantees.
problem Analyzing large decentralized datasets with distributed Bayesian inference.
method Establishes Frequentist properties for distributed (non-)Bayesian inference.
result Distributed Bayesian inference retains parametric efficiency and enhances robustness.
Distributed Lion optimizes large model training by reducing communication costs.
problem Training large AI models efficiently with reduced communication costs.
method Adapted Lion optimizer for distributed training, using binary or lower-precision vectors for communication.
result Distributed Lion achieves comparable performance to standard optimizers but with significantly reduced communication bandwidth.
A communication-efficient method controls FDR in network settings.
problem Controlling FDR in networks with limited communication.
method Sample-and-Forward: a flexible procedure for multihop networks.
result Nodes can control FDR without sharing p-values, achieving power and FDR control.
This paper considers a distributed reinforcement learning problem in which a network of multiple agents aim to cooperatively maximize the globally averaged return through communication with only local neighbors. A randomized communication-efficient multi-agent actor-critic algorithm is proposed for possibly unidirectio…
We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of communication efficiency. We consider various practical scenarios of distributed learning setup and …
LotteryFL improves federated learning efficiency and personalization.
problem Statistical heterogeneity and communication efficiency in federated learning.
method Exploiting Lottery Ticket hypothesis to learn compact lottery networks.
result LotteryFL achieves better personalization and reduces communication cost.
CyBeR-0 optimizes federated learning with Byzantine resilience and reduced communication costs.
problem Byzantine attacks and communication inefficiency in federated learning.
method Transformed robust aggregation for zero-order optimization under client heterogeneity.
result CyBeR-0 achieves stable performance with minimal communication costs and reduced memory usage.
In distributed systems, communication is a major concern due to issues such as its vulnerability or efficiency. In this paper, we are interested in estimating sparse inverse covariance matrices when samples are distributed into different machines. We address communication efficiency by proposing a method where, in a si…