Communication on heterogeneous edge networks is a fundamental bottleneck in Federated Learning (FL), restricting both model capacity and user participation. To address this issue, we introduce two novel strategies to reduce communication costs: (1) the use of lossy compression on the global model sent server-to-client;…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Communication Reduction”
We propose an efficient protocol for decentralized training of deep neural networks from distributed data sources. The proposed protocol allows to handle different phases of model training equally well and to quickly adapt to concept drifts. This leads to a reduction of communication by an order of magnitude compared t…
A new method reduces communication in Federated Learning by pulling less often.
problem Reducing communication overhead in Federated Learning.
method Pulling Reduction with Local Compensation (PRLC) for SGD.
result PRLC achieves lower pulling frequency and maintains the same convergence rate as synchronous SGD.
The present paper develops a novel aggregated gradient approach for distributed machine learning that adaptively compresses the gradient communication. The key idea is to first quantize the computed gradients, and then skip less informative quantized gradient communications by reusing outdated gradients. Quantizing and…
Paper proposes FedQ-Advantage for federated Q-learning with near-optimal regret and low communication cost.
problem Near-optimal federated Q-learning with low communication cost.
method Reference-advantage decomposition for variance reduction, synchronization between agents and server, policy update.
result Achieves almost optimal regret and near-linear regret speedup compared to single-agent learning.
Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies rem…
Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote …
Communication and privacy are two critical concerns in distributed learning. Many existing works treat these concerns separately. In this work, we argue that a natural connection exists between methods for communication reduction and privacy preservation in the context of distributed machine learning. In particular, we…
Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysiscs.LG
New algorithms improve distributed optimization under specific conditions.
problem Distributed optimization problems with high communication costs.
method SVRS and AccSVRS algorithms combining gradient sliding and variance reduction.
result Achieved better communication complexity in distributed optimization.
We introduce novel communication strategies in synchronous distributed Deep Learning consisting of decentralized gradient reduction orchestration and computational graph-aware grouping of gradient tensors. These new techniques produce an optimal overlap between computation and communication and result in near-linear sc…
There is growing interest in large-scale machine learning and optimization over decentralized networks, e.g. in the context of multi-agent learning and federated learning. Due to the imminent need to alleviate the communication burden, the investigation of communication-efficient distributed optimization algorithms - p…
New estimator for overlapping community detection in graphs.
problem Reconstructing community relations in graphs with overlapping memberships.
method Proposes a new estimator for Mixed-Membership Stochastic Block Model (MMSB) under diverging spiked eigenvalues condition.
result New estimator matches theoretical minimax lower bound on estimation error.
Reducing communication in training large-scale machine learning applications on distributed platform is still a big challenge. To address this issue, we propose a distributed hierarchical averaging stochastic gradient descent (Hier-AVG) algorithm with infrequent global reduction by introducing local reduction. As a gen…
A new algorithm improves Bayesian federated learning by reducing communication overhead.
problem Bayesian federated learning constraints, including privacy, data ownership, and communication overhead.
method Proposes Quantised Langevin Stochastic Dynamics (QLSD) for Bayesian federated learning, using gradient compression and variance reduction techniques.
result Non-asymptotic and asymptotic convergence guarantees for QLSD and its improved versions.
Paper improves distributed mean estimation and variance reduction without relying on input norm.
problem Distributed mean estimation and variance reduction with large input norms.
method Quantization and lattice theory connection for improved error bounds.
result Output error bounds depend only on input distance, not norm.
New algorithms reduce communication in GNN training.
problem Higher communication costs in GNNs due to sparse connectivity.
method Parallel algorithms for sparse-dense matrix multiplication.
result Asymptotic reduction in communication compared to previous methods.
Large-scale distributed training of neural networks is often limited by network bandwidth, wherein the communication time overwhelms the local computation time. Motivated by the success of sketching methods in sub-linear/streaming algorithms, we introduce Sketched SGD, an algorithm for carrying out distributed SGD by c…
This paper presents a new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation. Simple rules are designed to detect slowly-varying gradients and, therefore, trigger the reuse of outdated gradients. The resultant gra…
Reduces data leakage in distributed deep learning models.
problem Prevents reconstruction of sensitive raw data patterns during client communications.
method Reduces distance correlation between raw data and learned representations.
result Resilient to reconstruction attacks while maintaining model accuracy.
This paper deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multi-agent reinforcement learning (RL) and parallel RL, where frequent information exchan…
New method reduces distributed non-convex optimization rounds and bits.
problem Efficiently optimizing non-convex models in distributed systems.
method Introduces permutation compressors to reduce communication complexity.
result PermK compressors lead to significant communication complexity improvements.
VRL-SGD reduces communication complexity in non-identical data settings.
problem Training machine learning models with non-identical data distribution.
method VRL-SGD, which eliminates gradient variance dependency and achieves linear speedup with lower communication complexity.
result VRL-SGD reduces communication complexity from $O(T^{ rac{3}{4}} N^{ rac{3}{4}})$ to $O(T^{ rac{1}{2}} N^{ rac{3}{2}})$.
We present a novel active learning algorithm for community detection on networks. Our proposed algorithm uses a Maximal Expected Model Change (MEMC) criterion for querying network nodes label assignments. MEMC detects nodes that maximally change the community assignment likelihood model following a query. Our method is…
New optimization for federated learning with local models.
problem Training models with private data from multiple devices.
method Proposes a new optimization formulation and efficient SGD variants.
result Local steps can improve communication for heterogeneous data.
Due to its efficiency and ease to implement, stochastic gradient descent (SGD) has been widely used in machine learning. In particular, SGD is one of the most popular optimization methods for distributed learning. Recently, quantized SGD (QSGD), which adopts quantization to reduce the communication cost in SGD-based di…
Optimal client sampling reduces communication in federated learning.
problem Efficiently aggregate model updates from distributed clients in federated learning.
method Model weights as an Ornstein-Uhlenbeck process to estimate uncommunicated updates; optimal client sampling strategy.
result Significant reduction in communication with competitive or superior performance.
Spectral method detects communities in sparse hypergraphs, achieving detection threshold.
problem Community detection in sparse hypergraphs.
method Non-backtracking operator and spectral approach.
result Spectral method achieves detection threshold for sparse HSBMs.
MpFL models clients as strategic players to reach equilibrium with less communication.
problem Real-world clients act independently with individual objectives, not aligned with a shared global model.
method MpFL uses game-theoretic modeling and PEARL-SGD algorithm for local updates and communication.
result PEARL-SGD reaches an equilibrium with less communication than non-local updates in stochastic setup.
Bayesian approach reduces FL communication cost by one-shot.
problem High communication cost in optimization-based FL for high-dimensional models.
method Bayesian pseudocoresets and function-space inference for one-shot FL.
result Achieves prediction performance competitive to state-of-the-art with up to 2 orders of magnitude reduction in communication cost.
CSE-FSL reduces communication and storage costs in federated learning.
problem High communication and storage costs in federated learning.
method CSE-FSL uses an auxiliary network to locally update client models and sends only selected epochs' smashed data.
result Significant communication reduction with state-of-the-art convergence and model accuracy.
In this paper, we study randomized reduction methods, which reduce high-dimensional features into low-dimensional space by randomized methods (e.g., random projection, random hashing), for large-scale high-dimensional classification. Previous theoretical results on randomized reduction methods hinge on strong assumptio…
The Gradient Boosted Tree (GBT) algorithm is one of the most popular machine learning algorithms used in production, for tasks that include Click-Through Rate (CTR) prediction and learning-to-rank. To deal with the massive datasets available today, many distributed GBT methods have been proposed. However, they all assu…
Training modern deep learning models requires large amounts of computation, often provided by GPUs. Scaling computation from one GPU to many can enable much faster training and research progress but entails two complications. First, the training library must support inter-GPU communication. Depending on the particular …
Variance reduction (VR) methods boost the performance of stochastic gradient descent (SGD) by enabling the use of larger, constant stepsizes and preserving linear convergence rates. However, current variance reduced SGD methods require either high memory usage or an exact gradient computation (using the entire dataset)…
Currently, progressively larger deep neural networks are trained on ever growing data corpora. As this trend is only going to increase in the future, distributed training schemes are becoming increasingly relevant. A major issue in distributed training is the limited communication bandwidth between contributing nodes o…
p-SNE embeds Poisson count data into low dimensions preserving structure.
problem Embedding high-dimensional sparse Poisson data into a low-dimensional space.
method p-SNE (Poisson Stochastic Neighbor Embedding) using KL divergence and Hellinger distance.
result p-SNE recovers meaningful structure in real-world count datasets.
Paper reduces recommender system model size by 90%.
problem Large model size in recommender systems.
method Hybrid frequency and double hashing for model size reduction.
result 90% reduction in model size with no performance loss.
FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learningcs.LG
FedNew improves federated learning efficiency and privacy.
problem Low communication efficiency and privacy issues in Newton-type methods for federated learning.
method Introduces a two-level framework using ADMM for inverse Hessian-gradient approximation and Newton's method for global model updates, reducing communication overhead.
result FedNew achieves superior communication efficiency and privacy compared to existing methods.
Paper improves robust spectral clustering for noisy data.
problem Noisy data and heavy-tailed entries hinder traditional clustering methods.
method Robust spectral clustering with rank statistics for latent structure recovery.
result Provable recovery of latent block structure in large data matrices.
Paper proposes a DRL-based controller for networked AP systems that reduces communication frequency.
problem Reduce communication frequency in networked AP systems while maintaining control performance.
method Develops a DRL-based controller that avoids explicit update timing learning, using a semi-Markov decision process (SMDP).
result Improves communication efficiency without sacrificing control performance.
Decentralized optimization is emerging as a viable alternative for scalable distributed machine learning, but also introduces new challenges in terms of synchronization costs. To this end, several communication-reduction techniques, such as non-blocking communication, quantization, and local steps, have been explored i…
A new parallel algorithm for learning optimal policies in MDPs with low communication costs.
problem Learning optimal policies for infinite-horizon MDPs.
method Primal-Dual Stochastic Mirror Descent for convex programming problems with inexact constraints.
result First parallel algorithm for average-reward MDPs with generative model and low communication costs.
Federated learning (FL) enables on-device training over distributed networks consisting of a massive amount of modern smart devices, such as smartphones and IoT (Internet of Things) devices. However, the leading optimization algorithm in such settings, i.e., federated averaging (FedAvg), suffers from heavy communicatio…
Quantized-TinyLLaVA reduces communication costs in split learning for multimodal models.
problem High communication costs in split learning for multimodal models.
method Integrates a compression module that quantizes intermediate features into discrete representations before transmission.
result Achieves an approximate 87.5% reduction in communication overhead with 2-bit quantization.
The R package psvmSDR: A Unified Algorithm for Sufficient Dimension Reduction via Principal Machinesstat.CO
R package psvmSDR simplifies SDR computation for machine learning.
problem Sufficient dimension reduction in machine learning.
method Principal machine (PM) generalized from PSVM.
result Efficient computation of SDR estimators in various scenarios.
In this paper, a novel joint transmit power and resource allocation approach for enabling ultra-reliable low-latency communication (URLLC) in vehicular networks is proposed. The objective is to minimize the network-wide power consumption of vehicular users (VUEs) while ensuring high reliability in terms of probabilisti…
A current challenge for data management systems is to support the construction and maintenance of machine learning models over data that is large, multi-dimensional, and evolving. While systems that could support these tasks are emerging, the need to scale to distributed, streaming data requires new models and algorith…
Synthetic data replaces model updates in federated learning.
problem High computational costs in transmitting model parameters.
method Transmitting synthetic data instead of model updates.
result Reduces communication costs by more than an order of magnitude.