Communication on heterogeneous edge networks is a fundamental bottleneck in Federated Learning (FL), restricting both model capacity and user participation. To address this issue, we introduce two novel strategies to reduce communication costs: (1) the use of lossy compression on the global model sent server-to-client;…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,982 papers · 148 categories
Trend · papers per month
48 results for “communication reduction”
We propose an efficient protocol for decentralized training of deep neural networks from distributed data sources. The proposed protocol allows to handle different phases of model training equally well and to quickly adapt to concept drifts. This leads to a reduction of communication by an order of magnitude compared t…
A new method reduces communication in Federated Learning by pulling less often.
problem Reducing communication overhead in Federated Learning.
method Pulling Reduction with Local Compensation (PRLC) for SGD.
result PRLC achieves lower pulling frequency and maintains the same convergence rate as synchronous SGD.
A new method reduces communication in distributed learning by 32x.
problem High communication costs in distributed machine learning.
method Distributed compressed SGD with Nesterov's momentum and blockwise compression.
result Achieves 32x reduction in communication cost and converges as fast as full-precision SGD.
This paper develops efficient decentralized optimization methods for large-scale machine learning.
problem Efficient distributed optimization over networks with reduced communication.
method Develops communication-efficient approximate Newton-type methods for decentralized optimization.
result Establishes linear convergence for various optimization problems.
A new method reduces communication in distributed learning by skipping less informative gradient updates.
problem Efficient communication in distributed machine learning.
method Quantizes and skips less informative gradients to reduce communication overhead.
result Proves linear convergence rate similar to gradient descent with significant communication savings.
Paper proposes FedQ-Advantage for federated Q-learning with near-optimal regret and low communication cost.
problem Near-optimal federated Q-learning with low communication cost.
method Reference-advantage decomposition for variance reduction, synchronization between agents and server, policy update.
result Achieves almost optimal regret and near-linear regret speedup compared to single-agent learning.
Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies rem…
Efficient deep learning on exascale supercomputers solves materials imaging inverse problems.
problem Solving scientific inverse problems in materials imaging using deep learning.
method Novel communication strategies in synchronous distributed deep learning, including decentralized gradient reduction and computational graph-aware grouping.
result Achieved near-linear scaling of distributed training up to 27,600 GPUs on Summit, reaching 2.15(4) EFLOPS.
DiffSketch combines privacy and communication efficiency in distributed learning.
problem Privacy and communication efficiency in distributed machine learning.
method DiffSketch uses Count Sketch for data stream summarization to achieve both privacy and efficiency.
result DiffSketch provides strong differential privacy guarantees and significant communication compression.
Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysiscs.LG
New algorithms improve distributed optimization under specific conditions.
problem Distributed optimization problems with high communication costs.
method SVRS and AccSVRS algorithms combining gradient sliding and variance reduction.
result Achieved better communication complexity in distributed optimization.
New estimator for overlapping community detection in graphs.
problem Reconstructing community relations in graphs with overlapping memberships.
method Proposes a new estimator for Mixed-Membership Stochastic Block Model (MMSB) under diverging spiked eigenvalues condition.
result New estimator matches theoretical minimax lower bound on estimation error.
Reducing communication in training large-scale machine learning applications on distributed platform is still a big challenge. To address this issue, we propose a distributed hierarchical averaging stochastic gradient descent (Hier-AVG) algorithm with infrequent global reduction by introducing local reduction. As a gen…
A new algorithm improves Bayesian federated learning by reducing communication overhead.
problem Bayesian federated learning constraints, including privacy, data ownership, and communication overhead.
method Proposes Quantised Langevin Stochastic Dynamics (QLSD) for Bayesian federated learning, using gradient compression and variance reduction techniques.
result Non-asymptotic and asymptotic convergence guarantees for QLSD and its improved versions.
New approach reduces communication in distributed optimization.
problem Reduces communication in distributed optimization.
method Each node performs multiple local optimization steps before communication, even for an infinite number of steps.
result Communication can be significantly reduced without sacrificing convergence.
Paper improves distributed mean estimation and variance reduction without relying on input norm.
problem Distributed mean estimation and variance reduction with large input norms.
method Quantization and lattice theory connection for improved error bounds.
result Output error bounds depend only on input distance, not norm.
New algorithms reduce communication in GNN training.
problem Higher communication costs in GNNs due to sparse connectivity.
method Parallel algorithms for sparse-dense matrix multiplication.
result Asymptotic reduction in communication compared to previous methods.
Large-scale distributed training of neural networks is often limited by network bandwidth, wherein the communication time overwhelms the local computation time. Motivated by the success of sketching methods in sub-linear/streaming algorithms, we introduce Sketched SGD, an algorithm for carrying out distributed SGD by c…
This paper presents a new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation. Simple rules are designed to detect slowly-varying gradients and, therefore, trigger the reuse of outdated gradients. The resultant gra…
Reduces data leakage in distributed deep learning models.
problem Prevents reconstruction of sensitive raw data patterns during client communications.
method Reduces distance correlation between raw data and learned representations.
result Resilient to reconstruction attacks while maintaining model accuracy.
This paper deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multi-agent reinforcement learning (RL) and parallel RL, where frequent information exchan…
New method reduces distributed non-convex optimization rounds and bits.
problem Efficiently optimizing non-convex models in distributed systems.
method Introduces permutation compressors to reduce communication complexity.
result PermK compressors lead to significant communication complexity improvements.
VRL-SGD reduces communication complexity in non-identical data settings.
problem Training machine learning models with non-identical data distribution.
method VRL-SGD, which eliminates gradient variance dependency and achieves linear speedup with lower communication complexity.
result VRL-SGD reduces communication complexity from $O(T^{ rac{3}{4}} N^{ rac{3}{4}})$ to $O(T^{ rac{1}{2}} N^{ rac{3}{2}})$.
We present a novel active learning algorithm for community detection on networks. Our proposed algorithm uses a Maximal Expected Model Change (MEMC) criterion for querying network nodes label assignments. MEMC detects nodes that maximally change the community assignment likelihood model following a query. Our method is…
New optimization for federated learning with local models.
problem Training models with private data from multiple devices.
method Proposes a new optimization formulation and efficient SGD variants.
result Local steps can improve communication for heterogeneous data.
Due to its efficiency and ease to implement, stochastic gradient descent (SGD) has been widely used in machine learning. In particular, SGD is one of the most popular optimization methods for distributed learning. Recently, quantized SGD (QSGD), which adopts quantization to reduce the communication cost in SGD-based di…
Optimal client sampling reduces communication in federated learning.
problem Efficiently aggregate model updates from distributed clients in federated learning.
method Model weights as an Ornstein-Uhlenbeck process to estimate uncommunicated updates; optimal client sampling strategy.
result Significant reduction in communication with competitive or superior performance.
Spectral method detects communities in sparse hypergraphs, achieving detection threshold.
problem Community detection in sparse hypergraphs.
method Non-backtracking operator and spectral approach.
result Spectral method achieves detection threshold for sparse HSBMs.
SCAFFOLD improves Federated Learning by reducing client-drift and speeding up convergence.
problem Federated Learning's performance is limited by client-drift in heterogeneous data.
method SCAFFOLD uses control variates to correct client-drift and improve convergence.
result SCAFFOLD requires fewer communication rounds and is not affected by data heterogeneity.
MpFL models clients as strategic players to reach equilibrium with less communication.
problem Real-world clients act independently with individual objectives, not aligned with a shared global model.
method MpFL uses game-theoretic modeling and PEARL-SGD algorithm for local updates and communication.
result PEARL-SGD reaches an equilibrium with less communication than non-local updates in stochastic setup.
Bayesian approach reduces FL communication cost by one-shot.
problem High communication cost in optimization-based FL for high-dimensional models.
method Bayesian pseudocoresets and function-space inference for one-shot FL.
result Achieves prediction performance competitive to state-of-the-art with up to 2 orders of magnitude reduction in communication cost.
CSE-FSL reduces communication and storage costs in federated learning.
problem High communication and storage costs in federated learning.
method CSE-FSL uses an auxiliary network to locally update client models and sends only selected epochs' smashed data.
result Significant communication reduction with state-of-the-art convergence and model accuracy.
In this paper, we study randomized reduction methods, which reduce high-dimensional features into low-dimensional space by randomized methods (e.g., random projection, random hashing), for large-scale high-dimensional classification. Previous theoretical results on randomized reduction methods hinge on strong assumptio…
Proposes a new GBT algorithm for scalable high-dimensional data.
problem Scalability issues with existing distributed GBT methods.
method Block-distributed GBTs, using Quickscorer for efficient communication.
result Multiple orders of magnitude reduction in communication cost for sparse data.
Training modern deep learning models requires large amounts of computation, often provided by GPUs. Scaling computation from one GPU to many can enable much faster training and research progress but entails two complications. First, the training library must support inter-GPU communication. Depending on the particular …
Variance reduction (VR) methods boost the performance of stochastic gradient descent (SGD) by enabling the use of larger, constant stepsizes and preserving linear convergence rates. However, current variance reduced SGD methods require either high memory usage or an exact gradient computation (using the entire dataset)…
Two new methods reduce communication costs in federated learning.
problem Heavy communication costs in federated learning, especially with non-IID data.
method FedMMD uses MMD constraint for two-stream model training; FedFusion aggregates local and global features.
result FedMMD and FedFusion reduce communication costs by 20% and 60% respectively.
Currently, progressively larger deep neural networks are trained on ever growing data corpora. As this trend is only going to increase in the future, distributed training schemes are becoming increasingly relevant. A major issue in distributed training is the limited communication bandwidth between contributing nodes o…
p-SNE embeds Poisson count data into low dimensions preserving structure.
problem Embedding high-dimensional sparse Poisson data into a low-dimensional space.
method p-SNE (Poisson Stochastic Neighbor Embedding) using KL divergence and Hellinger distance.
result p-SNE recovers meaningful structure in real-world count datasets.
FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learningcs.LG
FedNew improves federated learning efficiency and privacy.
problem Low communication efficiency and privacy issues in Newton-type methods for federated learning.
method Introduces a two-level framework using ADMM for inverse Hessian-gradient approximation and Newton's method for global model updates, reducing communication overhead.
result FedNew achieves superior communication efficiency and privacy compared to existing methods.
Paper reduces recommender system model size by 90%.
problem Large model size in recommender systems.
method Hybrid frequency and double hashing for model size reduction.
result 90% reduction in model size with no performance loss.
Paper improves robust spectral clustering for noisy data.
problem Noisy data and heavy-tailed entries hinder traditional clustering methods.
method Robust spectral clustering with rank statistics for latent structure recovery.
result Provable recovery of latent block structure in large data matrices.
Paper proposes a DRL-based controller for networked AP systems that reduces communication frequency.
problem Reduce communication frequency in networked AP systems while maintaining control performance.
method Develops a DRL-based controller that avoids explicit update timing learning, using a semi-Markov decision process (SMDP).
result Improves communication efficiency without sacrificing control performance.
A new parallel algorithm for learning optimal policies in MDPs with low communication costs.
problem Learning optimal policies for infinite-horizon MDPs.
method Primal-Dual Stochastic Mirror Descent for convex programming problems with inexact constraints.
result First parallel algorithm for average-reward MDPs with generative model and low communication costs.
Quantized-TinyLLaVA reduces communication costs in split learning for multimodal models.
problem High communication costs in split learning for multimodal models.
method Integrates a compression module that quantizes intermediate features into discrete representations before transmission.
result Achieves an approximate 87.5% reduction in communication overhead with 2-bit quantization.
The R package psvmSDR: A Unified Algorithm for Sufficient Dimension Reduction via Principal Machinesstat.CO
R package psvmSDR simplifies SDR computation for machine learning.
problem Sufficient dimension reduction in machine learning.
method Principal machine (PM) generalized from PSVM.
result Efficient computation of SDR estimators in various scenarios.
In this paper, a novel joint transmit power and resource allocation approach for enabling ultra-reliable low-latency communication (URLLC) in vehicular networks is proposed. The objective is to minimize the network-wide power consumption of vehicular users (VUEs) while ensuring high reliability in terms of probabilisti…
A current challenge for data management systems is to support the construction and maintenance of machine learning models over data that is large, multi-dimensional, and evolving. While systems that could support these tasks are emerging, the need to scale to distributed, streaming data requires new models and algorith…