Distributed gradient descent (DGD) is an efficient way of implementing gradient descent (GD), especially for large data sets, by dividing the computation tasks into smaller subtasks and assigning to different computing servers (CSs) to be executed in parallel. In standard parallel execution, per-iteration waiting time …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “straggling servers”
Hybrid approach reduces computation time and decoding complexity.
problem Straggling servers in distributed computing.
method Coded partial gradient computation (CPGC) that balances gradient accuracy and completion time.
result Reduces both computation time and decoding complexity.
Gradient codes use block designs to resist adversarial stragglers in distributed computing.
problem Mitigating slow machines (stragglers) in distributed gradient-based methods.
method Gradient coding based on balanced incomplete block designs (BIBDs) to resist adversarial selection of stragglers.
result Adversarial stragglers have no advantage over random selection, and codes based on symmetric BIBDs maximize the adversarial threshold.
Coded Federated Learning speeds up model convergence by preemptively computing on parity data.
problem Federated learning's convergence is slow on heterogeneous platforms due to stragglers.
method Develops CFL scheme where clients generate parity data and share it once, allowing the server to compute redundantly.
result CFL allows global model to converge nearly four times faster than uncoded federated learning.
CodedFedL speeds up federated learning in MEC networks by 15x.
problem Slow convergence in federated learning due to heterogeneity and stochastic fluctuations.
method Injects structured coding redundancy into federated learning to mitigate stragglers and speed up training.
result CodedFedL speeds up the training procedure by up to 15x compared to benchmark schemes.
This paper considers the problem of implementing large-scale gradient descent algorithms in a distributed computing setting in the presence of {\em straggling} processors. To mitigate the effect of the stragglers, it has been previously proposed to encode the data with an erasure-correcting code and decode at the maste…
Federated learning supports exact support recovery with minimal communication.
problem Learning the exact support of sparse linear regression in federated learning.
method One-shot communication algorithm for exact support recovery without optimization.
result Polynomial sample complexity and logarithmic number of clients required.
Asynchronous SGD can speed up training with a trade-off of gradient staleness.
problem Asynchronous SGD suffers from gradient staleness, affecting convergence error.
method Theoretical analysis of the error-runtime trade-off considering random straggling delays.
result A method of gradually varying synchronicity in distributed SGD is proposed and demonstrated.
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcs.LG
AdaComm optimizes SGD by dynamically adjusting communication frequency for faster convergence.
problem Achieving optimal error-runtime trade-off in distributed SGD.
method Adaptive communication strategy that starts with infrequent averaging to save delay and improve speed, then increases frequency.
result AdaComm reduces training time by 3x while maintaining the same final loss.
Lapse improves parameter servers by dynamically allocating parameters, achieving near-linear scaling.
problem Efficiently managing distributed training with reduced communication overhead.
method Integrate dynamic parameter allocation into parameter servers, proposing Lapse.
result Lapse provides near-linear scaling and can be orders of magnitude faster than existing parameter servers.
A new scheme for private computation splits client data into shares for server operations.
problem Private computation between client and server without revealing data.
method Stochastic scheme splitting client data into privatized shares.
result Server performs operations on privatized shares without learning raw data.
New algorithm for federated learning without a central server.
problem Federated learning in unidirectional trust social networks.
method Central Server Free Federated Learning (OPS) method.
result Users can benefit from communication with trusted users in federated learning.
Algorithm stabilizes queues in asymmetric systems with unknown service rates.
problem Stabilizing queues in multi-class multi-server systems with unknown service rates.
method Proposes UCB and Thompson Sampling algorithms to stabilize queues while learning service rates.
result Achieves system stability with an average queue length bound of \(O(\min\{N,K\}/ε)\) for large time horizon \(T\).
Corella protects client data privacy in multi-server learning with correlated queries.
problem Protecting client data privacy in multi-server machine learning.
method Proposes a private multi-server learning approach using correlated queries and strong noise.
result Mitigates client data leakage with high accuracy and minimal computational effort.
New gradient coding schemes reduce decoding error in both random and adversarial straggler settings.
problem Creating efficient approximate gradient coding schemes for distributed optimization.
method Introduced novel approximate gradient codes based on expander graphs, achieving optimal decoding coefficients.
result Achieved nearly optimal error in random setting and nearly half the error in adversarial setting compared to existing codes.
LiuBei is a resilient ML algorithm that tolerates Byzantine workers and servers without trusting any component.
problem Byzantine failures in distributed ML solutions.
method Byzantine-resilient ML algorithm that aggregates gradients and replicates parameter servers, using a filtering mechanism and scatter/gather protocol.
result LiuBei achieves Byzantine resilience to both servers and workers and guarantees convergence, with an accuracy loss of around 5% and a 24% convergence overhead.
New method speeds up distributed linear regression.
problem Efficiently solve distributed linear regression problems.
method Iteratively Pre-conditioned Stochastic Gradient Descent (IPSG)
result Converges linearly in expectation to the solution.
Paper explores differential privacy in high-dimensional federated learning, tackling server trustworthiness and estimation.
problem Maintaining privacy in distributed environments with high-dimensional data.
method Investigates scenarios with untrusted and trusted central servers, introduces novel federated estimation algorithms for linear regression models.
result Tight minimax rates depend on high-dimensionality even with sparsity assumptions, and novel algorithms handle slight variations among distributed models.
A smart device optimizes server selection for energy and latency in dynamic networks.
problem Optimizing server selection for mobile devices in edge computing with uncertainty and dynamic changes.
method Formulated as a budget-limited multi-armed bandit problem, a policy is proposed to minimize regret.
result The proposed method outperforms existing solutions in terms of energy and latency.
New method speeds up nuclear-norm constrained learning over multiple machines.
problem Synchronization slowdown and high communication costs in large-scale learning.
method Asynchronous Stochastic Frank-Wolfe (SFW-asyn) method.
result SFW-asyn achieves the same convergence rate as vanilla SFW but with speed-ups almost linear to the number of machines.
New algorithm solves distributed ML with no trusted central server.
problem Tackles Byzantine failures in distributed machine learning.
method ByzSGD algorithm using Scatter/Gather, DMC, and MDA.
result Solves general Byzantine-resilient distributed ML problem.
This paper makes two contributions to Bayesian machine learning algorithms. Firstly, we propose stochastic natural gradient expectation propagation (SNEP), a novel alternative to expectation propagation (EP), a popular variational inference algorithm. SNEP is a black box variational algorithm, in that it does not requi…
CSE-FSL reduces communication and storage costs in federated learning.
problem High communication and storage costs in federated learning.
method CSE-FSL uses an auxiliary network to locally update client models and sends only selected epochs' smashed data.
result Significant communication reduction with state-of-the-art convergence and model accuracy.
Enhanced federated learning reduces communication costs and improves model accuracy.
problem Reducing communication costs in federated learning.
method Asynchronous model update and temporally weighted aggregation.
result The proposed algorithm outperforms baseline in terms of communication cost and model accuracy.
New method reduces FL communication cost by leveraging server's side information.
problem High communication cost in federated learning.
method Exploits server's side information to compress client updates.
result Up to 82 times smaller bitrate with higher accuracy.
Detects backdoors in outsourced models by replicating training steps across multiple servers.
problem Detecting backdoors in models trained on cloud providers without prior knowledge.
method Replicate training steps across multiple servers to identify deviations and malicious updates.
result 99.6% accuracy in identifying backdoored models out of 50% malicious providers.
Stanza separates convolutional and fully connected layers for faster deep learning training.
problem Heavy data transfer between workers and servers in distributed deep learning.
method Layer separation: most nodes train convolutional layers, others train fully connected layers only.
result Significant acceleration of training time (1.34x--13.9x) over current systems.
This paper improves privacy in federated learning without a trusted server.
problem Privacy in federated learning with silos that distrust each other.
method Introduces Inter-Silo Record-Level Differential Privacy (ISRL-DP) and accelerated algorithms for convex and smooth losses.
result Achieves optimal privacy and accuracy tradeoffs in federated learning.
BrainTorrent uses peer-to-peer FL for medical image segmentation without a central server.
problem Lack of sufficient annotated data for personalized medical models.
method Peer-to-peer federated learning framework without a central server.
result BrainTorrent outperforms traditional server-based FL and achieves similar performance to pooled data training.
Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learningcs.LG
This work provides bounds on generalization error and privacy leakage in federated learning.
problem Bounding generalization error and privacy leakage in federated learning.
method Information-theoretic framework for classical, distributed, and federated learning.
result Upper and lower bounds on generalization error and privacy leakage.
Paper tackles federated linear bandit learning with AirComp for noisy channels.
problem Minimize cumulative regret in federated linear bandit learning.
method Proposes a federated linear bandits scheme using over-the-air computation (AirComp) over noisy fading channels.
result Determines the regret bound of the proposed scheme.
Client-based machine learning uses mobile devices for computation, improving privacy and reducing data upload.
problem Exploiting mobile devices for machine learning tasks to protect privacy and reduce data upload.
method Leveraging local hardware and data on mobile devices for computation-intensive tasks, only uploading results.
result Client-based machine learning can relieve server burdens and protect user privacy.
Paper proposes a GPU-based system for training massive deep learning models in ads systems.
problem Training massive deep learning models with terabyte-scale parameters in ads systems.
method Hierarchical GPU parameter server with 3-layer storage (GPU High-Bandwidth Memory, CPU main memory, SSD).
result 4-node hierarchical GPU parameter server trains a model 2X faster than a 150-node in-memory system.
DFedAvgM is a decentralized FedAvg with momentum for privacy and communication efficiency.
problem Efficiently train models with privacy and communication efficiency in federated learning.
method Decentralized Federated Averaging with Momentum (DFedAvgM) on clients connected by an undirected graph, using stochastic gradient descent with momentum and quantization.
result DFedAvgM converges under trivial assumptions and can be improved with the PŁ property, numerically verified.
Paper improves communication in distributed optimization, reducing worker-to-server data exchanges.
problem Efficiency in server-to-worker communication in distributed optimization.
method MARINA-P, a novel downlink compression method using correlated compressors; M3, combining MARINA-P with uplink compression.
result MARINA-P achieves provably superior server-to-worker communication complexity with increasing number of workers.
We propose three new robust aggregation rules for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilie…
New strategies reduce FL's impact on client resources, enabling larger models and more users.
problem Communication bottleneck in Federated Learning on heterogeneous edge networks.
method Lossy compression and Federated Dropout to reduce client-to-server communication and local computation.
result Up to 14x reduction in server-to-client communication, 1.7x reduction in local computation, and 28x reduction in upload communication.
We propose a novel robust aggregation rule for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilience…
Byzantine-resilient federated learning with local iterations and robust mean estimation.
problem Byzantine clients disrupt federated learning with local iterations.
method Local SGD iterations, robust mean estimation, and matrix concentration result.
result Convergence analysis for strongly-convex and non-convex smooth objectives in heterogeneous data settings.
In this work, we propose a novel framework for privacy-preserving client-distributed machine learning. It is motivated by the desire to achieve differential privacy guarantees in the local model of privacy in a way that satisfies all systems constraints using asynchronous client-server communication and provides attrac…
This work improves federated learning privacy and accuracy with non-private data sharing and approximate gradient coding.
problem Challenges of non-IID data and stragglers in federated learning.
method Data-driven strategy combining offline data sharing and approximate gradient coding.
result Achieves a trade-off between privacy and utility, leading to improved model convergence and accuracy.
Hybrid-FL improves ML model accuracy in non-IID data environments.
problem Performance degradation in FL due to non-IID data.
method Hybrid-FL combines client and server learning, selecting optimal clients and data for aggregation.
result 13.5% higher classification accuracy than previous methods.
Mix2FLD improves FL accuracy with FD, reducing convergence time.
problem Uplink-downlink capacity asymmetry in federated learning.
method Two-way mixup of local samples and model parameters, preserving privacy.
result Achieves up to 16.7% higher test accuracy with reduced convergence time.
Paper studies IPG method's robustness in noisy distributed linear regression.
problem Distributed linear regression with noise.
method Iteratively Pre-conditioned Gradient-descent (IPG) method.
result IPG method's robustness against noise compared favorably to state-of-the-art algorithms.
Paper develops a federated learning method to protect privacy without sacrificing model utility.
problem Privacy leakage in federated learning due to information exchange between edge devices and server.
method Combines local gradient perturbation, secure aggregation, and zCDP for privacy protection.
result Demonstrates superior trade-off between privacy and model utility through extensive experiments.
Develops a new deep learning framework for privacy-preserving text representations.
problem Privacy concerns in deep learning frameworks requiring data pooling to a trusted server.
method Three modules: embedding, randomization, and classifier. Novel LDP protocol reduces privacy impact on accuracy.
result Framework delivers comparable or better performance than non-private and existing LDP protocols.
In AI research and industry, machine learning is the most widely used tool. One of the most important machine learning algorithms is Gradient Boosting Decision Tree, i.e. GBDT whose training process needs considerable computational resources and time. To shorten GBDT training time, many works tried to apply GBDT on Par…
Slow running or straggler tasks can significantly reduce computation speed in distributed computation. Recently, coding-theory-inspired approaches have been applied to mitigate the effect of straggling, through embedding redundancy in certain linear computational steps of the optimization algorithm, thus completing the…