Lapse improves parameter servers by dynamically allocating parameters, achieving near-linear scaling.
problem Efficiently managing distributed training with reduced communication overhead.
method Integrate dynamic parameter allocation into parameter servers, proposing Lapse.
result Lapse provides near-linear scaling and can be orders of magnitude faster than existing parameter servers.
Paper proposes a GPU-based system for training massive deep learning models in ads systems.
problem Training massive deep learning models with terabyte-scale parameters in ads systems.
method Hierarchical GPU parameter server with 3-layer storage (GPU High-Bandwidth Memory, CPU main memory, SSD).
result 4-node hierarchical GPU parameter server trains a model 2X faster than a 150-node in-memory system.
Stanza separates convolutional and fully connected layers for faster deep learning training.
problem Heavy data transfer between workers and servers in distributed deep learning.
method Layer separation: most nodes train convolutional layers, others train fully connected layers only.
result Significant acceleration of training time (1.34x--13.9x) over current systems.
This work provides bounds on generalization error and privacy leakage in federated learning.
problem Bounding generalization error and privacy leakage in federated learning.
method Information-theoretic framework for classical, distributed, and federated learning.
result Upper and lower bounds on generalization error and privacy leakage.
New algorithm solves distributed ML with no trusted central server.
problem Tackles Byzantine failures in distributed machine learning.
method ByzSGD algorithm using Scatter/Gather, DMC, and MDA.
result Solves general Byzantine-resilient distributed ML problem.
LiuBei is a resilient ML algorithm that tolerates Byzantine workers and servers without trusting any component.
problem Byzantine failures in distributed ML solutions.
method Byzantine-resilient ML algorithm that aggregates gradients and replicates parameter servers, using a filtering mechanism and scatter/gather protocol.
result LiuBei achieves Byzantine resilience to both servers and workers and guarantees convergence, with an accuracy loss of around 5% and a 24% convergence overhead.
DFedAvgM is a decentralized FedAvg with momentum for privacy and communication efficiency.
problem Efficiently train models with privacy and communication efficiency in federated learning.
method Decentralized Federated Averaging with Momentum (DFedAvgM) on clients connected by an undirected graph, using stochastic gradient descent with momentum and quantization.
result DFedAvgM converges under trivial assumptions and can be improved with the PŁ property, numerically verified.
In AI research and industry, machine learning is the most widely used tool. One of the most important machine learning algorithms is Gradient Boosting Decision Tree, i.e. GBDT whose training process needs considerable computational resources and time. To shorten GBDT training time, many works tried to apply GBDT on Par…
We propose three new robust aggregation rules for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilie…
Paper explores differential privacy in high-dimensional federated learning, tackling server trustworthiness and estimation.
problem Maintaining privacy in distributed environments with high-dimensional data.
method Investigates scenarios with untrusted and trusted central servers, introduces novel federated estimation algorithms for linear regression models.
result Tight minimax rates depend on high-dimensionality even with sparsity assumptions, and novel algorithms handle slight variations among distributed models.
We propose a novel robust aggregation rule for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilience…
Mix2FLD improves FL accuracy with FD, reducing convergence time.
problem Uplink-downlink capacity asymmetry in federated learning.
method Two-way mixup of local samples and model parameters, preserving privacy.
result Achieves up to 16.7% higher test accuracy with reduced convergence time.
This paper presents an asynchronous incremental aggregated gradient algorithm and its implementation in a parameter server framework for solving regularized optimization problems. The algorithm can handle both general convex (possibly non-smooth) regularizers and general convex constraints. When the empirical data loss…
A novel decentralized deep learning algorithm using gradient-based optimization.
problem Decentralized deep learning in networked systems without a central server.
method Heavy-ball acceleration method and consensus protocol for model and gradient-momentum sharing.
result The proposed algorithm outperforms competing methods in various communication topologies.
Corella protects client data privacy in multi-server learning with correlated queries.
problem Protecting client data privacy in multi-server machine learning.
method Proposes a private multi-server learning approach using correlated queries and strong noise.
result Mitigates client data leakage with high accuracy and minimal computational effort.
This work improves communication efficiency in federated learning over wireless networks by optimizing energy consumption.
problem Optimizing energy consumption in federated learning over wireless networks.
method Adopting SignSGD for gradient sign exchange, considering channel capacity with outage, and proposing a stochastic sign-based algorithm for uneven data distribution.
result Proposed methods achieve a balance between learning performance and energy consumption.
Coded Federated Learning speeds up model convergence by preemptively computing on parity data.
problem Federated learning's convergence is slow on heterogeneous platforms due to stragglers.
method Develops CFL scheme where clients generate parity data and share it once, allowing the server to compute redundantly.
result CFL allows global model to converge nearly four times faster than uncoded federated learning.
This paper makes two contributions to Bayesian machine learning algorithms. Firstly, we propose stochastic natural gradient expectation propagation (SNEP), a novel alternative to expectation propagation (EP), a popular variational inference algorithm. SNEP is a black box variational algorithm, in that it does not requi…
Enhanced federated learning reduces communication costs and improves model accuracy.
problem Reducing communication costs in federated learning.
method Asynchronous model update and temporally weighted aggregation.
result The proposed algorithm outperforms baseline in terms of communication cost and model accuracy.
More frequent model updates in FL increase generalization error.
problem Negative impact of frequent communication on FL model generalization.
method Analyzed the effect of the number of rounds of model aggregation on generalization error.
result Generalization error increases with more frequent model updates.
Most commonly used distributed machine learning systems are either synchronous or centralized asynchronous. Synchronous algorithms like AllReduce-SGD perform poorly in a heterogeneous environment, while asynchronous algorithms using a parameter server suffer from 1) communication bottleneck at parameter servers when wo…
Machine learning with big data often involves large optimization models. For distributed optimization over a cluster of machines, frequent communication and synchronization of all model parameters (optimization variables) can be very costly. A promising solution is to use parameter servers to store different subsets of…
Develops a new deep learning framework for privacy-preserving text representations.
problem Privacy concerns in deep learning frameworks requiring data pooling to a trusted server.
method Three modules: embedding, randomization, and classifier. Novel LDP protocol reduces privacy impact on accuracy.
result Framework delivers comparable or better performance than non-private and existing LDP protocols.
Deep learning (DL) training-as-a-service (TaaS) is an important emerging industrial workload. The unique challenge of TaaS is that it must satisfy a wide range of customers who have no experience and resources to tune DL hyper-parameters, and meticulous tuning for each user's dataset is prohibitively expensive. Therefo…
FURL improves model accuracy in FL by locally training user embeddings.
problem Improving prediction accuracy of neural-network-based models in Federated Learning.
method FURL divides model parameters into federated and private parameters, training private parameters locally.
result Significant performance improvement with 8% and 51% increases on two datasets.
Paper develops a robust federated recommendation system against poisoning attacks.
problem Low-cost poisoning attacks degrade federated recommendation systems' performance.
method Develops a robust learning strategy using gradients to filter out Byzantine clients.
result Empirically validated robust learning strategy on four datasets.
DBCL defends collaborative learning by sketching parameters to prevent gradient-based privacy inference attacks.
problem Privacy leaks in collaborative machine learning due to gradient-based attacks.
method Random matrix sketching applied to parameters, followed by re-generation of sketching after each iteration.
result DBCL prevents effective gradient-based privacy inference attacks without significant computational or accuracy costs.
Federated learning is viewed as a hierarchical latent variable model for new algorithm development.
problem Training models privately across multiple clients while maintaining privacy and efficiency.
method Viewing federated learning as a hierarchical latent variable model and applying Expectation-Maximization (EM) algorithm.
result Proposes FedSparse, a federated learning algorithm that promotes sparsity and reduces communication and inference costs.
Hippo optimizes deep learning hyper-parameters by reducing redundant trials.
problem Redundant hyper-parameter trials in hyper-parameter optimization.
method Hippo breaks down hyper-parameter sequences into stages and executes them in a tree structure.
result Hippo reduces GPU-hours and training time significantly compared to existing methods.
A new federated learning algorithm improves on existing methods by exploiting data smoothness.
problem Federated learning optimization with smooth loss functions.
method Federated Low Rank Gradient Descent (FedLRGD) algorithm.
result FedLRGD outperforms Federated Averaging (FedAve) in federated oracle complexity under certain conditions.
A new scheme for private computation splits client data into shares for server operations.
problem Private computation between client and server without revealing data.
method Stochastic scheme splitting client data into privatized shares.
result Server performs operations on privatized shares without learning raw data.
DSSP improves deep learning training speed by dynamically adjusting staleness thresholds.
problem Time-consuming deep learning training on large datasets.
method Dynamic Stale Synchronous Parallel (DSSP) framework that adapts staleness threshold at runtime.
result DSSP converges faster and achieves higher accuracy than other paradigms.
Pipe-SGD improves deep net training time by 5.4x with pipelined AllReduce.
problem Efficiently training deep neural networks on distributed systems.
method Pipelined SGD with two worker nodes, considering network latency and bandwidth.
result Up to 5.4x improvement in wall-clock training time.
FedPAQ improves federated learning efficiency by averaging and quantizing updates.
problem Communication bottlenecks and scalability issues in federated learning.
method Periodic averaging, partial device participation, and quantized message-passing.
result FedPAQ achieves near-optimal theoretical guarantees and demonstrates communication-computation tradeoffs.
Asynchronous distributed stochastic gradient descent methods have trouble converging because of stale gradients. A gradient update sent to a parameter server by a client is stale if the parameters used to calculate that gradient have since been updated on the server. Approaches have been proposed to circumvent this pro…
In distributed ML applications, shared parameters are usually replicated among computing nodes to minimize network overhead. Therefore, proper consistency model must be carefully chosen to ensure algorithm's correctness and provide high throughput. Existing consistency models used in general-purpose databases and moder…
New algorithm reduces communication costs in distributed deep learning.
problem High communication costs in distributed deep learning.
method Sparse-SignSGD with Majority Vote (S3GD-MV).
result Significantly reduces communication costs while maintaining accuracy.
New algorithm for federated learning without a central server.
problem Federated learning in unidirectional trust social networks.
method Central Server Free Federated Learning (OPS) method.
result Users can benefit from communication with trusted users in federated learning.
As Machine Learning (ML) applications increase in data size and model complexity, practitioners turn to distributed clusters to satisfy the increased computational and memory demands. Unfortunately, effective use of clusters for ML requires considerable expertise in writing distributed code, while highly-abstracted fra…
Distributed gradient descent (DGD) is an efficient way of implementing gradient descent (GD), especially for large data sets, by dividing the computation tasks into smaller subtasks and assigning to different computing servers (CSs) to be executed in parallel. In standard parallel execution, per-iteration waiting time …
DaringFed incentivizes clients in OFL with dynamic rewards under TII.
problem Designing incentives for OFL clients under dynamic, incomplete information.
method Formulated as a dynamic signaling and pricing allocation problem in a Bayesian persuasion game.
result Optimal design of DaringFed improves accuracy and convergence speed by 16.99%.
Distributed model training is vulnerable to byzantine system failures and adversarial compute nodes, i.e., nodes that use malicious updates to corrupt the global model stored at a parameter server (PS). To guarantee some form of robustness, recent work suggests using variants of the geometric median as an aggregation r…
Algorithm stabilizes queues in asymmetric systems with unknown service rates.
problem Stabilizing queues in multi-class multi-server systems with unknown service rates.
method Proposes UCB and Thompson Sampling algorithms to stabilize queues while learning service rates.
result Achieves system stability with an average queue length bound of \(O(\min\{N,K\}/ε)\) for large time horizon \(T\).
One of the most significant bottleneck in training large scale machine learning models on parameter server (PS) is the communication overhead, because it needs to frequently exchange the model gradients between the workers and servers during the training iterations. Gradient quantization has been proposed as an effecti…
We propose a real-time context-aware learning system along with the architecture that runs on the mobile devices, provide services to the user and manage the IoT devices. In this system, an application running on mobile devices collected data from the sensors, learned about the user-defined context, made predictions in…
We consider the problem of distributed statistical machine learning in adversarial settings, where some unknown and time-varying subset of working machines may be compromised and behave arbitrarily to prevent an accurate model from being learned. This setting captures the potential adversarial attacks faced by Federate…
Federated learning improves by training central model with client model outputs.
problem Direct averaging of client models is limited in FL.
method Ensemble distillation for model fusion.
result Central model trained faster with fewer communication rounds.
New method for semi-supervised learning in federated learning with and without labels at clients.
problem Training federated learning models with partially or completely unlabeled data.
method Federated Matching (FedMatch) with inter-client consistency loss and disjoint learning.
result FedMatch outperforms local semi-supervised learning and naive federated learning combinations.