arXiv research

FedLog reduces communication in federated learning by sharing data summaries.

Deep learning reduces training overhead in massive MIMO systems.

When scaling distributed training, the communication overhead is often the bottleneck. In this paper, we propose a novel SGD variant with reduced communication and adaptive learning rates. We prove the convergence of the proposed algorithm for smooth but non-convex problems. Empirical results show that the proposed alg…

Federated LIDAR aided beam selection reduces mmWave beam search overhead.

Paper proposes ADC framework to reduce ViT SL training communication overhead.

Meta-learning reduces training overhead for communication systems.

SRG improves optimization efficiency with reduced memory and computation overhead.

New method reduces communication in deep learning training.

Low-precision training reduces computational cost and produces efficient models. Recent research in developing new low-precision training algorithms often relies on simulation to empirically evaluate the statistical effects of quantization while avoiding the substantial overhead of building specific hardware. To suppor…

A new method for efficient online federated learning reduces communication overhead.

Common privacy enhancing technologies fail to effectively hide certain statistical aspects of encrypted traffic, namely individual packets length, packets direction and, packets timing. Recent researches have shown that using such attributes, an adversary is able to extract various information from the encrypted traffi…

This paper optimizes AI inference on edge devices with reduced communication and computation costs.

METRO predicts reactions using minimal templates, reducing computational overhead and achieving state-of-the-art results.

The computation of convolution layers in deep neural networks typically rely on high performance routines that trade space for time by using additional memory (either for packing purposes or required as part of the algorithm) to improve performance. The problems with such an approach are two-fold. First, these routines…

Generative Adapter adapts LMs with a single forward pass, reducing inference overhead.

BayesAdapter turns pre-trained NNs into reliable BNNs with minimal overhead.

The performance and efficiency of distributed machine learning (ML) depends significantly on how long it takes for nodes to exchange state changes. Overly-aggressive attempts to reduce communication often sacrifice final model accuracy and necessitate additional ML techniques to compensate for this loss, limiting their…

Information-theoretic Bayesian optimisation techniques have demonstrated state-of-the-art performance in tackling important global optimisation problems. However, current information-theoretic approaches require many approximations in implementation, introduce often-prohibitive computational overhead and limit the choi…

Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling. However, these methods maintain second-order statistics for each parameter, thus introducing significant memory overheads that restrict the size of the model b…

Paper proposes neural network for efficient MIMO channel estimation and pilot reduction.

Modern information technology services largely depend on cloud infrastructures to provide their services. These cloud infrastructures are built on top of datacenter networks (DCNs) constructed with high-speed links, fast switching gear, and redundancy to offer better flexibility and resiliency. In this environment, net…

Paper proposes a method to estimate variance reduction in DNN training using importance sampling.

DCCNNs reduce computational overhead and ambiguity in convolutional neural networks.

Turbo-Aggregate reduces secure aggregation time from quadratic to nearly linear.

This paper deals with distributed policy optimization in reinforcement learning, which involves a central controller and a group of learners. In particular, two typical settings encountered in several applications are considered: multi-agent reinforcement learning (RL) and parallel RL, where frequent information exchan…

We use GG distributions to optimize LLMs, reducing size and training time.

One of the most significant bottleneck in training large scale machine learning models on parameter server (PS) is the communication overhead, because it needs to frequently exchange the model gradients between the workers and servers during the training iterations. Gradient quantization has been proposed as an effecti…

SIFT reduces training time by selecting samples with approximate losses.

Paper proposes efficient BNN inference flow to reduce computation and memory costs.

Automatically detects and down-weights noisy samples in machine learning training.

CBC makes CNNs robust against adversarial attacks with minimal computational overhead.

We present a sampling-free approach for computing the epistemic uncertainty of a neural network. Epistemic uncertainty is an important quantity for the deployment of deep neural networks in safety-critical applications, since it represents how much one can trust predictions on new data. Recently promising works were pr…

Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and…

HyperINF improves influence function estimation for large models with better accuracy and efficiency.

Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top-$k$ sparsification with error compensation (To…

Efficient FPGA virtualization for deep learning reduces user isolation and overhead.

Paper introduces a new adaptive gradient method with gradient compression for distributed training.

Clapping reduces memory usage in distributed optimization by reusing data samples.

Accelerates Bayesian optimization of function networks with partial evaluations.

Emerging resistive random-access memory (ReRAM) has recently been intensively investigated to accelerate the processing of deep neural networks (DNNs). Due to the in-situ computation capability, analog ReRAM crossbars yield significant throughput improvement and energy reduction compared to traditional digital methods.…

DeepCMC compresses CSI for massive MIMO systems, reducing overhead and improving performance.

Recently dictionary screening has been proposed as an effective way to improve the computational efficiency of solving the lasso problem, which is one of the most commonly used method for learning sparse representations. To address today's ever increasing large dataset, effective screening relies on a tight region boun…

EControl improves fast distributed optimization with compression and error control.

A new method for Bayesian optimization uses Voronoi tessellation candidates to reduce search time.

A new FL algorithm reduces communication overhead by selectively updating model parameters.

DGSAM improves domain generalization by minimizing individual sharpness.

A new algorithm improves Bayesian federated learning by reducing communication overhead.

Paper optimizes neural network layers to reduce energy usage without sacrificing accuracy.