arXiv research

Turbo-Aggregate reduces secure aggregation time from quadratic to nearly linear.

Paper connects MoE and self-attention, proposing active-attention.

CWGD measures gradient diversity weighted by curvature, improving SGD convergence.

FedLog reduces communication in federated learning by sharing data summaries.

A basic question in the theory of fault-tolerant quantum computation is to understand the fundamental resource costs for performing a universal logical set of gates on encoded qubits to arbitrary accuracy. Here we consider qubits encoded with constant space overhead (i.e. finite encoding rate) in the limit of arbitrari…

The augmented Lagrangian (AL) method that solves convex optimization problems with linear constraints has drawn more attention recently in imaging applications due to its decomposable structure for composite cost functions and empirical fast convergence rate under weak conditions. However, for problems such as X-ray co…

Deep learning reduces training overhead in massive MIMO systems.

We consider the problem of learning a binary classifier from $n$ different data sources, among which at most an $η$ fraction are adversarial. The overhead is defined as the ratio between the sample complexity of learning in this setting and that of learning the same hypothesis class on a single data distribution. We pr…

We study the collaborative PAC learning problem recently proposed in Blum et al.~\cite{BHPQ17}, in which we have $k$ players and they want to learn a target function collaboratively, such that the learned function approximates the target function well on all players' distributions simultaneously. The quality of the col…

New screening rules speed up optimal design calculations.

In data summarization we want to choose $k$ prototypes in order to summarize a data set. We study a setting where the data set comprises several demographic groups and we are restricted to choose $k_i$ prototypes belonging to group $i$. A common approach to the problem without the fairness constraint is to optimize a c…

SRG improves optimization efficiency with reduced memory and computation overhead.

Paper introduces FoMoH for optimization without backpropagation.

Federated LIDAR aided beam selection reduces mmWave beam search overhead.

When scaling distributed training, the communication overhead is often the bottleneck. In this paper, we propose a novel SGD variant with reduced communication and adaptive learning rates. We prove the convergence of the proposed algorithm for smooth but non-convex problems. Empirical results show that the proposed alg…

Two main obstacles preventing the widespread adoption of variational Bayesian neural networks are the high parameter overhead that makes them infeasible on large networks, and the difficulty of implementation, which can be thought of as "programming overhead." MC dropout [Gal and Ghahramani, 2016] is popular because it…

We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning. Inspired by expected sarsa, EPG integrates (or sums) across actions when estimating the gradient, instead of relying only on the action in the sampled trajectory…

First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the prohibitive computational cost in calculating the second-order information. In this pa…

The computation of convolution layers in deep neural networks typically rely on high performance routines that trade space for time by using additional memory (either for packing purposes or required as part of the algorithm) to improve performance. The problems with such an approach are two-fold. First, these routines…

Efficient FPGA virtualization for deep learning reduces user isolation and overhead.

Global Memory Augmentation (GMAT) improves Transformer performance on long documents.

Paper proposes ADC framework to reduce ViT SL training communication overhead.

SOTERIA optimizes neural networks for secure inference with minimal overhead.

ISAAC Newton uses input-based curvature for efficient training.

Improved stability and generalization for blackbox learned optimizers.

HCBM improves deep learning explainability by non-linear concept aggregation.

Generative Adapter adapts LMs with a single forward pass, reducing inference overhead.

Paper uses ML to predict insulator flashover risk.

TorchGAN is a PyTorch based framework for writing succinct and comprehensible code for training and evaluation of Generative Adversarial Networks. The framework's modular design allows effortless customization of the model architecture, loss functions, training paradigms, and evaluation metrics. The key features of Tor…

Information-theoretic Bayesian optimisation techniques have demonstrated state-of-the-art performance in tackling important global optimisation problems. However, current information-theoretic approaches require many approximations in implementation, introduce often-prohibitive computational overhead and limit the choi…

We show LLMs can be locally linear, enabling better control of activations.

Common privacy enhancing technologies fail to effectively hide certain statistical aspects of encrypted traffic, namely individual packets length, packets direction and, packets timing. Recent researches have shown that using such attributes, an adversary is able to extract various information from the encrypted traffi…

Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling. However, these methods maintain second-order statistics for each parameter, thus introducing significant memory overheads that restrict the size of the model b…

Paper introduces a new adaptive gradient method with gradient compression for distributed training.

Low-precision training reduces computational cost and produces efficient models. Recent research in developing new low-precision training algorithms often relies on simulation to empirically evaluate the statistical effects of quantization while avoiding the substantial overhead of building specific hardware. To suppor…

A major obstacle for widespread deployment of frequency division duplex (FDD)-based Massive multiple-input multiple-output (MIMO) communications is the large signaling overhead for reporting full downlink (DL) channel state information (CSI) back to the basestation (BS), in order to enable closed-loop precoding. We com…

Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any p…

Ensuring differential privacy of models learned from sensitive user data is an important goal that has been studied extensively in recent years. It is now known that for some basic learning problems, especially those involving high-dimensional data, producing an accurate private model requires much more data than learn…

Paper introduces FDM for efficient training of Neural SDEs.

We construct a finitely presented group $G$ with non-quadratic Dehn function $f$ majorizable by a quadratic function on arbitrary long intervals.

Paper proposes a pre-conditioning method to speed up gradient descent in multi-agent optimization.

The paper classifies biharmonic quadratic maps between spheres, proving their energy density properties.

Compact Recurrent Transformer (CRT) improves Transformer efficiency for long sequences.

Recently, deep learning has achieved huge successes in many important applications. In our previous studies, we proposed quadratic/second-order neurons and deep quadratic neural networks. In a quadratic neuron, the inner product of a vector of data and the corresponding weights in a conventional neuron is replaced with…

A new FL algorithm reduces communication overhead by selectively updating model parameters.

We present a sampling-free approach for computing the epistemic uncertainty of a neural network. Epistemic uncertainty is an important quantity for the deployment of deep neural networks in safety-critical applications, since it represents how much one can trust predictions on new data. Recently promising works were pr…

The study examines how quadratic inequalities affect distances in length spaces.

Paper proposes neural network for efficient MIMO channel estimation and pilot reduction.