arXiv research

Unified sign-based compression for federated learning with faster convergence.

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD does not converge to the optimum. Further, even when it does conver…

We develop a communication-efficient distributed learning algorithm that is robust against Byzantine worker machines. We propose and analyze a distributed gradient-descent algorithm that performs a simple thresholding based on gradient norms to mitigate Byzantine failures. We show the (statistical) error-rate of our al…

Unified signSGD and gradient descent analysis for neural networks.

Two algorithms improve federated learning efficiency and resilience.

New analysis of signSGD with random reshuffling shows faster convergence rates.

Is AdamW effective under heavy-tailed noise?

Enhances LLM quantization with MDBF, improving perplexity and accuracy.

Flexible framework compresses models using LC algorithm.

Reduces multiclass and regression compression schemes to binary ones.

Proposes a link between randomness and compression in deep learning.

Paper introduces a new adaptive gradient method with gradient compression for distributed training.

Galen algorithm compresses neural networks for specific hardware with reduced latency.

The (isothermic) compressibility of lattice knots can be examined as a model of the effects of topology and geometry on the compressibility of ring polymers. In this paper, the compressibility of minimal length lattice knots in the simple cubic, face centered cubic and body centered cubic lattices are determined. Our r…

HiLLoC compresses large images losslessly using VAEs.

Neural NCD reveals LLMs don't compress well for classification.

BDC compresses both sample size and dimensionality of large datasets.

DoCoFL compresses model updates for cross-device federated learning.

Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quan…

New taxonomy and evaluation of neural network compression methods.

Proposes a method to train neural networks directly on compressed text data.

Study shows LLC correlates with neural network compressibility.

Paper proposes SCALLION and SCAFCOM for compressed FL with reduced communication.

New uncertainty principle limits compression in distributed learning, suggesting optimal methods.

One of the biggest issues in deep learning theory is the generalization ability of networks with huge model size. The classical learning theory suggests that overparameterized models cause overfitting. However, practically used large deep models avoid overfitting, which is not well explained by the classical approaches…

Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size.As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This paper introduces a method to compress RNNs for resource constrained e…

Proposes a new linearity-based neural network compression method.

Randomized matrix compression techniques, such as the Johnson-Lindenstrauss transform, have emerged as an effective and practical way for solving large-scale problems efficiently. With a focus on computational efficiency, however, forsaking solutions quality and accuracy becomes the trade-off. In this paper, we investi…

Can textual data be compressed intelligently without losing accuracy in evaluating sentiment? In this study, we propose a novel evolutionary compression algorithm, PARSEC (PARts-of-Speech for sEntiment Compression), which makes use of Parts-of-Speech tags to compress text in a way that sacrifices minimal classification…

Unified framework compresses GANs up to 47x with minimal quality loss.

Proposes Stochastic-Sign SGD for federated learning with theoretical guarantees.

New compression theory justifies model pruning for neural networks.

SHVC improves image compression with fewer parameters.

Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model com…

Compressive learning is a framework where (so far unsupervised) learning tasks use not the entire dataset but a compressed summary (sketch) of it. We propose a compressive learning classification method, and a novel sketch function for images.

COIN++ compresses multiple data types efficiently.

Paper introduces adversarial lossy compression for video artifacts reduction.

LLMs compress financial texts, but distort decision-making.

A new method, REC, compresses images by encoding their latent representations efficiently.

Wavelets help compress neural networks efficiently.

This work compresses sequences by treating them as continuous-time processes, enabling efficient discretization.

Auto-Compressing Subset Pruning reduces model size for faster inference.

DP-Net uses dynamic programming for efficient deep neural network compression.

LEAD algorithm speeds up decentralized optimization with compression.

We show that model compression can improve the population risk of a pre-trained model, by studying the tradeoff between the decrease in the generalization error and the increase in the empirical risk with model compression. We first prove that model compression reduces an information-theoretic bound on the generalizati…

A new framework compresses neural networks using sparse optimization.

A new method compresses conditional distributions of labelled data.

In this paper, we present a novel approach for fine-tuning a decoder-side neural network in the context of image compression, such that the weight-updates are better compressible. At encoder side, we fine-tune a pre-trained artifact removal network on target data by using a compression objective applied on the weight-u…