arXiv research

Flexible framework compresses models using LC algorithm.

Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quan…

Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model com…

Proposes a new linearity-based neural network compression method.

We show that model compression can improve the population risk of a pre-trained model, by studying the tradeoff between the decrease in the generalization error and the increase in the empirical risk with model compression. We first prove that model compression reduces an information-theoretic bound on the generalizati…

New compression theory justifies model pruning for neural networks.

Proposes a method to train neural networks directly on compressed text data.

DoCoFL compresses model updates for cross-device federated learning.

Auto-Compressing Subset Pruning reduces model size for faster inference.

The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main te…

Study shows LLC correlates with neural network compressibility.

Iterative AutoML improves ASR model compression by 5x without WER degradation.

One of the biggest issues in deep learning theory is the generalization ability of networks with huge model size. The classical learning theory suggests that overparameterized models cause overfitting. However, practically used large deep models avoid overfitting, which is not well explained by the classical approaches…

Develops a method for lossless compression using latent variable models.

Model-based compression is an effective, facilitating, and expanded model of neural network models with limited computing and low power. However, conventional models of compression techniques utilize crafted features [2,3,12] and explore specialized areas for exploration and design of large spaces in terms of size, spe…

A new framework compresses neural networks using sparse optimization.

Randomized matrix compression techniques, such as the Johnson-Lindenstrauss transform, have emerged as an effective and practical way for solving large-scale problems efficiently. With a focus on computational efficiency, however, forsaking solutions quality and accuracy becomes the trade-off. In this paper, we investi…

We make the following striking observation: fully convolutional VAE models trained on 32x32 ImageNet can generalize well, not just to 64x64 but also to far larger photographs, with no changes to the model. We use this property, applying fully convolutional models to lossless compression, demonstrating a method to scale…

Unified framework compresses GANs up to 47x with minimal quality loss.

LLMs compress financial texts, but distort decision-making.

Neural NCD reveals LLMs don't compress well for classification.

Deep model compression has been extensively studied, and state-of-the-art methods can now achieve high compression ratios with minimal accuracy loss. This paper studies model compression through a different lens: could we compress models without hurting their robustness to adversarial attacks, in addition to maintainin…

NeLLoC improves image compression with parallel decoding.

Model compression is eminently suited for deploying deep learning on IoT-devices. However, existing model compression techniques rely on access to the original or some alternate dataset. In this paper, we address the model compression problem when no real data is available, e.g., when data is private. To this end, we p…

Galen algorithm compresses neural networks for specific hardware with reduced latency.

Compression techniques for deep neural network models are becoming very important for the efficient execution of high-performance deep learning systems on edge-computing devices. The concept of model compression is also important for analyzing the generalization error of deep learning, known as the compression-based er…

This work compresses sequences by treating them as continuous-time processes, enabling efficient discretization.

Model compression improves dynamic forecasting ensembles while reducing computational costs.

Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size.As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This paper introduces a method to compress RNNs for resource constrained e…

The (isothermic) compressibility of lattice knots can be examined as a model of the effects of topology and geometry on the compressibility of ring polymers. In this paper, the compressibility of minimal length lattice knots in the simple cubic, face centered cubic and body centered cubic lattices are determined. Our r…

This study compresses BERT to make it suitable for low-capability devices.

LanguaShrink compresses prompts using psycholinguistic principles to reduce costs.

This paper improves compression of large NLP models using doped Kronecker Products.

In this paper we present a a deep generative model for lossy video compression. We employ a model that consists of a 3D autoencoder with a discrete latent space and an autoregressive prior used for entropy coding. Both autoencoder and prior are trained jointly to minimize a rate-distortion loss, which is closely relate…

The recent advances in deep neural networks (DNNs) make them attractive for embedded systems. However, it can take a long time for DNNs to make an inference on resource-constrained computing devices. Model compression techniques can address the computation issue of deep inference on embedded devices. This technique is …

Paper introduces adversarial lossy compression for video artifacts reduction.

Paper proposes compressive ICA algorithms for ICA model.

PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.

MoDeGPT compresses large language models without accuracy loss, saving 98% compute costs.

SHVC improves image compression with fewer parameters.

Theoretical framework for neural network compression using sparsity norms.

The paper uses information geometry to analyze model compression techniques, focusing on operator factorization.

As the industry deploys increasingly large and complex neural networks to mobile devices, more pressure is put on the memory and compute resources of those devices. Deep compression, or compression of deep neural network weight matrices, is a technique to stretch resources for such scenarios. Existing compression metho…

Unified strategy for efficient data compression and model estimation.

New video compression method outperforms traditional approaches.

Lossless compression of deep neural networks using NTK and RMT.

Proposes a link between randomness and compression in deep learning.

Paper introduces a new adaptive gradient method with gradient compression for distributed training.