Flexible framework compresses models using LC algorithm.
problem Efficiently compressing neural networks for resource constraints.
method Decouples learning and compression steps with alternating L and C phases.
result Compressed models maintain performance and accuracy.
HiLLoC compresses large images losslessly using VAEs.
problem Lossless compression of large color photographs.
method Fully convolutional VAE models trained on ImageNet are applied to lossless compression.
result Achieves state-of-the-art compression for full-size ImageNet images.
Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quan…
Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model com…
Proposes a new linearity-based neural network compression method.
problem Reduction of neural network model size while maintaining accuracy.
method Integrates linearity-based intuition with ReLU activation functions to merge layers.
result Achieves up to 75% reduction in model size without loss of accuracy.
We show that model compression can improve the population risk of a pre-trained model, by studying the tradeoff between the decrease in the generalization error and the increase in the empirical risk with model compression. We first prove that model compression reduces an information-theoretic bound on the generalizati…
New compression theory justifies model pruning for neural networks.
problem Improving neural network performance with reduced model size.
method Information-theoretic rate-distortion theory applied to NN compression.
result Pruning improves model performance on CIFAR-10 and ImageNet datasets.
Compressive Transformer learns long-range sequences by compressing past memories.
problem Learning long-range sequences in language and speech models.
method Compressive Transformer compresses past memories for efficient long-range sequence learning.
result State-of-the-art performance on language and speech benchmarks.
Proposes a method to train neural networks directly on compressed text data.
problem Training neural networks on compressed text data without decompression.
method Introduces composer modules to encode symbols from grammar compression rules into vector representations.
result Demonstrates that the proposed method can achieve both memory and computational efficiency while maintaining moderate performance.
DoCoFL compresses model updates for cross-device federated learning.
problem Downlink compression for cross-device federated learning where clients may appear only once.
method Proposes DoCoFL framework for downlink compression in cross-device federated learning.
result Significant bi-directional bandwidth reduction with competitive accuracy.
Auto-Compressing Subset Pruning reduces model size for faster inference.
problem High parameter counts and slow inference times in semantic segmentation models.
method Learning a channel selection mechanism based on temperature annealing schedule.
result Significant compression of segmentation models with acceptable inference performance.
The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main te…
Study shows LLC correlates with neural network compressibility.
problem Evaluating limits of neural network compression.
method Extended minimum description length principle using singular learning theory.
result Complexity estimates based on LLC are linearly correlated with compressibility.
Iterative AutoML improves ASR model compression by 5x without WER degradation.
problem Challenges in achieving high compression levels without degrading ASR performance.
method Iterative AutoML-based Low Rank Factorization (LRF) approach.
result Achieved over 5x compression without WER degradation.
One of the biggest issues in deep learning theory is the generalization ability of networks with huge model size. The classical learning theory suggests that overparameterized models cause overfitting. However, practically used large deep models avoid overfitting, which is not well explained by the classical approaches…
Develops a method for lossless compression using latent variable models.
problem Lossless compression of large datasets.
method Bits back with asymmetric numeral systems (BB-ANS) using latent variable models.
result Achieves state-of-the-art lossless compression of full-size colour images.
Model-based compression is an effective, facilitating, and expanded model of neural network models with limited computing and low power. However, conventional models of compression techniques utilize crafted features [2,3,12] and explore specialized areas for exploration and design of large spaces in terms of size, spe…
A new framework compresses neural networks using sparse optimization.
problem Efficiently reducing the size of deep neural networks for practical deployment.
method Sparse optimization for model compression, tailored for stochastic learning.
result Up to 7.2 and 2.9 times FLOPs reduction with comparable accuracy.
Randomized matrix compression techniques, such as the Johnson-Lindenstrauss transform, have emerged as an effective and practical way for solving large-scale problems efficiently. With a focus on computational efficiency, however, forsaking solutions quality and accuracy becomes the trade-off. In this paper, we investi…
Unified framework compresses GANs up to 47x with minimal quality loss.
problem High parameter complexity of GANs for resource-constrained devices.
method Unified optimization framework combining model distillation, channel pruning, and quantization.
result 47x compression of CartoonGAN with minimal quality degradation.
LLMs compress financial texts, but distort decision-making.
problem LLMs compress financial texts, altering decision-making.
method Analyzed two diagnostic patterns: decontextualization and model dependency. Proposed Agentic Context Compression.
result LLM-compressed financial texts alter decision-making.
Neural NCD reveals LLMs don't compress well for classification.
problem The disconnect between compression and classification in neural networks.
method Developed Neural NCD to compare LLMs to classic algorithms, finding classification accuracy not correlated with compression rate.
result Classification accuracy is not predictable by compression rate alone, challenging current understanding.
Deep model compression has been extensively studied, and state-of-the-art methods can now achieve high compression ratios with minimal accuracy loss. This paper studies model compression through a different lens: could we compress models without hurting their robustness to adversarial attacks, in addition to maintainin…
NeLLoC improves image compression with parallel decoding.
problem Image compression with OOD generalization.
method Local autoregressive model with parallel decoding.
result Significant gains in compression runtime.
TOCO framework compresses neural networks based on tolerance analysis.
problem Deploying large neural networks on edge devices with limited resources.
method TOCO uses tolerance analysis to perform fine-grained compression, allowing flexibility to hardware changes.
result Fine-grained compression of neural networks on edge devices.
Model compression is eminently suited for deploying deep learning on IoT-devices. However, existing model compression techniques rely on access to the original or some alternate dataset. In this paper, we address the model compression problem when no real data is available, e.g., when data is private. To this end, we p…
Galen algorithm compresses neural networks for specific hardware with reduced latency.
problem Finding optimal compression policies for neural networks on specific hardware.
method Reinforcement learning using pruning and quantization to optimize inference latency.
result Compressed ResNet18 for ARM processor reduced inference latency by 80%.
Compression techniques for deep neural network models are becoming very important for the efficient execution of high-performance deep learning systems on edge-computing devices. The concept of model compression is also important for analyzing the generalization error of deep learning, known as the compression-based er…
This work compresses sequences by treating them as continuous-time processes, enabling efficient discretization.
problem Efficient compression of sequences, especially with deep learning models that scale with sequence length.
method Treat sequences as continuous-time processes, learn efficient discretization, and decode at different time intervals.
result Automatic bit rate reductions in video and motion capture sequences using learned discretization.
Model compression improves dynamic forecasting ensembles while reducing computational costs.
problem High computational costs and lack of transparency in dynamic forecasting ensembles.
method Model compression applied to dynamic forecasting ensembles of various types of models.
result Compressed models achieve comparable predictive performance and significant computational savings.
Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size.As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This paper introduces a method to compress RNNs for resource constrained e…
The (isothermic) compressibility of lattice knots can be examined as a model of the effects of topology and geometry on the compressibility of ring polymers. In this paper, the compressibility of minimal length lattice knots in the simple cubic, face centered cubic and body centered cubic lattices are determined. Our r…
This study compresses BERT to make it suitable for low-capability devices.
problem Large Transformer models are resource-intensive.
method Survey and analysis of model compression techniques for BERT.
result Insights into best practices for compressing large-scale Transformer models.
LanguaShrink compresses prompts using psycholinguistic principles to reduce costs.
problem Reducing computational cost and efficiency in large language model inference.
method Leverages psycholinguistic principles and the Ebbinghaus memory curve to compress prompts.
result Achieves up to 26 times compression while maintaining semantic similarity.
Proposes a new method for clustering compressed data.
problem Limited communication bandwidth and low-power consumption.
method Joint Variational Autoencoders with Bernoulli mixture models (VAB).
result The model can perform clustering in the compressed data domain.
This paper improves compression of large NLP models using doped Kronecker Products.
problem Accuracy loss when compressing large NLP tasks with Kronecker Products.
method Doping Kronecker Products with an overlay matrix to recover accuracy, and a new regularization scheme called co matrix dropout regularization (CMR).
result Compression of a large language model with LSTM layers of size 25 MB by 25x with 1.4% loss in perplexity score.
In this paper we present a a deep generative model for lossy video compression. We employ a model that consists of a 3D autoencoder with a discrete latent space and an autoregressive prior used for entropy coding. Both autoencoder and prior are trained jointly to minimize a rate-distortion loss, which is closely relate…
The recent advances in deep neural networks (DNNs) make them attractive for embedded systems. However, it can take a long time for DNNs to make an inference on resource-constrained computing devices. Model compression techniques can address the computation issue of deep inference on embedded devices. This technique is …
Paper introduces adversarial lossy compression for video artifacts reduction.
problem Unpleasant reconstruction artifacts in standard video coding schemes at low bit-rates.
method Adversarial lossy video compression model minimizing an adversarial distortion objective.
result Reduction of perceptual artifacts and detail reconstruction under extreme compression.
Paper proposes compressive ICA algorithms for ICA model.
problem Efficiently solving ICA model with reduced memory and computational complexity.
method Compressive learning approach to ICA model, proving existence of compressive ICA scheme, proposing two algorithms (IPG and ASD).
result Proposed algorithms achieve substantial memory gains over well-known ICA algorithms.
PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.
problem Communication bottleneck in decentralized deep learning models.
method Low-rank linear compressors applied on model differences using power iteration steps.
result Asymptotically independent of network and compression, faster convergence, and comparable performance to tuned compression algorithms.
MoDeGPT compresses large language models without accuracy loss, saving 98% compute costs.
problem Compression of large language models for resource-constrained devices.
method Structured compression framework using modular decomposition and matrix pair reduction.
result MoDeGPT achieves 90-95% zero-shot performance with 25-30% compression rates.
SHVC improves image compression with fewer parameters.
problem Challenges in VAE compression, especially with bits-back coding.
method Introduces autoregressive sub-pixel convolution and autoregressive initial bits.
result Achieves state-of-the-art compression performance with fewer model parameters.
Theoretical framework for neural network compression using sparsity norms.
problem Understanding and quantifying compressibility and accuracy trade-offs in neural networks.
method Using sparsity-sensitive ℓ_q-norm to characterize compressibility and developing adaptive pruning algorithms.
result Theoretical relationship between network sparsity and compressibility with controlled accuracy degradation.
The paper uses information geometry to analyze model compression techniques, focusing on operator factorization.
problem Efficiently compressing deep learning models for resource-constrained devices.
method Information geometry applied to model compression, focusing on operator factorization.
result Iterative methods are crucial for fine-tuning models, especially when compression ratios are fixed.
As the industry deploys increasingly large and complex neural networks to mobile devices, more pressure is put on the memory and compute resources of those devices. Deep compression, or compression of deep neural network weight matrices, is a technique to stretch resources for such scenarios. Existing compression metho…
Unified strategy for efficient data compression and model estimation.
problem Limited interactive exploration and data interaction in linear model development and deployment.
method Conditionally sufficient statistics for optimal data compression and estimation of linear models.
result Linear models can be estimated from compressed data without loss of parameters or covariances.
New video compression method outperforms traditional approaches.
problem Efficient video compression in low latency mode.
method Feedback Recurrent Autoencoder network architecture.
result State of the art MS-SSIM/rate performance on UVG dataset.