Unified framework for non-compressed networks from compression bounds.
problem Generalization of large deep neural networks with huge model size.
method Unified framework converting compression based bounds to non-compressed networks.
result Unified data-dependent generalization error bound with tighter evaluation.
New taxonomy and evaluation of neural network compression methods.
problem Efficiency of deep neural networks in real-world applications.
method Categorization and evaluation of tensor factorization and probabilistic compression methods.
result SVD and probabilistic compression methods are complementary and give the best results.
Paper tackles optimal network compression for financial systems.
problem Optimal network compression for financial systems under shocks.
method Formulated as an NP-hard problem, studied systemic risk measures, and analyzed specific networks.
result Systemic fragility results no longer hold generally under shocks and heterogeneous networks.
Paper compresses neural network weight-updates for image artifacts removal.
problem Efficiently compressing neural network weight-updates for image artifacts removal.
method Fine-tuning a pre-trained artifact removal network on target data with a compression objective that encourages sparse and quantized weight-updates.
result Achieves reconstruction quality comparable to traditional codecs at comparable bitrates.
Wavelets help compress neural networks efficiently.
problem Efficiently compressing linear layers in neural networks.
method Learnable wavelet transforms to compress RNNs.
result Wavelet compressed RNNs have fewer parameters and perform competitively.
Study shows LLC correlates with neural network compressibility.
problem Evaluating limits of neural network compression.
method Extended minimum description length principle using singular learning theory.
result Complexity estimates based on LLC are linearly correlated with compressibility.
Proposes a new linearity-based neural network compression method.
problem Reduction of neural network model size while maintaining accuracy.
method Integrates linearity-based intuition with ReLU activation functions to merge layers.
result Achieves up to 75% reduction in model size without loss of accuracy.
Galen algorithm compresses neural networks for specific hardware with reduced latency.
problem Finding optimal compression policies for neural networks on specific hardware.
method Reinforcement learning using pruning and quantization to optimize inference latency.
result Compressed ResNet18 for ARM processor reduced inference latency by 80%.
Few-shot network compression improves accuracy with minimal data.
problem High estimation errors from original network during inference.
method Cross distillation, layer-wise knowledge distillation approach.
result Cross distillation significantly improves student network's accuracy with few training instances.
New study reveals how heavy-tailed SGD dynamics lead to compressible neural networks.
problem Understanding why large neural networks can be compressed effectively.
method Linking SGD dynamics to compressibility properties of neural networks.
result Large step-size/batch-size ratios and overparametrization lead to heavy-tailed SGD dynamics, making networks compressible.
Paper introduces compressibility loss for learning sparse neural network weights.
problem Learning highly compressible neural network weights.
method Applying a compressibility loss to minimize the negated sparsity of the signal.
result At critical points, weight vectors are ternary signals with a sparsity directly related to the objective value.
To improve how neural networks function it is crucial to understand their learning process. The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task. However, empirical evid…
G-CREWE efficiently aligns large networks using node embeddings and compression.
problem Efficiently aligning large networks for various applications.
method Uses node embeddings and compression to align networks at fine and coarse resolutions.
result G-CREWE achieves efficient and accurate network alignment, twice as fast as existing methods.
Theoretical framework for neural network compression using sparsity norms.
problem Understanding and quantifying compressibility and accuracy trade-offs in neural networks.
method Using sparsity-sensitive ℓ_q-norm to characterize compressibility and developing adaptive pruning algorithms.
result Theoretical relationship between network sparsity and compressibility with controlled accuracy degradation.
We compress large neural networks for quick adaptation to specific contexts.
problem How to quickly adapt a pretrained large neural network to specific contexts.
method Propose a Bayesian hypernetwork framework to compress the network and encourage sparsity.
result Generated compressed networks are significantly smaller than baseline methods.
New method compresses neural networks up to 14x with minimal performance loss.
problem Compressing neural networks for real-time applications.
method Post-training rank-selection method called Rank-Tuning.
result High compression rates with minimal performance degradation.
Layer fusion reduces deep neural network layers with minimal loss in accuracy.
problem Model compression to reduce neural network size and computation.
method Fusion of similar layers to reduce model size with minimal performance loss.
result Deep networks can be compressed up to 3.33x with minimal accuracy loss.
DP-Net uses dynamic programming for efficient deep neural network compression.
problem Efficiently compressing deep neural networks while maintaining accuracy.
method Dynamic Programming for optimal weight quantization and clustering-friendly training.
result Achieves up to 77X compression ratio on Wide ResNet with minimal accuracy loss.
A new framework compresses neural networks using sparse optimization.
problem Efficiently reducing the size of deep neural networks for practical deployment.
method Sparse optimization for model compression, tailored for stochastic learning.
result Up to 7.2 and 2.9 times FLOPs reduction with comparable accuracy.
Proposes a method to train neural networks directly on compressed text data.
problem Training neural networks on compressed text data without decompression.
method Introduces composer modules to encode symbols from grammar compression rules into vector representations.
result Demonstrates that the proposed method can achieve both memory and computational efficiency while maintaining moderate performance.
New method reduces neural image compression run-time by 50%.
problem Computational efficiency of neural image compression models.
method Automatic network optimization to reduce decoder complexity.
result Decreased decoder run-time by over 50%.
As the industry deploys increasingly large and complex neural networks to mobile devices, more pressure is put on the memory and compute resources of those devices. Deep compression, or compression of deep neural network weight matrices, is a technique to stretch resources for such scenarios. Existing compression metho…
NAC-FL optimizes model updates in FL systems by adapting compression to network congestion.
problem Federated Learning systems face congestion and delays in data exchanges.
method NAC-FL dynamically adjusts client compression based on network congestion.
result NAC-FL reduces training time and achieves robust performance improvements.
TOCO framework compresses neural networks based on tolerance analysis.
problem Deploying large neural networks on edge devices with limited resources.
method TOCO uses tolerance analysis to perform fine-grained compression, allowing flexibility to hardware changes.
result Fine-grained compression of neural networks on edge devices.
Adjoined Networks trains both base and compressed networks together for efficient model compression.
problem Efficiently compressing deep neural networks while maintaining accuracy.
method Adjoined Networks (AN) trains both a base network and a smaller compressed network simultaneously, sharing parameters.
result AN achieves 71.8% top-1 accuracy with 1.8M parameters and 1.6 GFLOPs on ImageNet.
New compression theory justifies model pruning for neural networks.
problem Improving neural network performance with reduced model size.
method Information-theoretic rate-distortion theory applied to NN compression.
result Pruning improves model performance on CIFAR-10 and ImageNet datasets.
HOTCAKE compresses CNNs by decomposing kernels into smaller parts.
problem Compressing deep CNNs without significant accuracy loss.
method Input channel decomposition, guided Tucker rank selection, higher order Tucker decomposition, fine-tuning.
result HOTCAKE produces highly compressed CNN models with good accuracy.
Modern neural networks are highly overparameterized, with capacity to substantially overfit to training data. Nevertheless, these networks often generalize well in practice. It has also been observed that trained networks can often be "compressed" to much smaller representations. The purpose of this paper is to connect…
This paper proposes an automatic neural network compression method.
problem Reducing resource requirements for deep neural networks on resource-constrained devices.
method Jointly prunes and quantizes neural networks without manual hyper-parameter tuning.
result Significant reduction in model size with minimal accuracy loss.
Flexible framework compresses models using LC algorithm.
problem Efficiently compressing neural networks for resource constraints.
method Decouples learning and compression steps with alternating L and C phases.
result Compressed models maintain performance and accuracy.
A simple method compresses neural network weights using entropy penalties.
problem Neural network weight compression for scalability and accuracy.
method Reparameterization with learned entropy penalty for compression.
result Maximized classification accuracy and model compressibility.
Lossless compression of deep neural networks using NTK and RMT.
problem Compressing large-scale deep neural networks for low-power devices.
method High-dimensional neural tangent kernel approach.
result Asymptotic spectral equivalence between NTK matrices of wide DNNs enables lossless compression.
Compression affects deep networks differently, impacting underrepresented data points.
problem Disparate impact of compression on different classes and images.
method Analysis of deep neural network pruning and quantization effects.
result Compression disproportionately impacts model performance on underrepresented data points.
This paper studies how to compress neural networks while maintaining accuracy.
problem Compressing a two-layer neural network with fewer nodes without losing accuracy.
method Using tools from high-dimensional probability, the authors minimize the L_2 loss between the target and compressed networks.
result The error rate of the approximation is shown as a function of input dimension and network size in the mean-field limit.
This paper compresses neural networks by permuting and quantizing weights.
problem Efficiently compressing large neural networks for resource-constrained platforms.
method Permuting and quantizing weights, connecting to rate-distortion theory, and using annealed quantization.
result Significant compression with minimal accuracy loss, e.g., 40-70% reduction in gap with uncompressed model.
Deep neural networks (DNNs) have become the state-of-the-art technique for machine learning tasks in various applications. However, due to their size and the computational complexity, large DNNs are not readily deployable on edge devices in real-time. To manage complexity and accelerate computation, network compression…
Distiller simplifies DNN compression research with a Python package.
problem Efficiently compressing deep neural networks.
method Open-source Python package with DNN compression algorithms.
result Facilitates new research and learning tasks in DNN compression.
Paper proposes a compression principle for neural networks using Bayesian optimization.
problem Finding methods for making generalizable predictions in machine learning.
method Compression principle and Bayesian optimization approach.
result Optimal predictive models minimize total compressed message length of data and model definition.
WNQ uses weight normalization to reduce quantization error in deep neural networks.
problem High quantization error in deep neural networks due to long-tail distribution of weights.
method Weight normalization to avoid long-tail distribution and reduce quantization error.
result WNQ achieves state-of-the-art performance on CIFAR-100 and ImageNet.
Advances understanding of neural network generalization via tensor analysis.
problem Understanding the generalizability of deep neural networks.
method Tensor analysis to measure compressibility and generalizability.
result Proposed generalization bound outperforms previous methods, especially for tensor-based networks.
This work compresses reinforcement learning models for Atari games, improving localization.
problem Expensive deep neural networks in reinforcement learning.
method Model compression, global max-pooling, Actor-Mimic, weakly supervised localization.
result Compression reduces model size to 3% of original, enabling object localization.
This paper proposes a method to automatically compress neural networks using Bayesian tensor decomposition.
problem Challenges in directly applying tensor compression in neural network training.
method Bayesian tensorized neural network with automatic rank selection.
result Produces significantly more compact neural networks (7.4x to 137x) directly from training.
Generative networks trained with sketching reduce computational burden.
problem Computational expense of training generative networks on large datasets.
method Compressive learning using a single sketch vector to approximate Maximum Mean Discrepancy.
result Cost-efficient optimization of generative networks on large datasets.
This work compresses heavy-tailed weight matrices for tighter generalization bounds.
problem Empirical evidence linking heavy-tailed weight matrices to test set accuracy but lack of formal relationship with generalization bounds.
method Utilized the compression framework to show that heavy-tailed matrices can be compressed, resulting in sparse weight matrices.
result Demonstrated a non-vacuous generalization bound for compressed networks with heavy-tailed weight matrices.
Paper explores pruning and quantisation to compress neural networks.
problem Reduces computational and memory costs of deep neural networks.
method Investigates network pruning and quantisation for AlexNet, ShuffleNet, and MobileNet.
result Pruning and quantisation compress networks to less than half their size and improve efficiency.
We propose a general framework for neural network compression that is motivated by the Minimum Description Length (MDL) principle. For that we first derive an expression for the entropy of a neural network, which measures its complexity explicitly in terms of its bit-size. Then, we formalize the problem of neural netwo…
This paper proposes a new method for efficient data compression using Bayesian neural networks.
problem Efficient compression of data represented as functions mapping coordinates to signal values.
method Overfitting variational Bayesian neural networks to the data and compressing an approximate posterior weight sample using relative entropy coding.
result Our method achieves strong performance on image and audio compression while retaining simplicity.
New method compresses LSTM networks using MPS tensor trains.
problem Challenges in maintaining performance of compressed RNNs.
method Use of MPS tensor trains for LSTM network compression.
result MPS tensor trains outperform MPOs in storage and inference time.