Study examines effects of pruning techniques on deep learning models.
problem Understanding the impact of pruning methods on deep learning model structure and dynamics.
method Investigated differences in connectivity and learning dynamics of pruned models using various iterative pruning techniques.
result Emergence of structure in pruned models through magnitude-based unstructured pruning and weight rewinding.
Neural network pruning lacks standardized benchmarks and metrics.
problem Lack of standardized benchmarks and metrics in neural network pruning.
method Meta-analysis of 81 papers, controlled conditions, ShrinkBench framework.
result Neural network pruning community lacks standardized benchmarks and metrics.
Pruning is an efficient model compression technique to remove redundancy in the connectivity of deep neural networks (DNNs). Computations using sparse matrices obtained by pruning parameters, however, exhibit vastly different parallelism depending on the index representation scheme. As a result, fine-grained pruning ha…
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
Survey on pruning CNN models to reduce size for edge devices.
problem Reducing large CNN models for edge deployment.
method Comprehensive review of pruning strategies, criteria, and techniques.
result Pruning accelerates CNN models for edge applications.
Although deep neural networks (NNs) have achievedstate-of-the-art accuracy in many visual recognition tasks,the growing computational complexity and energy con-sumption of networks remains an issue, especially for ap-plications on platforms with limited resources and requir-ing real-time processing. Filter pruning tech…
Two retraining techniques outperform fine-tuning in neural network pruning.
problem Improving accuracy and compression in neural network pruning.
method Weight rewinding and learning rate rewinding compared to fine-tuning.
result Rewinding techniques outperform fine-tuning in accuracy and compression.
New pruning methods improve energy efficiency of neural networks.
problem Energy-efficient neural networks for devices with limited resources.
method Magnitude and Gradient based pruning at initialization and training of sparse architectures.
result Proposed novel pruning methods prevent full layer pruning and improve training.
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
Recent advances in the sparse neural network literature have made it possible to prune many large feed forward and convolutional networks with only a small quantity of data. Yet, these same techniques often falter when applied to the problem of recovering sparse recurrent networks. These failures are quantitative: when…
Bayesian inference improves neural network pruning efficiency.
problem Reducing computational and memory demands of large neural networks.
method Utilizes Bayesian inference to calculate Bayes factors for iterative pruning.
result Achieves desired levels of sparsity while maintaining competitive accuracy.
A new method prunes neural networks faster and more efficiently.
problem Reducing training time and memory usage for neural networks.
method Set-based Task-Adaptive Meta Pruning (STAMP) that meta-learns a pruning mask.
result Significantly improved compression rates and faster training speed.
Pruning is one of the most effective model reduction techniques. Deep networks require massive computation and such models need to be compressed to bring them on edge devices. Most existing pruning techniques are focused on vision-based models like convolutional networks, while text-based models are still evolving. The…
HYDRA prunes robust neural networks to improve both benign and adversarial robustness.
problem Lack of robustness against adversarial attacks and large neural network size in deep learning.
method HYDRA integrates pruning techniques with adversarial training and verifiable robust training objectives.
result HYDRA achieves compressed networks with state-of-the-art benign and robust accuracy.
Gibbs pruning optimizes neural networks by combining physics and regularization.
problem Large neural networks are impractical for many applications.
method Combines statistical physics and stochastic regularization to train and prune networks simultaneously.
result Gibbs pruning achieves state-of-the-art performance on ResNet-56.
Gradient-based training and pruning for radial basis function networks in materials physics.
problem Interpretable and robust machine learning for materials physics problems.
method Gradient-based training and pruning of radial basis function networks with closed-form optimization criteria.
result Pruned models provide compact and interpretable versions of larger models, offering insights into atom-level migration processes.
CPOT prunes deep networks by identifying redundant filters using optimal transport.
problem Redundant filters in deep neural networks make models hard to deploy on resource-limited platforms.
method CPOT uses optimal transport to find the mean of channel distributions, pruning redundant information.
result CPOT outperforms state-of-the-art methods in pruning ResNet models and image-to-image translation tasks.
The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.
problem Flaws in existing pruning criteria for CNNs.
method Empirical experiments and Convolutional Weight Distribution Assumption.
result The Convolutional Weight Distribution Assumption improves filter selection in pruning.
Structured weight pruning is a representative model compression technique of DNNs to reduce the storage and computation requirements and accelerate inference. An automatic hyperparameter determination process is necessary due to the large number of flexible hyperparameters. This work proposes AutoCompress, an automatic…
New pruning method for sparse additive models speeds up causal structure learning.
problem Efficiently prune spurious edges from fully-connected DAG induced by estimated topological order.
method Sparse additive models combined with randomized tree embedding and group-wise sparse regression.
result Significantly faster than existing pruning methods while maintaining comparable accuracy.
Compression techniques for deep neural networks are important for implementing them on small embedded devices. In particular, channel-pruning is a useful technique for realizing compact networks. However, many conventional methods require manual setting of compression ratios in each layer. It is difficult to analyze th…
Gradual pruning reduces inference cost by pruning least important channels during training.
problem Reduction of deep neural network inference cost.
method Gradual channel pruning using feature relevance scores during training.
result Achieved significant model compression with minimal accuracy loss.
Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work have been conducted on DNN model compression or pruning. However, most of the previous work took heuristic approac…
Paper combines regularization and pruning to reduce FLOPs in DNNs.
problem Efficient inference in deep learning models.
method Apply mixup and cutout regularizations and soft filter pruning to the ResNet architecture.
result Combining regularization and pruning reduces FLOPs more effectively than each technique alone.
PyTorch adds tools for pruning neural networks.
problem Model size and resource constraints in machine learning.
method Pruning techniques to reduce model size and capacity.
result Facilitates adoption of pruning in PyTorch.
PCNN prunes CNN weights efficiently for hardware acceleration.
problem Efficiently compressing CNN models for hardware acceleration.
method PCNN uses a novel Sparsity Pattern Mask (SPM) to encode and prune weights.
result PCNN achieves up to 8.4X compression with minimal accuracy loss.
This paper introduces GLT for better input data representation in BNN and proposes a compact topology with block pruning.
problem Improving input data representation for Binary Neural Networks (BNN).
method Generic Learned Thermometer (GLT) for encoding, block pruning and Knowledge Distillation for compact topology.
result Significant accuracy gains and lightweight fully-binarized models with limited accuracy degradation.
Pruning is a standard technique for removing unnecessary structure from a neural network to reduce its storage footprint, computational demands, or energy consumption. Pruning can reduce the parameter-counts of many state-of-the-art neural networks by an order of magnitude without compromising accuracy, meaning these n…
Performance-aware channel pruning improves CNN on embedded GPUs.
problem Inefficient channel pruning on embedded GPUs leads to performance slowdowns.
method Evaluate higher-level libraries that analyze input characteristics for optimized code generation.
result Performance-aware pruning can achieve significant performance speedups, up to 10x.
The state-of-art DNN structures involve high computation and great demand for memory storage which pose intensive challenge on DNN framework resources. To mitigate the challenges, weight pruning techniques has been studied. However, high accuracy solution for extreme structured pruning that combines different types of …
This paper analyzes privacy risks in neural network pruning and proposes a defense mechanism.
problem Privacy risks in neural network pruning due to membership inference attacks.
method Investigates the impact of pruning on prediction divergence and proposes a self-attention membership inference attack.
result Proposed defense mechanism mitigates privacy risks while maintaining sparsity and accuracy.
A new pruning method reduces neural network computation without retraining.
problem Efficiently reduce neural network computation while maintaining accuracy.
method Structured directional pruning via perturbation orthogonal projection.
result Achieves state-of-the-art pruned accuracy without retraining.
A new pruning method improves neural network efficiency and accuracy.
problem Optimizing neural network efficiency and accuracy through pruning.
method Formulated as a Knapsack Problem, the method optimizes trade-off between neuron importance and computational cost. Channels are pruned while maintaining network structure, and fine-tuned using inner knowledge distillation from parent network.
result State-of-the-art pruning results on ImageNet, CIFAR-10, and CIFAR-100.
Model compression techniques on Deep Neural Network (DNN) have been widely acknowledged as an effective way to achieve acceleration on a variety of platforms, and DNN weight pruning is a straightforward and effective method. There are currently two mainstreams of pruning methods representing two extremes of pruning reg…
Paper proposes a method to prune neural networks, reducing storage and computation costs.
problem Reduction of storage and computational costs for deep neural networks.
method Statistical analysis of component significance using F-statistic-based screening technique.
result Pruned models are highly competitive with state-of-the-art approaches.
A framework for privacy-preserving DNN pruning and acceleration.
problem Privacy concerns in DNN weight pruning for mobile devices.
method ADMM-based iterative pruning with synthetic data, compiler optimizations.
result 4.2X, 2.5X, and 2.0X speedup with almost no accuracy loss.
New pruning methods improve dynamic sparse training performance.
problem Improving dynamic sparse training performance.
method Design and empirical analysis of pruning criteria.
result Most pruning methods yield similar results, but magnitude-based pruning performs best in low-density regimes.
We introduce Delay Pruning, a simple yet powerful technique to regularize dynamic Boltzmann machines (DyBM). The recently introduced DyBM provides a particularly structured Boltzmann machine, as a generative model of a multi-dimensional time-series. This Boltzmann machine can have infinitely many layers of units but al…
A new method quantifies feature-map discriminativeness for efficient pruning of deep neural networks.
problem Efficiently pruning deep neural networks to reduce computation while maintaining accuracy.
method Presented a novel mathematical formulation (Discriminant Information, DI) to quantify feature-map discriminativeness, enabling efficient pruning and intra-layer mixed precision quantization.
result Our pruned ResNet50 achieves 44% FLOPs reduction without any Top-1 accuracy loss.
Dynamic sample pruning speeds up spatio-temporal forecasting models.
problem Training deep learning models on large, redundant datasets is computationally expensive.
method Dynamic sample pruning based on real-time learning state.
result Significant acceleration of training speed with improved performance.
Dirichlet pruning compresses neural networks by removing unimportant units.
problem Compressing large neural network models without sacrificing performance.
method Assigns Dirichlet distribution over network layers' units and uses variational inference to estimate parameters.
result Achieves state-of-the-art compression performance on larger architectures like VGG and ResNet.
To improve the execution speed and efficiency of neural networks in embedded systems, it is crucial to decrease the model size and computational complexity. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can reduce the amount of data communica…
Pruning improves DNNs against MIA while reducing model size and computation.
problem Vulnerability of DNNs to membership inference attacks (MIA).
method Proposes a pruning algorithm to reduce model size and computational operations.
result Pruned subnetwork prevents privacy leakage from MIA with competitive accuracy.
New flaw found in SAP defense, reducing its effectiveness to 0.1%.
problem Weakness in Stochastic Activation Pruning defense against adversarial attacks.
method Re-examined the implementation of SAP and introduced a new BPDA attack.
result SAP's effectiveness reduced to 0.1% when properly applied.
ALF reduces network parameters and operations by 70% and 61%, respectively, on embedded hardware.
problem Efficient deployment of deep learning models on resource-constrained hardware.
method Autoencoder-based low-rank filter-sharing technique.
result ALF achieves significant compression with minimal accuracy loss.
New method grows deep networks efficiently by dynamically pruning and growing layers.
problem Training deep networks is computationally expensive and inefficient.
method Structured continuous sparsification starting from a small seed architecture.
result 49.7% inference FLOPs and 47.4% training FLOPs savings with 75.2% top-1 accuracy.
Paper proposes neural network for efficient MIMO channel estimation and pilot reduction.
problem High overhead from pilot transmission in wideband MIMO systems.
method Neural network architecture for frequency-aware pilot design and channel estimation, with pruning technique.
result Neural network outperforms linear minimum mean square error (LMMSE) estimation.
This paper improves Koopman operator approximations by pruning subspaces in RKHS.
problem Improving predictive accuracy of Koopman operator approximations.
method Computes principal angles and vectors in RKHS to prune subspaces.
result Validated approach enhances Koopman operator approximations for large datasets.