ICE-Pruning accelerates deep neural network pruning by 9.61x.
problem Efficiently pruning deep neural networks while maintaining accuracy.
method Iterative pruning with automatic fine-tuning steps, freezing strategy, and custom learning rate scheduler.
result Significantly reduces pruning time by up to 9.61x.
Deep neural networks have dramatically achieved great success on a variety of challenging tasks. However, most successful DNNs have an extremely complex structure, leading to extensive research on model compression.As a significant area of progress in model compression, traditional gradual pruning approaches involve an…
A new framework explains why early pruning works well.
problem Understanding why early pruning of neural networks leads to good performance.
method Gradient flow framework to unify pruning measures.
result Magnitude-based pruning removes least contributing parameters, leading to faster convergence.
Pruning improves model generalization in over-parameterized models, contradicting traditional theories.
problem Pruning's effect on generalization in over-parameterized models.
method Empirical study on standard pruning algorithms and additional regularization effects.
result Pruning leads to better training and regularization, improving generalization.
This work explores the importance of model weights and Hessian bias in pruning.
problem Understanding the relative importance of model weights for efficient pruning.
method A principled exploration of pruning, focusing on linear models and neural networks.
result Asymptotic formulas reveal the performance of different pruning methods.
This paper introduces blind adversarial pruning to balance accuracy, efficiency, and robustness in neural networks.
problem Balancing accuracy, efficiency, and robustness in neural networks with limited resources.
method Adversarial pruning with a cutoff-scale strategy to dynamically adjust the strength of adversarial examples.
result Blind adversarial pruning improves the overall AER of pruned models compared to adversarial pruning.
Pruning neural network parameters is often viewed as a means to compress models, but pruning has also been motivated by the desire to prevent overfitting. This motivation is particularly relevant given the perhaps surprising observation that a wide variety of pruning approaches increase test accuracy despite sometimes …
We examine how recently documented, fundamental phenomena in deep learning models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through a set of common iterative pruning techniques, to …
A model predicts how hyperparameters affect pruning performance.
problem Predicting the impact of hyperparameters on pruning performance.
method Phenomenological model using temperature-like and load-like parameters.
result A sharp transition phenomenon in pruning performance.
Neural network for subgraph similarity computation with pruning.
problem Computing subgraph similarity between a target and query graph.
method Convert pruning to node relabeling, relax to differentiable problem, design neural network for SED computation.
result Establishes new state-of-the-art results across multiple benchmark datasets.
Pruning method removes less important features in linear models.
problem Removing less important features in linear models trained by gradient flow.
method Iterative Magnitude Pruning (IMP) applied to linear models trained by gradient flow.
result IMP prunes features with smallest projection onto the data.
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
problem Improving pruning of large language models to reduce size without sacrificing performance.
method AlphaPruning uses HT-SR theory to allocate layerwise sparsity ratios more theoretically.
result AlphaPruning prunes LLaMA-7B to 80% sparsity with reasonable perplexity.
Hyperflux models pruning as a system to reveal weight importance.
problem Pruning large neural networks to reduce latency and power consumption.
method Introduces Hyperflux, a novel L0 method that models pruning as flux and pressure. result Achieves competitive results with ResNet-50, VGG-19, and DeiT-T/S on various datasets.
This paper analyzes privacy risks in neural network pruning and proposes a defense mechanism.
problem Privacy risks in neural network pruning due to membership inference attacks.
method Investigates the impact of pruning on prediction divergence and proposes a self-attention membership inference attack.
result Proposed defense mechanism mitigates privacy risks while maintaining sparsity and accuracy.
Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weigh…
Survey on pruning CNN models to reduce size for edge devices.
problem Reducing large CNN models for edge deployment.
method Comprehensive review of pruning strategies, criteria, and techniques.
result Pruning accelerates CNN models for edge applications.
DSA efficiently allocates sparsity across layers for budgeted pruning.
problem Efficiently distributing resources (sparsity) across layers in pruning under resource constraints.
method DSA uses differentiable pruning to find continuous layer-wise pruning ratios via gradient-based optimization.
result DSA achieves superior performance and significantly reduces the time cost of pruning.
New pruning method retains model expressiveness for NLP tasks.
problem Pruning large pretrained transformer models for real-world deployment.
method Mixture Gaussian Prior Pruning (MGPP) algorithm.
result MGPP outperforms existing pruning methods in high sparsity settings.
Optimization-based pruning eliminates backpropagation for large language models.
problem Suboptimal pruning performance due to heuristic metrics.
method Optimization of Bernoulli distribution to learn pruning masks without backpropagation.
result Efficient pruning of large language models with improved performance.
Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work have been conducted on DNN model compression or pruning. However, most of the previous work took heuristic approac…
Gibbs pruning optimizes neural networks by combining physics and regularization.
problem Large neural networks are impractical for many applications.
method Combines statistical physics and stochastic regularization to train and prune networks simultaneously.
result Gibbs pruning achieves state-of-the-art performance on ResNet-56.
New insights on pruning deep networks by preserving function locality.
problem Designing effective pruning methods for deep neural networks.
method Revisited loss modeling using first and second order Taylor expansions, emphasizing locality.
result Both first and second order Taylor expansions can achieve similar performance in pruning.
PyTorch adds tools for pruning neural networks.
problem Model size and resource constraints in machine learning.
method Pruning techniques to reduce model size and capacity.
result Facilitates adoption of pruning in PyTorch.
Network pruning is a promising avenue for compressing deep neural networks. A typical approach to pruning starts by training a model and then removing redundant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done at initialization prior to training,…
Pruning + fine-tuning reduces model complexity and generalizes well for matrix sensing.
problem Reducing model complexity for matrix sensing problems.
method Group Lasso regularization and greedy pruning.
result Pruning results in a solution with minimum columns close to the ground truth.
Neural network pruning lacks standardized benchmarks and metrics.
problem Lack of standardized benchmarks and metrics in neural network pruning.
method Meta-analysis of 81 papers, controlled conditions, ShrinkBench framework.
result Neural network pruning community lacks standardized benchmarks and metrics.
The state-of-art DNN structures involve high computation and great demand for memory storage which pose intensive challenge on DNN framework resources. To mitigate the challenges, weight pruning techniques has been studied. However, high accuracy solution for extreme structured pruning that combines different types of …
BMRS offers a Bayesian approach to structured pruning of neural networks.
problem Overparameterized neural networks lead to high compute costs.
method Bayesian Model Reduction for Structured pruning (BMRS) based on two recent methods: Bayesian structured pruning with multiplicative noise and Bayesian model reduction.
result BMRS yields high compression rates and accuracy without tuning thresholds.
New pruning method for sparse additive models speeds up causal structure learning.
problem Efficiently prune spurious edges from fully-connected DAG induced by estimated topological order.
method Sparse additive models combined with randomized tree embedding and group-wise sparse regression.
result Significantly faster than existing pruning methods while maintaining comparable accuracy.
Model compression techniques on Deep Neural Network (DNN) have been widely acknowledged as an effective way to achieve acceleration on a variety of platforms, and DNN weight pruning is a straightforward and effective method. There are currently two mainstreams of pruning methods representing two extremes of pruning reg…
Pruning is an efficient model compression technique to remove redundancy in the connectivity of deep neural networks (DNNs). Computations using sparse matrices obtained by pruning parameters, however, exhibit vastly different parallelism depending on the index representation scheme. As a result, fine-grained pruning ha…
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
Holistic Filter Pruning reduces DNN complexity efficiently.
problem Redundant parameters in deep neural networks.
method Holistic Filter Pruning (HFP) for efficient DNN training.
result Achieves state-of-the-art performance with 60% reduction in multiplications on ImageNet.
Random layer-wise pruning profiles are as effective as metric-based ones for various datasets.
problem Reduction of model size and computational resources in neural networks.
method Conducted baseline experiments, developed RL-based search algorithm for finding transferable layer-wise pruning profiles.
result RL-based layer-wise pruning profiles are as good or better than best profiles found on the original dataset via exhaustive search.
PruneNet efficiently prunes channels in deep networks, improving accuracy and performance.
problem Improving deep neural network performance and efficiency through channel pruning.
method PruneNet uses a computationally light-weight optimization step to identify and prune channels based on layer redundancy.
result Pruned ResNet models achieve higher accuracy and better performance than non-pruned models.
We present a filter pruning approach for deep model compression, using a multitask network. Our approach is based on learning a a pruner network to prune a pre-trained target network. The pruner is essentially a multitask deep neural network with binary outputs that help identify the filters from each layer of the orig…
CPOT prunes deep networks by identifying redundant filters using optimal transport.
problem Redundant filters in deep neural networks make models hard to deploy on resource-limited platforms.
method CPOT uses optimal transport to find the mean of channel distributions, pruning redundant information.
result CPOT outperforms state-of-the-art methods in pruning ResNet models and image-to-image translation tasks.
Dynamic sample pruning speeds up spatio-temporal forecasting models.
problem Training deep learning models on large, redundant datasets is computationally expensive.
method Dynamic sample pruning based on real-time learning state.
result Significant acceleration of training speed with improved performance.
New compression theory justifies model pruning for neural networks.
problem Improving neural network performance with reduced model size.
method Information-theoretic rate-distortion theory applied to NN compression.
result Pruning improves model performance on CIFAR-10 and ImageNet datasets.
PAC-Net prunes deep models for better transfer learning.
problem Improving transfer learning with over-parameterized models.
method Prune, Allocate, Calibrate (PAC) approach.
result PAC-Net achieves state-of-the-art performance in inductive transfer learning.
DJPQ optimizes neural network pruning and quantization for hardware efficiency.
problem Efficiently compress neural networks for hardware inference.
method Joint gradient-based optimization of pruning and quantization into a differentiable loss function.
result Significant reduction in Bit-Operations (BOPs) with minimal accuracy loss.
A framework for privacy-preserving DNN pruning and acceleration.
problem Privacy concerns in DNN weight pruning for mobile devices.
method ADMM-based iterative pruning with synthetic data, compiler optimizations.
result 4.2X, 2.5X, and 2.0X speedup with almost no accuracy loss.
Magnitude-based pruning is one of the simplest methods for pruning neural networks. Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable performances for pruning modern architectures. Based on the observation that magnitude-based pruning indeed minimizes the Frobenius distortion of a…
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.
problem Flaws in existing pruning criteria for CNNs.
method Empirical experiments and Convolutional Weight Distribution Assumption.
result The Convolutional Weight Distribution Assumption improves filter selection in pruning.
Large language models have recently achieved state of the art performance across a wide variety of natural language tasks. Meanwhile, the size of these models and their latency have significantly increased, which makes their usage costly, and raises an interesting question: do language models need to be large? We study…
Gradient-based training and pruning for radial basis function networks in materials physics.
problem Interpretable and robust machine learning for materials physics problems.
method Gradient-based training and pruning of radial basis function networks with closed-form optimization criteria.
result Pruned models provide compact and interpretable versions of larger models, offering insights into atom-level migration processes.
Paper combines regularization and pruning to reduce FLOPs in DNNs.
problem Efficient inference in deep learning models.
method Apply mixup and cutout regularizations and soft filter pruning to the ResNet architecture.
result Combining regularization and pruning reduces FLOPs more effectively than each technique alone.