Lookahead pruning extends single-layer optimization to multi-layer, outperforming magnitude-based pruning.
problem Pruning neural networks to reduce computational cost and memory usage.
method Developed a multi-layer optimization approach extending the single-layer optimization of magnitude-based pruning.
result Consistently outperforms magnitude-based pruning on various networks, especially in high sparsity.
DSA efficiently allocates sparsity across layers for budgeted pruning.
problem Efficiently distributing resources (sparsity) across layers in pruning under resource constraints.
method DSA uses differentiable pruning to find continuous layer-wise pruning ratios via gradient-based optimization.
result DSA achieves superior performance and significantly reduces the time cost of pruning.
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
PruneNet efficiently prunes channels in deep networks, improving accuracy and performance.
problem Improving deep neural network performance and efficiency through channel pruning.
method PruneNet uses a computationally light-weight optimization step to identify and prune channels based on layer redundancy.
result Pruned ResNet models achieve higher accuracy and better performance than non-pruned models.
Random layer-wise pruning profiles are as effective as metric-based ones for various datasets.
problem Reduction of model size and computational resources in neural networks.
method Conducted baseline experiments, developed RL-based search algorithm for finding transferable layer-wise pruning profiles.
result RL-based layer-wise pruning profiles are as good or better than best profiles found on the original dataset via exhaustive search.
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
problem Improving pruning of large language models to reduce size without sacrificing performance.
method AlphaPruning uses HT-SR theory to allocate layerwise sparsity ratios more theoretically.
result AlphaPruning prunes LLaMA-7B to 80% sparsity with reasonable perplexity.
Study examines effects of pruning techniques on deep learning models.
problem Understanding the impact of pruning methods on deep learning model structure and dynamics.
method Investigated differences in connectivity and learning dynamics of pruned models using various iterative pruning techniques.
result Emergence of structure in pruned models through magnitude-based unstructured pruning and weight rewinding.
Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers, often with little or no drop in classification accuracy. However, most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain c…
Pruning neural networks adds differential privacy noise, preserving data utility.
problem Achieving differential privacy in neural networks without sacrificing data utility.
method Proving equivalence between pruning and adding differential privacy noise to hidden-layer activations.
result Pruning can be a more effective alternative to adding differential privacy noise for neural networks.
Randomly initialized networks can perform as well as pruned networks.
problem Understanding and improving network pruning methods.
method Sanity checks on recent pruning methods, proposing random tickets.
result Randomly initialized networks can perform as well as pruned networks.
The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.
problem Flaws in existing pruning criteria for CNNs.
method Empirical experiments and Convolutional Weight Distribution Assumption.
result The Convolutional Weight Distribution Assumption improves filter selection in pruning.
New algorithm finds important synapses without training data.
problem Finding important synapses in neural networks without data.
method Iterative Synaptic Flow Pruning (SynFlow) based on conservation law.
result Algorithm consistently outperforms existing pruning methods.
A new method prunes deep networks in one go without specifying pruning levels.
problem Deep model compression to reduce model size and inference time.
method Learning a pruner network to identify and prune unnecessary filters from a pre-trained network.
result Pruned networks achieve comparable performance to unpruned ones, with significant reduction in model size.
Various forms of representations may arise in the many layers embedded in deep neural networks (DNNs). Of these, where can we find the most compact representation? We propose to use a pruning framework to answer this question: How compact can each layer be compressed, without losing performance? Most of the existing DN…
Dirichlet pruning compresses neural networks by removing unimportant units.
problem Compressing large neural network models without sacrificing performance.
method Assigns Dirichlet distribution over network layers' units and uses variational inference to estimate parameters.
result Achieves state-of-the-art compression performance on larger architectures like VGG and ResNet.
Channel Pruning, widely used for accelerating Convolutional Neural Networks, is an NP-hard problem due to the inter-layer dependency of channel redundancy. Existing methods generally ignored the above dependency for computation simplicity. To solve the problem, under the Bayesian framework, we here propose a layer-wise…
LTP learns per-layer thresholds for efficient pruning of deep networks.
problem Efficiently pruning deep neural networks to reduce computational cost and size.
method LTP learns thresholds via gradient descent, making pruning computationally efficient and scalable.
result LTP achieves competitive compression rates and maintains high accuracy on ImageNet networks.
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
problem Discovering efficient subnetworks within pre-trained dense networks.
method Mathematical analysis of a two-layer, fully-connected network, validating with a multi-layer perceptron trained on MNIST.
result A logarithmically dependent threshold on dataset size for successful pruning.
Study shows LLMs can remove half of layers without significant performance drop.
problem Understanding knowledge storage in LLMs' weights.
method Layer pruning and finetuning to identify and remove unnecessary parameters.
result Minimal degradation of performance after removing up to half of layers.
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
i-SpaSP prunes neural networks by identifying important groups of parameters, improving pruning efficiency.
problem Pruning neural networks to reduce computational cost and improve performance.
method i-SpaSP uses sparse signal recovery principles to iteratively identify and threshold important parameter groups.
result i-SpaSP achieves strong empirical results and theoretical convergence guarantees, improving pruning efficiency.
Unified framework for accelerating DNNs on resource-limited platforms.
problem Accelerating DNN execution on resource-limited platforms.
method Block-based pruning framework with reweighted regularization.
result First universal framework for both CNNs and RNNs with real-time acceleration and no accuracy compromise.
Compression techniques for deep neural networks are important for implementing them on small embedded devices. In particular, channel-pruning is a useful technique for realizing compact networks. However, many conventional methods require manual setting of compression ratios in each layer. It is difficult to analyze th…
Sparse Transformers degrade semantic information first, with early layers encoding more.
problem Understanding how sparse Transformers affect learned representations and semantic information.
method Probed Transformers with progressively pruned weights to observe changes in semantic information and model behavior.
result Complex semantic information is first to degrade in sparse Transformers, with early layers encoding more.
Data-independent pruning method reduces neural network size with accuracy guarantees.
problem Limited computational and memory resources for neural networks.
method Structured pruning using coresets.
result First efficient algorithm with worst-case guarantees on compression and accuracy.
ResRep prunes CNNs without losing accuracy by separating remembering and forgetting.
problem Pruning CNNs to reduce FLOPs without sacrificing accuracy.
method Decoupling remembering and forgetting in CNNs, using SGD for remembering and a novel update rule for forgetting.
result Achieved lossless pruning with high compression ratio (76.15% accuracy on ImageNet with 45% FLOPs reduction).
Paper proposes neural network for efficient MIMO channel estimation and pilot reduction.
problem High overhead from pilot transmission in wideband MIMO systems.
method Neural network architecture for frequency-aware pilot design and channel estimation, with pruning technique.
result Neural network outperforms linear minimum mean square error (LMMSE) estimation.
It is not easy to design and run Convolutional Neural Networks (CNNs) due to: 1) finding the optimal number of filters (i.e., the width) at each layer is tricky, given an architecture; and 2) the computational intensity of CNNs impedes the deployment on computationally limited devices. Oracle Pruning is designed to rem…
Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the …
New method grows deep networks efficiently by dynamically pruning and growing layers.
problem Training deep networks is computationally expensive and inefficient.
method Structured continuous sparsification starting from a small seed architecture.
result 49.7% inference FLOPs and 47.4% training FLOPs savings with 75.2% top-1 accuracy.
Following recent advances in morphological neural networks, we propose to study in more depth how Max-plus operators can be exploited to define morphological units and how they behave when incorporated in layers of conventional neural networks. Besides showing that they can be easily implemented with modern machine lea…
Deep Neural Network (DNN) is powerful but computationally expensive and memory intensive, thus impeding its practical usage on resource-constrained front-end devices. DNN pruning is an approach for deep model compression, which aims at eliminating some parameters with tolerable performance degradation. In this paper, w…
In this paper, we propose a novel progressive parameter pruning method for Convolutional Neural Network acceleration, named Structured Probabilistic Pruning (SPP), which effectively prunes weights of convolutional layers in a probabilistic manner. Unlike existing deterministic pruning approaches, where unimportant weig…
New pruning methods improve energy efficiency of neural networks.
problem Energy-efficient neural networks for devices with limited resources.
method Magnitude and Gradient based pruning at initialization and training of sparse architectures.
result Proposed novel pruning methods prevent full layer pruning and improve training.
Paper proposes efficient pruning method for neural networks.
problem Compressing deep neural networks for resource-constrained devices.
method Adaptive sparsity loss for budget-aware optimization during training.
result Demonstrated effectiveness on various architectures and datasets.
New pruning method captures global correlations for efficient neural network inference.
problem Efficiently pruning neural networks for faster inference and reduced memory usage.
method Second-order structured pruning (SOSP-H) with innovative saliency-based approaches.
result SOSP-H scales to large-scale vision tasks and improves accuracy without compromising efficiency.
This research investigates selectively pruning hyper and hypo neurons to improve neural network generalization.
problem Improving neural network generalization to unseen data.
method Investigates pruning hyper and hypo neurons selectively in fully connected layers of CNNs.
result Selective pruning of hyper and hypo neurons improves model performance on out-of-domain data.
In recent years, deep neural networks have achieved great success in the field of computer vision. However, it is still a big challenge to deploy these deep models on resource-constrained embedded devices such as mobile robots, smart phones and so on. Therefore, network compression for such platforms is a reasonable so…
Convolutional neural networks (CNNs) achieve state-of-the-art performance in a wide variety of tasks in computer vision. However, interpreting CNNs still remains a challenge. This is mainly due to the large number of parameters in these networks. Here, we investigate the role of compression and particularly pruning fil…
Dynamic Sparse Training finds efficient sparse networks from scratch.
problem Finding efficient sparse neural networks.
method Jointly optimizes network parameters and sparsity with trainable thresholds.
result Achieves state-of-the-art performance with minimal performance loss.
Binary Stochastic Filtering (BSF), the algorithm for feature selection and neuron pruning is proposed in this work. The method defines filtering layer which penalizes amount of the information involved in the training process. This information could be the input data or output of the previous layer, which directly lead…
The most common method for DNN pruning is hard thresholding of network weights, followed by retraining to recover any lost accuracy. Recently developed smart pruning algorithms use the DNN response over the training set for a variety of cost functions to determine redundant network weights, leading to less accuracy deg…
To deal with various datasets over different complexity, this paper presents an self-adaptive learning model that combines the proposed Dynamic Connected Neural Decision Networks (DNDN) and a new pruning method--Dynamic Soft Pruning (DSP). DNDN is a combination of random forests and deep neural networks that enjoys bot…
Optimization-based pruning eliminates backpropagation for large language models.
problem Suboptimal pruning performance due to heuristic metrics.
method Optimization of Bernoulli distribution to learn pruning masks without backpropagation.
result Efficient pruning of large language models with improved performance.
Despite enjoying extensive applications in video analysis, three-dimensional convolutional neural networks (3D CNNs)are restricted by their massive computation and storage consumption. To solve this problem, we propose a threedimensional regularization-based neural network pruning method to assign different regularizat…
A new pruning method improves neural network efficiency and accuracy.
problem Optimizing neural network efficiency and accuracy through pruning.
method Formulated as a Knapsack Problem, the method optimizes trade-off between neuron importance and computational cost. Channels are pruned while maintaining network structure, and fine-tuned using inner knowledge distillation from parent network.
result State-of-the-art pruning results on ImageNet, CIFAR-10, and CIFAR-100.
We develop a fast, tractable technique called Net-Trim for simplifying a trained neural network. The method is a convex post-processing module, which prunes (sparsifies) a trained network layer by layer, while preserving the internal responses. We present a comprehensive analysis of Net-Trim from both the algorithmic a…
Improved pruning method using iterative sensitivity ranking before training.
problem Improper sensitivity propagation in existing pruning methods.
method Iterative application of SNIP criterion before training.
result State-of-the-art sparsity-performance trade-offs achieved.