This work reduces computation cost for on-device CNN training.
problem High computation cost during on-device CNN training.
method Self-supervised instance filtering and error map pruning.
result Substantial computation saving without significant accuracy loss.
Pruned neural networks' error scales predictably with architecture and task.
problem Understanding the predictability of pruning across different scales and architectures.
method Functionally approximated the error of pruned networks, showing it is predictable in terms of invariant tying width, depth, and pruning level.
result The error of pruned networks follows a scaling law with interpretable coefficients that depend on architecture and task.
A new method quantifies feature-map discriminativeness for efficient pruning of deep neural networks.
problem Efficiently pruning deep neural networks to reduce computation while maintaining accuracy.
method Presented a novel mathematical formulation (Discriminant Information, DI) to quantify feature-map discriminativeness, enabling efficient pruning and intra-layer mixed precision quantization.
result Our pruned ResNet50 achieves 44% FLOPs reduction without any Top-1 accuracy loss.
Optimizes pruning masks for neural networks using probabilistic fine-tuning and PAC-Bayes bounds.
problem Improving neural network performance through adaptive pruning of weights.
method Optimizes stochastic pruning masks by minimizing expected loss, considering data-adaptive regularization and feature alignment.
result Probabilistic fine-tuning leads to improved test error over baseline methods in neural networks.
Renormalized pruning improves neural network accuracy.
problem Over-parameterized neural networks waste many parameters.
method Propose renormalizing sparse neural networks.
result Renormalized pruning converges to zero error.
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
This paper studies a theoretical pruning method for RNNs to reduce computational costs.
problem High computational costs in recurrent neural networks (RNNs).
method Spectral pruning inspired approach for RNNs.
result Generalization error bounds for compressed RNNs are provided.
Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers, often with little or no drop in classification accuracy. However, most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain c…
New pruning method breaks power law scaling, potentially reducing error to exponential.
problem Improving neural network performance through scaling alone is costly.
method Developed a new data pruning metric to break power law scaling.
result Pruned datasets show better than power law scaling on various image datasets.
Artificial neural networks (ANNs) may not be worth their computational/memory costs when used in mobile phones or embedded devices. Parameter-pruning algorithms combat these costs, with some algorithms capable of removing over 90% of an ANN's weights without harming the ANN's performance. Removing weights from an ANN i…
IMP finds sparse subnetworks that match full networks, revealing geometric insights.
problem Finding sparse subnetworks that match full, overparameterized networks.
method Iterative magnitude pruning (IMP) algorithm, analyzing error landscape geometry.
result IMP masks found at end of training convey useful information for rewound networks.
Bayesian Bits unifies quantization and pruning through gradient optimization.
problem Joint mixed precision quantization and pruning for efficient neural networks.
method Gradient-based optimization with a novel bit width decomposition and learnable stochastic gates.
result Bayesian Bits achieves better accuracy vs. efficiency trade-off compared to static bit width networks.
Data-independent pruning method reduces neural network size with accuracy guarantees.
problem Limited computational and memory resources for neural networks.
method Structured pruning using coresets.
result First efficient algorithm with worst-case guarantees on compression and accuracy.
Alpha-trimming prunes trees in random forests to improve predictive performance.
problem Improving predictive performance of random forests by locally adaptive tree pruning.
method Alpha-trimming is a fast pruning algorithm that prunes trees in a random forest based on signal-to-noise ratio, controlled by a tuning parameter.
result Alpha-trimming often lowers mean squared prediction error compared to fully grown random forests.
i-SpaSP prunes neural networks by identifying important groups of parameters, improving pruning efficiency.
problem Pruning neural networks to reduce computational cost and improve performance.
method i-SpaSP uses sparse signal recovery principles to iteratively identify and threshold important parameter groups.
result i-SpaSP achieves strong empirical results and theoretical convergence guarantees, improving pruning efficiency.
Compression techniques for deep neural network models are becoming very important for the efficient execution of high-performance deep learning systems on edge-computing devices. The concept of model compression is also important for analyzing the generalization error of deep learning, known as the compression-based er…
Grafting a measured lamination on a hyperbolic surface defines a self-map of Teichmuller space, which is a homeomorphism by a result of Scannell and Wolf. In this paper we study the large-scale behavior of pruning, which is the inverse of grafting. Specifically, for each conformal structure $X \in \T(S)$, pruning X g…
We propose a new formulation for pruning convolutional kernels in neural networks to enable efficient inference. We interleave greedy criteria-based pruning with fine-tuning by backpropagation - a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion ba…
Proposes dynamic channel pruning during neural network training.
problem Pruning neural networks during training to reduce computational cost and improve efficiency.
method Dynamic channel propagation to update channel utility values and selectively prune channels.
result Our scheme trains and prunes neural networks simultaneously, achieving superior performance.
A new method prunes neural network channels based on operation characteristics.
problem Compressing deep neural networks efficiently and maintaining accuracy.
method Differentiable masks for channel pruning considering BN and ReLU.
result Outstanding performance in accuracy with less resources compared to state-of-the-art methods.
Artificial neural networks face the well-known problem of catastrophic forgetting. What's worse, the degradation of previously learned skills becomes more severe as the task sequence increases, known as the long-term catastrophic forgetting. It is due to two facts: first, as the model learns more tasks, the intersectio…
Ensembles, as a widely used and effective technique in the machine learning community, succeed within a key element -- "diversity." The relationship between diversity and generalization, unfortunately, is not entirely understood and remains an open research issue. To reveal the effect of diversity on the generalization…
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
problem Discovering efficient subnetworks within pre-trained dense networks.
method Mathematical analysis of a two-layer, fully-connected network, validating with a multi-layer perceptron trained on MNIST.
result A logarithmically dependent threshold on dataset size for successful pruning.
Random forests perform bootstrap-aggregation by sampling the training samples with replacement. This enables the evaluation of out-of-bag error which serves as a internal cross-validation mechanism. Our motivation lies in using the unsampled training samples to improve each decision tree in the ensemble. We study the e…
As a result of the growing size of Deep Neural Networks (DNNs), the gap to hardware capabilities in terms of memory and compute increases. To effectively compress DNNs, quantization and connection pruning are usually considered. However, unconstrained pruning usually leads to unstructured parallelism, which maps poorly…
Pruning + fine-tuning reduces model complexity and generalizes well for matrix sensing.
problem Reducing model complexity for matrix sensing problems.
method Group Lasso regularization and greedy pruning.
result Pruning results in a solution with minimum columns close to the ground truth.
It is not easy to design and run Convolutional Neural Networks (CNNs) due to: 1) finding the optimal number of filters (i.e., the width) at each layer is tricky, given an architecture; and 2) the computational intensity of CNNs impedes the deployment on computationally limited devices. Oracle Pruning is designed to rem…
Gradual pruning reduces inference cost by pruning least important channels during training.
problem Reduction of deep neural network inference cost.
method Gradual channel pruning using feature relevance scores during training.
result Achieved significant model compression with minimal accuracy loss.
Dynamic Vocabulary Pruning stabilizes LLM training by removing low-probability tokens.
problem Training Large Language Models (LLMs) with Reinforcement Learning (RL) causes numerical divergence between inference and training.
method Dynamic Vocabulary Pruning (DVP) constrains the RL objective to a safe vocabulary that excludes low-probability tokens.
result DVP stabilizes training by reducing systematic bias introduced by the extreme tail of the token distribution.
We present a filter pruning approach for deep model compression, using a multitask network. Our approach is based on learning a a pruner network to prune a pre-trained target network. The pruner is essentially a multitask deep neural network with binary outputs that help identify the filters from each layer of the orig…
RicciNets prunes neural networks by removing edges of low importance based on Ricci curvature, reducing FLOPs by 35%.
problem Pruning neural networks to reduce computational load and improve efficiency.
method RicciNets uses Ricci curvature to prune edges of low importance in a randomly wired neural network, reducing FLOPs.
result Reduction of almost 35% in FLOPs with no performance degradation.
Convolutional Neural Networks(CNNs) are both computation and memory intensive which hindered their deployment in mobile devices. Inspired by the relevant concept in neural science literature, we propose Synaptic Pruning: a data-driven method to prune connections between input and output feature maps with a newly propos…
Real time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also …
New pruning method captures global correlations for efficient neural network inference.
problem Efficiently pruning neural networks for faster inference and reduced memory usage.
method Second-order structured pruning (SOSP-H) with innovative saliency-based approaches.
result SOSP-H scales to large-scale vision tasks and improves accuracy without compromising efficiency.
Neural networks are easier to optimise when they have many more weights than are required for modelling the mapping from inputs to outputs. This suggests a two-stage learning procedure that first learns a large net and then prunes away connections or hidden units. But standard training does not necessarily encourage ne…
Diffusion maps are a nonlinear manifold learning technique based on harmonic analysis of a diffusion process over the data. Out-of-sample extensions with computational complexity O(N), where N is the number of points comprising the manifold, frustrate applications to online learning applications requiring…
Optimal defenses protect FL models from gradient reconstruction attacks.
problem Gradient reconstruction attacks compromise FL models by recovering original data from shared gradients.
method Derive a theoretical lower bound of reconstruction error, customize noise and pruning defenses, and achieve optimal trade-off between leakage and utility.
result Our methods outperform Gradient Noise and Pruning in protecting training data and maintaining model utility.
This paper introduces GLT for better input data representation in BNN and proposes a compact topology with block pruning.
problem Improving input data representation for Binary Neural Networks (BNN).
method Generic Learned Thermometer (GLT) for encoding, block pruning and Knowledge Distillation for compact topology.
result Significant accuracy gains and lightweight fully-binarized models with limited accuracy degradation.
The paper studies statistical properties of CART regression trees.
problem Understanding the statistical properties of CART regression trees.
method The paper constructs a prior distribution on split points and solves a nonlinear optimization problem to bound the Pearson correlation between the optimal decision stump and response data.
result CART with cost-complexity pruning achieves an optimal complexity/goodness-of-fit tradeoff when the depth scales with the logarithm of the sample size.
Sparser Random Feature Models via IMP (ShRIMP) efficiently learns sparse models for high-dimensional data.
problem Learning sparse models for high-dimensional data with sparse variable dependencies.
method Iterative Magnitude Pruning applied to Random Feature Models.
result ShRIMP achieves better or competitive test accuracy compared to state-of-the-art methods.
New study reveals how heavy-tailed SGD dynamics lead to compressible neural networks.
problem Understanding why large neural networks can be compressed effectively.
method Linking SGD dynamics to compressibility properties of neural networks.
result Large step-size/batch-size ratios and overparametrization lead to heavy-tailed SGD dynamics, making networks compressible.
Paper proposes neural network for efficient MIMO channel estimation and pilot reduction.
problem High overhead from pilot transmission in wideband MIMO systems.
method Neural network architecture for frequency-aware pilot design and channel estimation, with pruning technique.
result Neural network outperforms linear minimum mean square error (LMMSE) estimation.
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
ICE-Pruning accelerates deep neural network pruning by 9.61x.
problem Efficiently pruning deep neural networks while maintaining accuracy.
method Iterative pruning with automatic fine-tuning steps, freezing strategy, and custom learning rate scheduler.
result Significantly reduces pruning time by up to 9.61x.
DSA efficiently allocates sparsity across layers for budgeted pruning.
problem Efficiently distributing resources (sparsity) across layers in pruning under resource constraints.
method DSA uses differentiable pruning to find continuous layer-wise pruning ratios via gradient-based optimization.
result DSA achieves superior performance and significantly reduces the time cost of pruning.
Predict accuracy of neural architectures using non-neural models.
problem Improving efficiency and effectiveness of neural architecture search.
method Used gradient boosting decision tree (GBDT) for accuracy prediction and pruned the search space based on features learned from GBDT.
result GBDT-based predictor achieves comparable or better accuracy than neural network-based predictors and is 22x more sample efficient on NASBench-101.
Magnitude-based pruning is one of the simplest methods for pruning neural networks. Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable performances for pruning modern architectures. Based on the observation that magnitude-based pruning indeed minimizes the Frobenius distortion of a…
While recent NAS algorithms are thousands of times faster than the pioneering works, it is often overlooked that they use fewer candidate operations, resulting in a significantly smaller search space. We present PR-DARTS, a NAS algorithm that discovers strong network configurations in a much larger search space and a s…