arXiv research

A new method prunes neural networks faster and more efficiently.

Random layer-wise pruning profiles are as effective as metric-based ones for various datasets.

Dataset pruning is the process of removing sub-optimal tuples from a dataset to improve the learning of a machine learning model. In this paper, we compared the performance of different algorithms, first on an unpruned dataset and then on an iteratively pruned dataset. The goal was to understand whether an algorithm (s…

New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.

Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work have been conducted on DNN model compression or pruning. However, most of the previous work took heuristic approac…

Dynamic sample pruning speeds up spatio-temporal forecasting models.

A new pruning method reduces neural network computation without retraining.

New pruning method breaks power law scaling, potentially reducing error to exponential.

Gibbs pruning optimizes neural networks by combining physics and regularization.

Hyperflux models pruning as a system to reveal weight importance.

ICE-Pruning accelerates deep neural network pruning by 9.61x.

Pruned neural networks' error scales predictably with architecture and task.

The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.

Parameter pruning is a promising approach for CNN compression and acceleration by eliminating redundant model parameters with tolerable performance degrade. Despite its effectiveness, existing regularization-based parameter pruning methods usually drive weights towards zero with large and constant regularization factor…

PARIS reduces imbalanced regression datasets by pruning uninformative samples.

Paper proposes efficient pruning method for neural networks.

Neural network for subgraph similarity computation with pruning.

Renormalized pruning improves neural network accuracy.

Optimization-based pruning eliminates backpropagation for large language models.

Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices. To achieve this goal, we introduce a novel network reparameterization based on the Kronecker-factored eigenbasis (KFE), and then apply Hessian-based structure…

i-SpaSP prunes neural networks by identifying important groups of parameters, improving pruning efficiency.

The enormous inference cost of deep neural networks can be scaled down by network compression. Pruning is one of the predominant approaches used for deep network compression. However, existing pruning techniques have one or more of the following limitations: 1) Additional energy cost on top of the compute heavy trainin…

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weigh…

We propose a new formulation for pruning convolutional kernels in neural networks to enable efficient inference. We interleave greedy criteria-based pruning with fine-tuning by backpropagation - a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion ba…

To deal with various datasets over different complexity, this paper presents an self-adaptive learning model that combines the proposed Dynamic Connected Neural Decision Networks (DNDN) and a new pruning method--Dynamic Soft Pruning (DSP). DNDN is a combination of random forests and deep neural networks that enjoys bot…

We propose to prune a random forest (RF) for resource-constrained prediction. We first construct a RF and then prune it to optimize expected feature cost & accuracy. We pose pruning RFs as a novel 0-1 integer program with linear constraints that encourages feature re-use. We establish total unimodularity of the constra…

New compression theory justifies model pruning for neural networks.

Compression techniques for deep neural networks are important for implementing them on small embedded devices. In particular, channel-pruning is a useful technique for realizing compact networks. However, many conventional methods require manual setting of compression ratios in each layer. It is difficult to analyze th…

A framework for privacy-preserving DNN pruning and acceleration.

BMRS offers a Bayesian approach to structured pruning of neural networks.

In recent years, deep neural networks have achieved great success in the field of computer vision. However, it is still a big challenge to deploy these deep models on resource-constrained embedded devices such as mobile robots, smart phones and so on. Therefore, network compression for such platforms is a reasonable so…

Proposes dynamic channel pruning during neural network training.

New pruning method for sparse additive models speeds up causal structure learning.

ForestPrune optimizes tree ensemble pruning for compactness and speed.

Structured weight pruning is a representative model compression technique of DNNs to reduce the storage and computation requirements and accelerate inference. An automatic hyperparameter determination process is necessary due to the large number of flexible hyperparameters. This work proposes AutoCompress, an automatic…

New algorithm finds important synapses without training data.

Pruning FCNs reveals sub-networks that match CNNs' performance.

Simple iterative method reduces deep network size significantly.

EVGAE improves VGAE's latent representation learning by mitigating over-pruning.

Neural networks have achieved dramatic improvements in recent years and depict the state-of-the-art methods for many real-world tasks nowadays. One drawback is, however, that many of these models are overparameterized, which makes them both computationally and memory intensive. Furthermore, overparameterization can als…

Predicting human fixations from images has recently seen large improvements by leveraging deep representations which were pretrained for object recognition. However, as we show in this paper, these networks are highly overparameterized for the task of fixation prediction. We first present a simple yet principled greedy…

Meta-learning with network pruning reduces overfitting and improves few-shot learning.

Artificial neural networks (ANNs) especially deep convolutional networks are very popular these days and have been proved to successfully offer quite reliable solutions to many vision problems. However, the use of deep neural networks is widely impeded by their intensive computational and memory cost. In this paper, we…

NTK-SAP improves neural network pruning by aligning training dynamics.

Besides accuracy, the model size of convolutional neural networks (CNN) models is another important factor considering limited hardware resources in practical applications. For example, employing deep neural networks on mobile systems requires the design of accurate yet fast CNN for low latency in classification and ob…

This paper improves Koopman operator approximations by pruning subspaces in RKHS.

Deep neural networks have achieved impressive performance in many applications but their large number of parameters lead to significant computational and storage overheads. Several recent works attempt to mitigate these overheads by designing compact networks using pruning of connections. However, we observe that most …

Federated learning (FL) allows model training from local data collected by edge/mobile devices while preserving data privacy, which has wide applicability to image and vision applications. A challenge is that client devices in FL usually have much more limited computation and communication resources compared to servers…