DSA efficiently allocates sparsity across layers for budgeted pruning.
problem Efficiently distributing resources (sparsity) across layers in pruning under resource constraints.
method DSA uses differentiable pruning to find continuous layer-wise pruning ratios via gradient-based optimization.
result DSA achieves superior performance and significantly reduces the time cost of pruning.
This paper introduces blind adversarial pruning to balance accuracy, efficiency, and robustness in neural networks.
problem Balancing accuracy, efficiency, and robustness in neural networks with limited resources.
method Adversarial pruning with a cutoff-scale strategy to dynamically adjust the strength of adversarial examples.
result Blind adversarial pruning improves the overall AER of pruned models compared to adversarial pruning.
Paper proposes efficient pruning method for neural networks.
problem Compressing deep neural networks for resource-constrained devices.
method Adaptive sparsity loss for budget-aware optimization during training.
result Demonstrated effectiveness on various architectures and datasets.
We consider the problem of learning decision rules for prediction with feature budget constraint. In particular, we are interested in pruning an ensemble of decision trees to reduce expected feature cost while maintaining high prediction accuracy for any test example. We propose a novel 0-1 integer program formulation …
PAC-MCTS addresses biased search in LLM-guided planning by dynamically pruning.
problem Systematic biases in LLMs lead to inefficient and unsafe search in deep planning tasks.
method Formulates node expansion as BAI under bounded bias, derives sample complexity bounds, and proposes PAC-MCTS for dynamic confidence bounds.
result PAC-MCTS improves robustness and efficiency by up to 78% fewer API evaluations and 3x higher sample efficiency.
We propose to prune a random forest (RF) for resource-constrained prediction. We first construct a RF and then prune it to optimize expected feature cost & accuracy. We pose pruning RFs as a novel 0-1 integer program with linear constraints that encourages feature re-use. We establish total unimodularity of the constra…
Holistic Filter Pruning reduces DNN complexity efficiently.
problem Redundant parameters in deep neural networks.
method Holistic Filter Pruning (HFP) for efficient DNN training.
result Achieves state-of-the-art performance with 60% reduction in multiplications on ImageNet.
A new method quantifies feature-map discriminativeness for efficient pruning of deep neural networks.
problem Efficiently pruning deep neural networks to reduce computation while maintaining accuracy.
method Presented a novel mathematical formulation (Discriminant Information, DI) to quantify feature-map discriminativeness, enabling efficient pruning and intra-layer mixed precision quantization.
result Our pruned ResNet50 achieves 44% FLOPs reduction without any Top-1 accuracy loss.
The task of accelerating large neural networks on general purpose hardware has, in recent years, prompted the use of channel pruning to reduce network size. However, the efficacy of pruning based approaches has since been called into question. In this paper, we turn to distillation for model compression---specifically,…
STR reparameterizes DNN weights with soft thresholds for better sparsity and accuracy.
problem Improving sparsity in DNNs for better accuracy and lower inference cost.
method Soft Threshold Reparameterization (STR) using the soft-threshold operator on DNN weights.
result STR achieves state-of-the-art accuracy and reduces FLOPs by up to 50%.
The search for efficient, sparse deep neural network models is most prominently performed by pruning: training a dense, overparameterized network and removing parameters, usually via following a manually-crafted heuristic. Additionally, the recent Lottery Ticket Hypothesis conjectures that, for a typically-sized neural…
Simultaneous Latent Budget Trees for stratified classification
problem Classification with stratification factors
method Probabilistic machine learning framework
result Interpretation of latent components and conditional split rule
PDO optimizes LLM prompts without labels, improving performance.
problem Optimizing prompts for LLMs without access to labeled data.
method Pairwise preference feedback, dueling bandits, Thompson Sampling, mutation.
result PDO identifies stronger prompts than label-free methods.
We introduce a DNN training technique that learns only a fraction of the full parameter set without incurring an accuracy penalty. To do this, our algorithm constrains the total number of weights updated during backpropagation to those with the highest total gradients. The remaining weights are not tracked, and their i…
FS&P uses birth-death process to ensure global convergence of stochastic conic particle gradient descent.
problem Global optimization of non-convex objective functions over measure space.
method Introduces Fast Spawn\&Prune (FS\&P) combining CPGD with birth-death process.
result First theoretical guarantee of global convergence for discrete-time stochastic algorithms.
Lookahead pruning extends single-layer optimization to multi-layer, outperforming magnitude-based pruning.
problem Pruning neural networks to reduce computational cost and memory usage.
method Developed a multi-layer optimization approach extending the single-layer optimization of magnitude-based pruning.
result Consistently outperforms magnitude-based pruning on various networks, especially in high sparsity.
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
ICE-Pruning accelerates deep neural network pruning by 9.61x.
problem Efficiently pruning deep neural networks while maintaining accuracy.
method Iterative pruning with automatic fine-tuning steps, freezing strategy, and custom learning rate scheduler.
result Significantly reduces pruning time by up to 9.61x.
Pruning neural network parameters is often viewed as a means to compress models, but pruning has also been motivated by the desire to prevent overfitting. This motivation is particularly relevant given the perhaps surprising observation that a wide variety of pruning approaches increase test accuracy despite sometimes …
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
Modern deep neural networks are typically highly overparameterized. Pruning techniques are able to remove a significant fraction of network parameters with little loss in accuracy. Recently, techniques based on dynamic reallocation of non-zero parameters have emerged, allowing direct training of sparse networks without…
The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.
problem Flaws in existing pruning criteria for CNNs.
method Empirical experiments and Convolutional Weight Distribution Assumption.
result The Convolutional Weight Distribution Assumption improves filter selection in pruning.
Deep neural networks have dramatically achieved great success on a variety of challenging tasks. However, most successful DNNs have an extremely complex structure, leading to extensive research on model compression.As a significant area of progress in model compression, traditional gradual pruning approaches involve an…
Gibbs pruning optimizes neural networks by combining physics and regularization.
problem Large neural networks are impractical for many applications.
method Combines statistical physics and stochastic regularization to train and prune networks simultaneously.
result Gibbs pruning achieves state-of-the-art performance on ResNet-56.
We examine how recently documented, fundamental phenomena in deep learning models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through a set of common iterative pruning techniques, to …
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
AdaDPIGU improves privacy in deep learning by adaptively clipping and pruning gradients.
problem Privacy in deep learning models, especially in high-dimensional settings.
method Importance-based gradient updates, adaptive clipping, differentially private SGD.
result AdaDPIGU achieves high accuracy while maintaining privacy, outperforming non-private models.
Speeds up training and inference by pruning entire channels before training.
problem Training and inference speed in deep neural networks.
method Structured pruning applied before training, focusing on removing entire channels and hidden units.
result 2x speedup in training and 3x speedup in inference.
This paper analyzes privacy risks in neural network pruning and proposes a defense mechanism.
problem Privacy risks in neural network pruning due to membership inference attacks.
method Investigates the impact of pruning on prediction divergence and proposes a self-attention membership inference attack.
result Proposed defense mechanism mitigates privacy risks while maintaining sparsity and accuracy.
Neural network pruning lacks standardized benchmarks and metrics.
problem Lack of standardized benchmarks and metrics in neural network pruning.
method Meta-analysis of 81 papers, controlled conditions, ShrinkBench framework.
result Neural network pruning community lacks standardized benchmarks and metrics.
Bayesian inference improves neural network pruning efficiency.
problem Reducing computational and memory demands of large neural networks.
method Utilizes Bayesian inference to calculate Bayes factors for iterative pruning.
result Achieves desired levels of sparsity while maintaining competitive accuracy.
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
problem Improving pruning of large language models to reduce size without sacrificing performance.
method AlphaPruning uses HT-SR theory to allocate layerwise sparsity ratios more theoretically.
result AlphaPruning prunes LLaMA-7B to 80% sparsity with reasonable perplexity.
This work characterizes the fundamental limit of network pruning using statistical dimension and convex geometry.
problem The fundamental limit of network pruning is still lacking, especially for deep neural networks.
method Directly imposing sparsity constraint on the loss function and using statistical dimension in convex geometry.
result Characterizes the sharp phase transition point as the fundamental limit of pruning ratio.
A new framework explains why early pruning works well.
problem Understanding why early pruning of neural networks leads to good performance.
method Gradient flow framework to unify pruning measures.
result Magnitude-based pruning removes least contributing parameters, leading to faster convergence.
CupNet prunes neural nets for cup-shaped data.
problem Pruning neural networks for cup-shaped data.
method Used simulated cup drawing data to prune a neural network.
result Pruning effectively reduces network size for cup-shaped data.
This work explores the importance of model weights and Hessian bias in pruning.
problem Understanding the relative importance of model weights for efficient pruning.
method A principled exploration of pruning, focusing on linear models and neural networks.
result Asymptotic formulas reveal the performance of different pruning methods.
A method to combine saliency metrics for better CNN pruning decisions.
problem Improving CNN pruning decisions by combining multiple saliency metrics.
method Proposes a method to compose different saliency metrics for better CNN pruning decisions.
result The composition of saliencies avoids many poor pruning choices identified by individual saliencies.
Pruning improves model generalization in over-parameterized models, contradicting traditional theories.
problem Pruning's effect on generalization in over-parameterized models.
method Empirical study on standard pruning algorithms and additional regularization effects.
result Pruning leads to better training and regularization, improving generalization.
Hyperflux models pruning as a system to reveal weight importance.
problem Pruning large neural networks to reduce latency and power consumption.
method Introduces Hyperflux, a novel L0 method that models pruning as flux and pressure. result Achieves competitive results with ResNet-50, VGG-19, and DeiT-T/S on various datasets.
This paper studies a theoretical pruning method for RNNs to reduce computational costs.
problem High computational costs in recurrent neural networks (RNNs).
method Spectral pruning inspired approach for RNNs.
result Generalization error bounds for compressed RNNs are provided.
Neural network for subgraph similarity computation with pruning.
problem Computing subgraph similarity between a target and query graph.
method Convert pruning to node relabeling, relax to differentiable problem, design neural network for SED computation.
result Establishes new state-of-the-art results across multiple benchmark datasets.
Network pruning is a promising avenue for compressing deep neural networks. A typical approach to pruning starts by training a model and then removing redundant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done at initialization prior to training,…
Data pruning algorithms struggle in high compression regimes, as shown by theoretical and empirical studies.
problem Limitations of score-based data pruning algorithms in high compression regimes.
method Theoretical and empirical analysis of score-based data pruning algorithms.
result Score-based data pruning algorithms fail in high compression regimes due to 'No Free Lunch' theorems.
StatLoRA uses statistical inference to allocate ranks in LoRA fine-tuning, improving performance.
problem Balancing efficiency, expressiveness, and generalization in LoRA rank allocation.
method Formulates LoRA rank allocation as a statistical hypothesis testing problem, using estimated p-values to determine component retention or pruning.
result StatLoRA achieves comparable or better performance than existing methods under matched rank budgets.
A new method prunes neural networks faster and more efficiently.
problem Reducing training time and memory usage for neural networks.
method Set-based Task-Adaptive Meta Pruning (STAMP) that meta-learns a pruning mask.
result Significantly improved compression rates and faster training speed.
Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work have been conducted on DNN model compression or pruning. However, most of the previous work took heuristic approac…
PruneNet efficiently prunes channels in deep networks, improving accuracy and performance.
problem Improving deep neural network performance and efficiency through channel pruning.
method PruneNet uses a computationally light-weight optimization step to identify and prune channels based on layer redundancy.
result Pruned ResNet models achieve higher accuracy and better performance than non-pruned models.
Pruning method removes less important features in linear models.
problem Removing less important features in linear models trained by gradient flow.
method Iterative Magnitude Pruning (IMP) applied to linear models trained by gradient flow.
result IMP prunes features with smallest projection onto the data.