New method prunes neural networks while conserving resources.
problem Design efficient neural networks for edge devices with limited computational power.
method Iterative channel-wise pruning with holonomic constraint in Lagrangian multipliers framework.
result Reduction of 15.47 GMAC to 3.87 GMAC for VGG-16 with only 1% top-1 accuracy loss.
Random layer-wise pruning profiles are as effective as metric-based ones for various datasets.
problem Reduction of model size and computational resources in neural networks.
method Conducted baseline experiments, developed RL-based search algorithm for finding transferable layer-wise pruning profiles.
result RL-based layer-wise pruning profiles are as good or better than best profiles found on the original dataset via exhaustive search.
Real time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also …
RecNets use RNNs to process image channels in a compact, recurrent way.
problem Creating efficient neural network architectures for computer vision.
method Introducing RecNets with CRC layers that simulate recurrent processing of image channels.
result RecNets achieve superior size-accuracy trade-off compared to other compact models.
A new method for early stopping in neural networks without validation sets.
problem Determining when to stop training neural networks to avoid overfitting.
method Channel-wise DeepNNK (CW-DeepNNK) using non-negative kernel regression and polytope interpolation.
result The proposed early stopping criterion based on CW-DeepNNK performs better than standard validation-based methods.
We propose doubly nested network(DNNet) where all neurons represent their own sub-models that solve the same task. Every sub-model is nested both layer-wise and channel-wise. While nesting sub-models layer-wise is straight-forward with deep-supervision as proposed in \cite{xie2015holistically}, channel-wise nesting has…
PixelHop++ improves image classification with a smaller model size.
problem Improving image classification models with smaller sizes.
method Decomposing input tensor, channel-wise Saab transform, successive subspace learning, feature ranking.
result PixelHop++ offers a flexible tradeoff between model size and performance.
New graph attention operators improve performance and reduce computational costs.
problem Excessive computational resources in graph attention operators.
method Proposed hGAO and cGAO using hard and channel-wise attention mechanisms.
result Improved performance and computational savings with new operators.
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
FEM improves attention mechanisms by applying value-driven log-linear tilts.
problem Standard attention mechanisms read via convex average, limiting channel-wise selection.
method Free Energy Mixer (FEM) applies a value-driven, per-channel log-linear tilt to a fast prior over indices.
result FEM outperforms strong baselines on NLP, vision, and time-series tasks.
SAMformer improves transformer performance in time series forecasting.
problem Transformers struggle with multivariate long-term forecasting.
method Sharpness-aware minimization and channel-wise attention.
result SAMformer surpasses state-of-the-art methods in multivariate time series forecasting.
New technique trains deep neural networks without normalization or minibatch statistics.
problem Training deep neural networks at high learning rates without normalization.
method Channel-wise zero-mean initialization and gradient modification to maintain common mode rejection.
result Achieves higher accuracy compared to batch normalization and shows minibatches are unnecessary.
Lookahead pruning extends single-layer optimization to multi-layer, outperforming magnitude-based pruning.
problem Pruning neural networks to reduce computational cost and memory usage.
method Developed a multi-layer optimization approach extending the single-layer optimization of magnitude-based pruning.
result Consistently outperforms magnitude-based pruning on various networks, especially in high sparsity.
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
ICE-Pruning accelerates deep neural network pruning by 9.61x.
problem Efficiently pruning deep neural networks while maintaining accuracy.
method Iterative pruning with automatic fine-tuning steps, freezing strategy, and custom learning rate scheduler.
result Significantly reduces pruning time by up to 9.61x.
DSA efficiently allocates sparsity across layers for budgeted pruning.
problem Efficiently distributing resources (sparsity) across layers in pruning under resource constraints.
method DSA uses differentiable pruning to find continuous layer-wise pruning ratios via gradient-based optimization.
result DSA achieves superior performance and significantly reduces the time cost of pruning.
Study examines effects of pruning techniques on deep learning models.
problem Understanding the impact of pruning methods on deep learning model structure and dynamics.
method Investigated differences in connectivity and learning dynamics of pruned models using various iterative pruning techniques.
result Emergence of structure in pruned models through magnitude-based unstructured pruning and weight rewinding.
Pruning neural networks can improve test accuracy even with significant parameter reduction.
problem The tradeoff between generalization and stability in neural network pruning.
method Analysis of pruning behavior over training, focusing on instability and its relation to generalization.
result Pruning's benefit to generalization increases with its instability.
Drop Pruning uses stochastic optimization to prune and recover weights, reducing model size and improving performance.
problem Complexity and inefficiency in pruning deep neural networks.
method Introduces stochastic optimization with 'drop away' and 'drop back' strategies to prune and recover weights.
result Achieves competitive compression performance and accuracy compared to state-of-the-art approaches.
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.
problem Flaws in existing pruning criteria for CNNs.
method Empirical experiments and Convolutional Weight Distribution Assumption.
result The Convolutional Weight Distribution Assumption improves filter selection in pruning.
Gibbs pruning optimizes neural networks by combining physics and regularization.
problem Large neural networks are impractical for many applications.
method Combines statistical physics and stochastic regularization to train and prune networks simultaneously.
result Gibbs pruning achieves state-of-the-art performance on ResNet-56.
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
New method prunes neural networks at initialization, improving performance.
problem Improving neural network compression at initialization.
method Formally characterizes initialization conditions for reliable pruning based on connection sensitivity.
result Improved neural network performance on image classification tasks.
This paper introduces blind adversarial pruning to balance accuracy, efficiency, and robustness in neural networks.
problem Balancing accuracy, efficiency, and robustness in neural networks with limited resources.
method Adversarial pruning with a cutoff-scale strategy to dynamically adjust the strength of adversarial examples.
result Blind adversarial pruning improves the overall AER of pruned models compared to adversarial pruning.
Speeds up training and inference by pruning entire channels before training.
problem Training and inference speed in deep neural networks.
method Structured pruning applied before training, focusing on removing entire channels and hidden units.
result 2x speedup in training and 3x speedup in inference.
This paper analyzes privacy risks in neural network pruning and proposes a defense mechanism.
problem Privacy risks in neural network pruning due to membership inference attacks.
method Investigates the impact of pruning on prediction divergence and proposes a self-attention membership inference attack.
result Proposed defense mechanism mitigates privacy risks while maintaining sparsity and accuracy.
Neural network pruning lacks standardized benchmarks and metrics.
problem Lack of standardized benchmarks and metrics in neural network pruning.
method Meta-analysis of 81 papers, controlled conditions, ShrinkBench framework.
result Neural network pruning community lacks standardized benchmarks and metrics.
Bayesian inference improves neural network pruning efficiency.
problem Reducing computational and memory demands of large neural networks.
method Utilizes Bayesian inference to calculate Bayes factors for iterative pruning.
result Achieves desired levels of sparsity while maintaining competitive accuracy.
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
problem Improving pruning of large language models to reduce size without sacrificing performance.
method AlphaPruning uses HT-SR theory to allocate layerwise sparsity ratios more theoretically.
result AlphaPruning prunes LLaMA-7B to 80% sparsity with reasonable perplexity.
This work characterizes the fundamental limit of network pruning using statistical dimension and convex geometry.
problem The fundamental limit of network pruning is still lacking, especially for deep neural networks.
method Directly imposing sparsity constraint on the loss function and using statistical dimension in convex geometry.
result Characterizes the sharp phase transition point as the fundamental limit of pruning ratio.
A new framework explains why early pruning works well.
problem Understanding why early pruning of neural networks leads to good performance.
method Gradient flow framework to unify pruning measures.
result Magnitude-based pruning removes least contributing parameters, leading to faster convergence.
CupNet prunes neural nets for cup-shaped data.
problem Pruning neural networks for cup-shaped data.
method Used simulated cup drawing data to prune a neural network.
result Pruning effectively reduces network size for cup-shaped data.
Structured pruning yields inferior results compared to training from scratch.
problem Efficacy of structured pruning in neural network compression.
method Alternating pruning and fine-tuning of large networks, comparing to training from scratch.
result Reduced networks trained from scratch outperform pruned networks.
This paper proposes an ADMM-based method for progressive weight pruning of deep neural networks.
problem Large model size of deep neural networks hinder their applications on edge devices.
method Progressive weight pruning using ADMM for non-convex optimization problems.
result Achieves up to 34 times pruning rate for ImageNet and 167 times for MNIST datasets.
Network pruning is often unnecessary and can be replaced by training a smaller model directly.
problem The inefficiency of network pruning and the need for more efficient model training.
method Examination of state-of-the-art structured pruning algorithms and direct training of target networks.
result Training a smaller model directly is often more efficient than pruning and fine-tuning a larger model.
Hard thresholding remains efficient for DNN pruning, but smart pruning offers faster accuracy recovery.
problem Efficiently pruning deep neural networks while minimizing accuracy loss.
method Proposes a novel smart pruning algorithm based on difference of convex functions optimization.
result Smart pruning is often orders of magnitude faster than competing approaches while achieving low accuracy degradation.
This work explores the importance of model weights and Hessian bias in pruning.
problem Understanding the relative importance of model weights for efficient pruning.
method A principled exploration of pruning, focusing on linear models and neural networks.
result Asymptotic formulas reveal the performance of different pruning methods.
A method to combine saliency metrics for better CNN pruning decisions.
problem Improving CNN pruning decisions by combining multiple saliency metrics.
method Proposes a method to compose different saliency metrics for better CNN pruning decisions.
result The composition of saliencies avoids many poor pruning choices identified by individual saliencies.
Pruning improves model generalization in over-parameterized models, contradicting traditional theories.
problem Pruning's effect on generalization in over-parameterized models.
method Empirical study on standard pruning algorithms and additional regularization effects.
result Pruning leads to better training and regularization, improving generalization.
Hyperflux models pruning as a system to reveal weight importance.
problem Pruning large neural networks to reduce latency and power consumption.
method Introduces Hyperflux, a novel L0 method that models pruning as flux and pressure. result Achieves competitive results with ResNet-50, VGG-19, and DeiT-T/S on various datasets.
This paper studies a theoretical pruning method for RNNs to reduce computational costs.
problem High computational costs in recurrent neural networks (RNNs).
method Spectral pruning inspired approach for RNNs.
result Generalization error bounds for compressed RNNs are provided.
Neural network for subgraph similarity computation with pruning.
problem Computing subgraph similarity between a target and query graph.
method Convert pruning to node relabeling, relax to differentiable problem, design neural network for SED computation.
result Establishes new state-of-the-art results across multiple benchmark datasets.
Paper proposes a faster DNN pruning method.
problem Efficiently prune DNNs without sacrificing accuracy.
method Uses PCA to automatically optimize pruning.
result Prunes DNNs in one shot, no retraining needed.
Data pruning algorithms struggle in high compression regimes, as shown by theoretical and empirical studies.
problem Limitations of score-based data pruning algorithms in high compression regimes.
method Theoretical and empirical analysis of score-based data pruning algorithms.
result Score-based data pruning algorithms fail in high compression regimes due to 'No Free Lunch' theorems.
A new method prunes neural networks faster and more efficiently.
problem Reducing training time and memory usage for neural networks.
method Set-based Task-Adaptive Meta Pruning (STAMP) that meta-learns a pruning mask.
result Significantly improved compression rates and faster training speed.
PruneNet efficiently prunes channels in deep networks, improving accuracy and performance.
problem Improving deep neural network performance and efficiency through channel pruning.
method PruneNet uses a computationally light-weight optimization step to identify and prune channels based on layer redundancy.
result Pruned ResNet models achieve higher accuracy and better performance than non-pruned models.
PCONV combines fine-grained and coarse-grained pruning for efficient DNN inference on mobile devices.
problem Achieving high sparsity and accuracy in DNN weight pruning for real-time mobile execution.
method PCONV introduces a new sparsity dimension by combining fine-grained pruning patterns inside coarse-grained structures.
result PCONV outperforms state-of-the-art frameworks in speed and efficiency without accuracy loss.