TF-Coder simplifies tensor manipulation programming in TensorFlow.
problem Difficulty in programming with TensorFlow due to steep learning curve.
method Bottom-up weighted enumerative search with value-based pruning and type/value filtering.
result Solves 63 out of 70 real-world tasks within 5 minutes.
New issue found in value-based reinforcement learning for stochastic environments.
problem Value-based reinforcement learning struggles with stochastic state transitions.
method Demonstrated using a multiobjective Markov Decision Process (MOMDP).
result Approaches may converge to Pareto-dominated solutions instead of optimal ones.
Overparameterized models generalize well in offline contextual bandits, but policy-based algorithms struggle.
problem The performance gap between value-based and policy-based algorithms in offline contextual bandits with overparameterized models.
method Analysis of action-stability in objectives and formal proofs of regret bounds.
result The performance gap is due to action-stability of objectives, with value-based objectives being stable and policy-based objectives unstable.
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
Modern deep learning methods provide effective means to learn good representations. However, is a good representation itself sufficient for sample efficient reinforcement learning? This question has largely been studied only with respect to (worst-case) approximation error, in the more classical approximate dynamic pro…
Lookahead pruning extends single-layer optimization to multi-layer, outperforming magnitude-based pruning.
problem Pruning neural networks to reduce computational cost and memory usage.
method Developed a multi-layer optimization approach extending the single-layer optimization of magnitude-based pruning.
result Consistently outperforms magnitude-based pruning on various networks, especially in high sparsity.
The basic financial purpose of a firm is to maximize its value. An inventory management system should also contribute to realization of this basic aim. Many current asset management models currently found in financial management literature were constructed with the assumption of book profit maximization as basic aim. H…
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
ICE-Pruning accelerates deep neural network pruning by 9.61x.
problem Efficiently pruning deep neural networks while maintaining accuracy.
method Iterative pruning with automatic fine-tuning steps, freezing strategy, and custom learning rate scheduler.
result Significantly reduces pruning time by up to 9.61x.
DSA efficiently allocates sparsity across layers for budgeted pruning.
problem Efficiently distributing resources (sparsity) across layers in pruning under resource constraints.
method DSA uses differentiable pruning to find continuous layer-wise pruning ratios via gradient-based optimization.
result DSA achieves superior performance and significantly reduces the time cost of pruning.
This paper improves reinforcement learning policies in a scalable way.
problem Ensuring monotonic policy improvement in entropy-regularized RL.
method Derives an entropy-aware lower bound and proposes a novel RL algorithm.
result Demonstrates effectiveness in continuous-state tasks using a linear function approximator.
Pruning neural network parameters is often viewed as a means to compress models, but pruning has also been motivated by the desire to prevent overfitting. This motivation is particularly relevant given the perhaps surprising observation that a wide variety of pruning approaches increase test accuracy despite sometimes …
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.
problem Flaws in existing pruning criteria for CNNs.
method Empirical experiments and Convolutional Weight Distribution Assumption.
result The Convolutional Weight Distribution Assumption improves filter selection in pruning.
Deep neural networks have dramatically achieved great success on a variety of challenging tasks. However, most successful DNNs have an extremely complex structure, leading to extensive research on model compression.As a significant area of progress in model compression, traditional gradual pruning approaches involve an…
Gibbs pruning optimizes neural networks by combining physics and regularization.
problem Large neural networks are impractical for many applications.
method Combines statistical physics and stochastic regularization to train and prune networks simultaneously.
result Gibbs pruning achieves state-of-the-art performance on ResNet-56.
We examine how recently documented, fundamental phenomena in deep learning models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through a set of common iterative pruning techniques, to …
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
This paper introduces blind adversarial pruning to balance accuracy, efficiency, and robustness in neural networks.
problem Balancing accuracy, efficiency, and robustness in neural networks with limited resources.
method Adversarial pruning with a cutoff-scale strategy to dynamically adjust the strength of adversarial examples.
result Blind adversarial pruning improves the overall AER of pruned models compared to adversarial pruning.
Speeds up training and inference by pruning entire channels before training.
problem Training and inference speed in deep neural networks.
method Structured pruning applied before training, focusing on removing entire channels and hidden units.
result 2x speedup in training and 3x speedup in inference.
Proposes a value-based method for continuous control without an actor.
problem Computational infeasibility of evaluating Q-values in continuous action spaces.
method Structurally maximizable Q-functions, actor-free approach.
result Performance and sample efficiency comparable to actor-critic methods.
This paper analyzes privacy risks in neural network pruning and proposes a defense mechanism.
problem Privacy risks in neural network pruning due to membership inference attacks.
method Investigates the impact of pruning on prediction divergence and proposes a self-attention membership inference attack.
result Proposed defense mechanism mitigates privacy risks while maintaining sparsity and accuracy.
Neural network pruning lacks standardized benchmarks and metrics.
problem Lack of standardized benchmarks and metrics in neural network pruning.
method Meta-analysis of 81 papers, controlled conditions, ShrinkBench framework.
result Neural network pruning community lacks standardized benchmarks and metrics.
Bayesian inference improves neural network pruning efficiency.
problem Reducing computational and memory demands of large neural networks.
method Utilizes Bayesian inference to calculate Bayes factors for iterative pruning.
result Achieves desired levels of sparsity while maintaining competitive accuracy.
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
problem Improving pruning of large language models to reduce size without sacrificing performance.
method AlphaPruning uses HT-SR theory to allocate layerwise sparsity ratios more theoretically.
result AlphaPruning prunes LLaMA-7B to 80% sparsity with reasonable perplexity.
This work characterizes the fundamental limit of network pruning using statistical dimension and convex geometry.
problem The fundamental limit of network pruning is still lacking, especially for deep neural networks.
method Directly imposing sparsity constraint on the loss function and using statistical dimension in convex geometry.
result Characterizes the sharp phase transition point as the fundamental limit of pruning ratio.
A new framework explains why early pruning works well.
problem Understanding why early pruning of neural networks leads to good performance.
method Gradient flow framework to unify pruning measures.
result Magnitude-based pruning removes least contributing parameters, leading to faster convergence.
CupNet prunes neural nets for cup-shaped data.
problem Pruning neural networks for cup-shaped data.
method Used simulated cup drawing data to prune a neural network.
result Pruning effectively reduces network size for cup-shaped data.
This work explores the importance of model weights and Hessian bias in pruning.
problem Understanding the relative importance of model weights for efficient pruning.
method A principled exploration of pruning, focusing on linear models and neural networks.
result Asymptotic formulas reveal the performance of different pruning methods.
A method to combine saliency metrics for better CNN pruning decisions.
problem Improving CNN pruning decisions by combining multiple saliency metrics.
method Proposes a method to compose different saliency metrics for better CNN pruning decisions.
result The composition of saliencies avoids many poor pruning choices identified by individual saliencies.
Pruning improves model generalization in over-parameterized models, contradicting traditional theories.
problem Pruning's effect on generalization in over-parameterized models.
method Empirical study on standard pruning algorithms and additional regularization effects.
result Pruning leads to better training and regularization, improving generalization.
Hyperflux models pruning as a system to reveal weight importance.
problem Pruning large neural networks to reduce latency and power consumption.
method Introduces Hyperflux, a novel L0 method that models pruning as flux and pressure. result Achieves competitive results with ResNet-50, VGG-19, and DeiT-T/S on various datasets.
Value functions struggle to represent transition dynamics, impacting statistical efficiency.
problem Limited representational power of value functions in capturing transition dynamics.
method Case studies of various reinforcement learning problems to explore the limitations of value-based methods.
result Value-based methods can be as efficient as model-based ones in some cases but severely underperform in others due to information loss.
This paper studies a theoretical pruning method for RNNs to reduce computational costs.
problem High computational costs in recurrent neural networks (RNNs).
method Spectral pruning inspired approach for RNNs.
result Generalization error bounds for compressed RNNs are provided.
Neural network for subgraph similarity computation with pruning.
problem Computing subgraph similarity between a target and query graph.
method Convert pruning to node relabeling, relax to differentiable problem, design neural network for SED computation.
result Establishes new state-of-the-art results across multiple benchmark datasets.
Network pruning is a promising avenue for compressing deep neural networks. A typical approach to pruning starts by training a model and then removing redundant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done at initialization prior to training,…
Data pruning algorithms struggle in high compression regimes, as shown by theoretical and empirical studies.
problem Limitations of score-based data pruning algorithms in high compression regimes.
method Theoretical and empirical analysis of score-based data pruning algorithms.
result Score-based data pruning algorithms fail in high compression regimes due to 'No Free Lunch' theorems.
A new method prunes neural networks faster and more efficiently.
problem Reducing training time and memory usage for neural networks.
method Set-based Task-Adaptive Meta Pruning (STAMP) that meta-learns a pruning mask.
result Significantly improved compression rates and faster training speed.
Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work have been conducted on DNN model compression or pruning. However, most of the previous work took heuristic approac…
PruneNet efficiently prunes channels in deep networks, improving accuracy and performance.
problem Improving deep neural network performance and efficiency through channel pruning.
method PruneNet uses a computationally light-weight optimization step to identify and prune channels based on layer redundancy.
result Pruned ResNet models achieve higher accuracy and better performance than non-pruned models.
Pruning method removes less important features in linear models.
problem Removing less important features in linear models trained by gradient flow.
method Iterative Magnitude Pruning (IMP) applied to linear models trained by gradient flow.
result IMP prunes features with smallest projection onto the data.
In this paper, we propose a novel progressive parameter pruning method for Convolutional Neural Network acceleration, named Structured Probabilistic Pruning (SPP), which effectively prunes weights of convolutional layers in a probabilistic manner. Unlike existing deterministic pruning approaches, where unimportant weig…
Randomly initialized networks can perform as well as pruned networks.
problem Understanding and improving network pruning methods.
method Sanity checks on recent pruning methods, proposing random tickets.
result Randomly initialized networks can perform as well as pruned networks.
Structured pruning is a popular method for compressing a neural network: given a large trained network, one alternates between removing channel connections and fine-tuning; reducing the overall width of the network. However, the efficacy of structured pruning has largely evaded scrutiny. In this paper, we examine ResNe…
A model predicts how hyperparameters affect pruning performance.
problem Predicting the impact of hyperparameters on pruning performance.
method Phenomenological model using temperature-like and load-like parameters.
result A sharp transition phenomenon in pruning performance.
i-SpaSP prunes neural networks by identifying important groups of parameters, improving pruning efficiency.
problem Pruning neural networks to reduce computational cost and improve performance.
method i-SpaSP uses sparse signal recovery principles to iteratively identify and threshold important parameter groups.
result i-SpaSP achieves strong empirical results and theoretical convergence guarantees, improving pruning efficiency.
Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weigh…
New pruning methods improve energy efficiency of neural networks.
problem Energy-efficient neural networks for devices with limited resources.
method Magnitude and Gradient based pruning at initialization and training of sparse architectures.
result Proposed novel pruning methods prevent full layer pruning and improve training.