Dynamic sample pruning speeds up spatio-temporal forecasting models.
problem Training deep learning models on large, redundant datasets is computationally expensive.
method Dynamic sample pruning based on real-time learning state.
result Significant acceleration of training speed with improved performance.
PAC-MCTS addresses biased search in LLM-guided planning by dynamically pruning.
problem Systematic biases in LLMs lead to inefficient and unsafe search in deep planning tasks.
method Formulates node expansion as BAI under bounded bias, derives sample complexity bounds, and proposes PAC-MCTS for dynamic confidence bounds.
result PAC-MCTS improves robustness and efficiency by up to 78% fewer API evaluations and 3x higher sample efficiency.
NTK-SAP improves neural network pruning by aligning training dynamics.
problem Improving neural network pruning to reduce training time and memory.
method Prune connections based on the spectrum of the Neural Tangent Kernel (NTK), using multiple random weight realizations and random inputs.
result Empirically, NTK-SAP achieves better performance than all baselines on multiple datasets.
New method grows deep networks efficiently by dynamically pruning and growing layers.
problem Training deep networks is computationally expensive and inefficient.
method Structured continuous sparsification starting from a small seed architecture.
result 49.7% inference FLOPs and 47.4% training FLOPs savings with 75.2% top-1 accuracy.
Dynamic Vocabulary Pruning stabilizes LLM training by removing low-probability tokens.
problem Training Large Language Models (LLMs) with Reinforcement Learning (RL) causes numerical divergence between inference and training.
method Dynamic Vocabulary Pruning (DVP) constrains the RL objective to a safe vocabulary that excludes low-probability tokens.
result DVP stabilizes training by reducing systematic bias introduced by the extreme tail of the token distribution.
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
New pruning methods improve dynamic sparse training performance.
problem Improving dynamic sparse training performance.
method Design and empirical analysis of pruning criteria.
result Most pruning methods yield similar results, but magnitude-based pruning performs best in low-density regimes.
We introduce Delay Pruning, a simple yet powerful technique to regularize dynamic Boltzmann machines (DyBM). The recently introduced DyBM provides a particularly structured Boltzmann machine, as a generative model of a multi-dimensional time-series. This Boltzmann machine can have infinitely many layers of units but al…
Proposes dynamic channel pruning during neural network training.
problem Pruning neural networks during training to reduce computational cost and improve efficiency.
method Dynamic channel propagation to update channel utility values and selectively prune channels.
result Our scheme trains and prunes neural networks simultaneously, achieving superior performance.
A new framework explains why early pruning works well.
problem Understanding why early pruning of neural networks leads to good performance.
method Gradient flow framework to unify pruning measures.
result Magnitude-based pruning removes least contributing parameters, leading to faster convergence.
We examine how recently documented, fundamental phenomena in deep learning models subject to pruning are affected by changes in the pruning procedure. Specifically, we analyze differences in the connectivity structure and learning dynamics of pruned models found through a set of common iterative pruning techniques, to …
Dynamic Sparse Training finds efficient sparse networks from scratch.
problem Finding efficient sparse neural networks.
method Jointly optimizes network parameters and sparsity with trainable thresholds.
result Achieves state-of-the-art performance with minimal performance loss.
Dynamic model pruning improves performance on deep neural networks without retraining.
problem High memory and latency requirements for deep neural networks on low-end devices.
method Dynamic allocation of sparsity pattern and feedback signal to reactivate pruned weights.
result Sparse models achieve state-of-the-art performance with no additional retraining.
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
To deal with various datasets over different complexity, this paper presents an self-adaptive learning model that combines the proposed Dynamic Connected Neural Decision Networks (DNDN) and a new pruning method--Dynamic Soft Pruning (DSP). DNDN is a combination of random forests and deep neural networks that enjoys bot…
Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work have been conducted on DNN model compression or pruning. However, most of the previous work took heuristic approac…
Geometric pruning rules improve change point detection in multiple time series.
problem Detecting multiple changes in multiple independent time series.
method Dynamic programming algorithms with inequality-based and geometric pruning rules.
result Geometric pruning rules offer close-to-linear time complexity for multiple independent time series.
ES improves training efficiency by dynamically selecting data samples.
problem Efficiently selecting informative data samples for faster learning.
method Evolved Sampling (ES) dynamically selects data samples based on loss dynamics and differences.
result ES achieves significant training acceleration without compromising model performance.
Stochastic Gradient TreeBoost is often found in many winning solutions in public data science challenges. Unfortunately, the best performance requires extensive parameter tuning and can be prone to overfitting. We propose PaloBoost, a Stochastic Gradient TreeBoost model that uses novel regularization techniques to guar…
This paper introduces blind adversarial pruning to balance accuracy, efficiency, and robustness in neural networks.
problem Balancing accuracy, efficiency, and robustness in neural networks with limited resources.
method Adversarial pruning with a cutoff-scale strategy to dynamically adjust the strength of adversarial examples.
result Blind adversarial pruning improves the overall AER of pruned models compared to adversarial pruning.
In this paper, we propose a novel progressive parameter pruning method for Convolutional Neural Network acceleration, named Structured Probabilistic Pruning (SPP), which effectively prunes weights of convolutional layers in a probabilistic manner. Unlike existing deterministic pruning approaches, where unimportant weig…
Data pruning algorithms struggle in high compression regimes, as shown by theoretical and empirical studies.
problem Limitations of score-based data pruning algorithms in high compression regimes.
method Theoretical and empirical analysis of score-based data pruning algorithms.
result Score-based data pruning algorithms fail in high compression regimes due to 'No Free Lunch' theorems.
PARIS reduces imbalanced regression datasets by pruning uninformative samples.
problem Imbalanced regression where models focus on high-frequency regions, ignoring rare but impactful events.
method PARIS uses the representer theorem to compute a closed-form representer deletion residual for iterative pruning of the training set.
result PARIS reduces training set by up to 75% while preserving or improving overall performance, outperforming other methods.
Accelerating DNN execution on various resource-limited computing platforms has been a long-standing problem. Prior works utilize l1-based group lasso or dynamic regularization such as ADMM to perform structured pruning on DNN models to leverage the parallel computing architectures. However, both of the pruning dimensio…
Random Forests automatically prune a latent 'true' tree, explaining their overfitting without tuning.
problem Difficulty in building bad Random Forests and overfitting without apparent consequences.
method Bootstrap aggregation and model perturbation in Random Forests.
result Randomized ensembles implicitly perform optimal early stopping out-of-sample, explaining overfitting.
Optimizes pruning masks for neural networks using probabilistic fine-tuning and PAC-Bayes bounds.
problem Improving neural network performance through adaptive pruning of weights.
method Optimizes stochastic pruning masks by minimizing expected loss, considering data-adaptive regularization and feature alignment.
result Probabilistic fine-tuning leads to improved test error over baseline methods in neural networks.
DASH simplifies neural networks for gene regulatory dynamics using domain knowledge.
problem Pruning neural networks for gene regulatory dynamics lacks biologically meaningful structure learning.
method DASH uses domain-specific structural information to guide network pruning, leading to sparser, better interpretable models.
result DASH outperforms general pruning methods in gene regulatory network inference, yielding deeper insights.
Pruning neural networks improves feature learning in high-dimensional models.
problem Improving feature quality in high-dimensional statistical models.
method Demonstrated that pruning neural networks can optimally learn sparse models using gradient descent.
result Pruning neural networks proportional to the sparsity level of the model directions improves sample complexity.
This paper analyzes privacy risks in neural network pruning and proposes a defense mechanism.
problem Privacy risks in neural network pruning due to membership inference attacks.
method Investigates the impact of pruning on prediction divergence and proposes a self-attention membership inference attack.
result Proposed defense mechanism mitigates privacy risks while maintaining sparsity and accuracy.
Optimization-based pruning eliminates backpropagation for large language models.
problem Suboptimal pruning performance due to heuristic metrics.
method Optimization of Bernoulli distribution to learn pruning masks without backpropagation.
result Efficient pruning of large language models with improved performance.
We present a provable, sampling-based approach for generating compact Convolutional Neural Networks (CNNs) by identifying and removing redundant filters from an over-parameterized network. Our algorithm uses a small batch of input data points to assign a saliency score to each filter and constructs an importance sampli…
We introduce a pruning algorithm that provably sparsifies the parameters of a trained model in a way that approximately preserves the model's predictive accuracy. Our algorithm uses a small batch of input points to construct a data-informed importance sampling distribution over the network's parameters, and adaptively …
To improve the execution speed and efficiency of neural networks in embedded systems, it is crucial to decrease the model size and computational complexity. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can reduce the amount of data communica…
Random forests perform bootstrap-aggregation by sampling the training samples with replacement. This enables the evaluation of out-of-bag error which serves as a internal cross-validation mechanism. Our motivation lies in using the unsampled training samples to improve each decision tree in the ensemble. We study the e…
Pruning + fine-tuning reduces model complexity and generalizes well for matrix sensing.
problem Reducing model complexity for matrix sensing problems.
method Group Lasso regularization and greedy pruning.
result Pruning results in a solution with minimum columns close to the ground truth.
New test improves tree ensemble pruning for better model performance.
problem Lack of robust theoretical justification for penalty terms in tree ensembles.
method Developed a novel hypothesis test for tree ensemble split quality.
result Significant reduction in out-of-sample loss using the new test.
Survey on reducing deep neural network training complexity.
problem Efficient training of large deep neural networks.
method Pruning and freezing parts of the network during training.
result Dimensionality reduction improves training efficiency.
OPNP prunes parameters and neurons to improve OOD detection without training.
problem Detecting out-of-distribution samples in real-world machine learning models.
method OPNP approach that identifies and removes sensitive parameters and neurons.
result OPNP consistently outperforms existing methods on multiple OOD detection tasks.
As a result of the growing size of Deep Neural Networks (DNNs), the gap to hardware capabilities in terms of memory and compute increases. To effectively compress DNNs, quantization and connection pruning are usually considered. However, unconstrained pruning usually leads to unstructured parallelism, which maps poorly…
Data-independent pruning method reduces neural network size with accuracy guarantees.
problem Limited computational and memory resources for neural networks.
method Structured pruning using coresets.
result First efficient algorithm with worst-case guarantees on compression and accuracy.
Sparsification is an efficient approach to accelerate CNN inference, but it is challenging to take advantage of sparsity in training procedure because the involved gradients are dynamically changed. Actually, an important observation shows that most of the activation gradients in back-propagation are very close to zero…
This paper improves forest pruning to balance accuracy and interpretability.
problem Limited interpretability of regression forests.
method Lasso-pruning and theoretical analysis of regression forests.
result Pruned regression forests can achieve equal or better accuracy than unpruned ones, with significant size reduction.
TENP prunes experts and neurons in Mixture-of-Experts models for efficient deployment.
problem Efficient deployment of large language models constrained by static parameter footprint.
method Structured Trapezoidal ExpertNeuron Pruning (TENP) identifies and retains important experts and neurons.
result DeepSeek model achieves 10% better performance on code generation tasks with 40% expert sparsity.
BWS selects best window subsets for efficient data pruning.
problem Challenges in selecting subsets of large datasets for neural network training.
method Best Window Selection (BWS) by choosing optimal window intervals from ordered sample scores.
result BWS outperforms other methods across various selection ratios and datasets.
We propose a new random pruning method (called "submodular sparsification (SS)") to reduce the cost of submodular maximization. The pruning is applied via a "submodularity graph" over the n ground elements, where each directed edge is associated with a pairwise dependency defined by the submodular function. In each s…
A plethora of recent research has focused on improving the memory footprint and inference speed of deep networks by reducing the complexity of (i) numerical representations (for example, by deterministic or stochastic quantization) and (ii) arithmetic operations (for example, by binarization of weights). We propose a s…
For a density f on Rd, a {\it high-density cluster} is any connected component of {x:f(x)≥λ}, for some λ>0. The set of all high-density clusters forms a hierarchy called the {\it cluster tree} of f. We present two procedures for estimating the cluster tree given samples from f. The first…
ICE-Pruning accelerates deep neural network pruning by 9.61x.
problem Efficiently pruning deep neural networks while maintaining accuracy.
method Iterative pruning with automatic fine-tuning steps, freezing strategy, and custom learning rate scheduler.
result Significantly reduces pruning time by up to 9.61x.