Pruning large weights improves ANN accuracy without increasing generalization error.
problem Reduction of ANN weights without sacrificing performance and generalization.
method Targeting large weights for pruning in ANNs.
result Pruning large weights leads to higher image classification accuracy and reduced parameter count.
Paper proposes efficient network pruning method for deep neural networks.
problem High computational and memory cost of deep neural networks.
method Annealing and direct sparsity control for channel-level pruning.
result Proposed method achieves better or competitive performance compared to other methods.
Mixed integer programming identifies critical neurons in neural networks.
problem Identifying neurons critical for network performance and generalization.
method Developed a mixed integer program (MIP) to assign importance scores to neurons, guiding pruning decisions.
result The method identifies multiple 'lucky' sub-networks resulting in optimized architectures that generalize across datasets.
Paper simplifies ANN structure into a functional form.
problem Current ANN structure is complex and difficult to analyze.
method Uses activation integral concept to represent ANN structure as a function.
result Simplified mathematical representation of ANN structure.
Improves ANN performance by normalizing data with PCA and eigenvalue weighting.
problem Boosting ANN performance through data preprocessing.
method PCA followed by weighting principle components by eigenvalues.
result Significantly improves ANN performance in classification tasks.
Paper tackles adversarial attacks on ANN state estimation in smart grids.
problem Adversarial attacks degrade ANN state estimation accuracy without detection.
method Proposes population-based and gradient-based algorithms to generate attack vectors.
result DE algorithm is more effective than SLSQP in generating attack vectors.
Book introduces deep learning methods with math, theory, and applications.
problem Understanding deep learning algorithms and their mathematical foundations.
method Reviews various ANN architectures and optimization methods, covers theoretical aspects.
result Provides a solid mathematical foundation for deep learning.
Transform ANNs into interpretable decision trees.
problem Lack of interpretability in ANNs.
method Developed two MDT algorithms: EC-DT and Extended C-Net.
result Extended C-Net generates the most compact and effective trees.
Study proposes hybrid machine learning models for crop yield prediction.
problem Crop yield prediction for food security.
method Hybrid machine learning models (ANN-ICA and ANN-GWO).
result ANN-GWO model outperformed ANN-ICA in crop yield prediction.
Study compares ANN and GARCH models for volatility prediction across sectors.
problem Comparing ANN and GARCH models for volatility prediction.
method Examined five sectors with low, medium, and high volatility, using three GARCH specifications and three ANN architectures.
result ANN model performs better for low volatility, GARCH for medium and high.
Study examines adversarial robustness of ANN variants, revealing differences in black-box settings.
problem Adversarial robustness of alternative neural network architectures.
method Analysis of conventional, stochastic ANNs, and SNNs across three datasets; experiments in white-box and black-box settings.
result Stochastic ANNs are more robust than conventional ANNs in black-box settings, especially with surrogate attacks.
New method converts ANN gates to SNNs with AMOS neurons for improved image classification.
problem Efficiently converting ANN gates to SNNs for neuromorphic hardware.
method Introducing AMOS conversion for gates in ANNs, improving accuracy and throughput.
result Improved accuracy of SNNs for ImageNet from 74.60% to 80.97%.
Study helps identify which ANN parameters cause forgetting.
problem Catastrophic forgetting in neural networks.
method Determines individual parameter contributions to forgetting.
result Identifies specific ANN parameters causing forgetting.
GPEX interprets deep neural networks without strict assumptions.
problem Interpreting deep neural networks without strict assumptions.
method Derive an evidence lower-bound that encourages GP's posterior to match ANN's output.
result GPs can closely match ANN's outputs without strict assumptions.
Systems based on artificial neural networks (ANNs) have achieved state-of-the-art results in many natural language processing tasks. Although ANNs do not require manually engineered features, ANNs have many hyperparameters to be optimized. The choice of hyperparameters significantly impacts models' performances. Howeve…
Lookahead pruning extends single-layer optimization to multi-layer, outperforming magnitude-based pruning.
problem Pruning neural networks to reduce computational cost and memory usage.
method Developed a multi-layer optimization approach extending the single-layer optimization of magnitude-based pruning.
result Consistently outperforms magnitude-based pruning on various networks, especially in high sparsity.
ANNs efficiently approximate high-dimensional Black-Scholes PDEs without the curse of dimensionality.
problem Efficiently approximating high-dimensional Black-Scholes PDEs.
method Rigorous mathematical analysis of ANN approximations of Black-Scholes PDEs.
result ANNs can approximate the solution of the Black-Scholes PDE with polynomial growth in parameters relative to accuracy and dimension.
Improved ANN-based Monte Carlo simulation for Higgs decay events.
problem Accurate simulation of Higgs boson decay events.
method Monte Carlo simulation using an Artificial Neural Network (ANN) with improved training algorithm.
result The ANN simulation of Higgs decay is within 0.7% of the true value and achieves 26% unweighting efficiency.
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
ICE-Pruning accelerates deep neural network pruning by 9.61x.
problem Efficiently pruning deep neural networks while maintaining accuracy.
method Iterative pruning with automatic fine-tuning steps, freezing strategy, and custom learning rate scheduler.
result Significantly reduces pruning time by up to 9.61x.
ANNs solve financial option valuation problems without numerical methods.
problem Valuation of European and American financial options.
method Unsupervised learning with artificial neural networks (ANNs) for solving PDEs.
result ANNs accurately compute option values for various stock scenarios.
DSA efficiently allocates sparsity across layers for budgeted pruning.
problem Efficiently distributing resources (sparsity) across layers in pruning under resource constraints.
method DSA uses differentiable pruning to find continuous layer-wise pruning ratios via gradient-based optimization.
result DSA achieves superior performance and significantly reduces the time cost of pruning.
New ADANNs improve PDE approximations.
problem Approximating operators for parametric PDEs.
method Custom ANN architectures and initialization schemes.
result ADANNs significantly outperform existing methods.
ANN improves option pricing models by calibrating parameters faster and more accurately.
problem Calibration of GARCH-type option pricing models is computationally intensive and model-dependent.
method Trained ANN models on Monte Carlo simulation data to calibrate GARCH parameters.
result ANN outperforms traditional methods in calibration speed and accuracy.
Study examines effects of pruning techniques on deep learning models.
problem Understanding the impact of pruning methods on deep learning model structure and dynamics.
method Investigated differences in connectivity and learning dynamics of pruned models using various iterative pruning techniques.
result Emergence of structure in pruned models through magnitude-based unstructured pruning and weight rewinding.
Named-entity recognition (NER) aims at identifying entities of interest in a text. Artificial neural networks (ANNs) have recently been shown to outperform existing NER systems. However, ANNs remain challenging to use for non-expert users. In this paper, we present NeuroNER, an easy-to-use named-entity recognition tool…
Pruning neural networks can improve test accuracy even with significant parameter reduction.
problem The tradeoff between generalization and stability in neural network pruning.
method Analysis of pruning behavior over training, focusing on instability and its relation to generalization.
result Pruning's benefit to generalization increases with its instability.
Drop Pruning uses stochastic optimization to prune and recover weights, reducing model size and improving performance.
problem Complexity and inefficiency in pruning deep neural networks.
method Introduces stochastic optimization with 'drop away' and 'drop back' strategies to prune and recover weights.
result Achieves competitive compression performance and accuracy compared to state-of-the-art approaches.
New method converts conventional ANNs to SNNs with minimal loss and efficiency.
problem Difficulty in training SNNs directly from conventional ANNs due to discreteness.
method Proposes a novel pipeline combining threshold balance and soft-reset mechanisms for efficient conversion.
result Achieves almost no accuracy loss with only 1/10 of typical SNN simulation time.
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.
problem Flaws in existing pruning criteria for CNNs.
method Empirical experiments and Convolutional Weight Distribution Assumption.
result The Convolutional Weight Distribution Assumption improves filter selection in pruning.
Efficient ANN search for sparse embeddings in ads targeting.
problem Efficiently searching near neighbors in sparse data for applications like ads targeting.
method Graph-based ANN algorithms (HNSW, chi-square two-tower model, Sign Cauchy Projections).
result Sparse embeddings and ANN algorithms improve efficiency in EBR applications.
SBAF activation function improves ANN for exoplanet habitability classification.
problem Classifying exoplanets into habitable and non-habitable categories.
method Developed Saha-Bora Activation Function (SBAF) for ANN, demonstrating its analytical properties and improved performance.
result SBAF activation function outperforms traditional functions in ANN for exoplanet habitability classification.
The paper uses neural networks to speed up option pricing and implied volatility calculations.
problem Speeding up the computation of financial option prices and implied volatilities.
method Training an Artificial Neural Network (ANN) on a financial model to approximate and accelerate the calculation of option prices and implied volatilities.
result The ANN solver significantly reduces computing time compared to traditional methods.
ELNN uses neural networks for improved option pricing.
problem Inconsistent pricing of over-the-counter products and unacceptable outcomes in ANN-based models.
method ELNN integrates ANNs with the exponential Levy model, addressing issues with existing models.
result ELNN outperforms Merton and Kou models in fitting performance and stability of estimates.
Gibbs pruning optimizes neural networks by combining physics and regularization.
problem Large neural networks are impractical for many applications.
method Combines statistical physics and stochastic regularization to train and prune networks simultaneously.
result Gibbs pruning achieves state-of-the-art performance on ResNet-56.
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
New method prunes neural networks at initialization, improving performance.
problem Improving neural network compression at initialization.
method Formally characterizes initialization conditions for reliable pruning based on connection sensitivity.
result Improved neural network performance on image classification tasks.
This paper introduces blind adversarial pruning to balance accuracy, efficiency, and robustness in neural networks.
problem Balancing accuracy, efficiency, and robustness in neural networks with limited resources.
method Adversarial pruning with a cutoff-scale strategy to dynamically adjust the strength of adversarial examples.
result Blind adversarial pruning improves the overall AER of pruned models compared to adversarial pruning.
A fast pruning algorithm for DNNs with GE guarantees.
problem Efficiently pruning DNNs without sacrificing accuracy.
method FeTa algorithm based on DC optimization, with GE analysis.
result FeTa is orders of magnitude faster and maintains GE.
Speeds up training and inference by pruning entire channels before training.
problem Training and inference speed in deep neural networks.
method Structured pruning applied before training, focusing on removing entire channels and hidden units.
result 2x speedup in training and 3x speedup in inference.
This paper analyzes privacy risks in neural network pruning and proposes a defense mechanism.
problem Privacy risks in neural network pruning due to membership inference attacks.
method Investigates the impact of pruning on prediction divergence and proposes a self-attention membership inference attack.
result Proposed defense mechanism mitigates privacy risks while maintaining sparsity and accuracy.
Add expert knowledge to resolve ambiguities in ANN models.
problem Non-unique parameter fitting in material science.
method Augment a black-box ANN model with expert knowledge at two levels.
result Expert knowledge resolves ambiguities in parameter space.
SPP prunes CNN weights probabilistically for faster inference.
problem Efficiently accelerate Convolutional Neural Networks (CNNs) without significant accuracy loss.
method Structured Probabilistic Pruning (SPP) with adjustable pruning probabilities.
result 4x speedup with minimal accuracy loss (0.3% for AlexNet, 0.8% for VGG-16).
Neural network pruning lacks standardized benchmarks and metrics.
problem Lack of standardized benchmarks and metrics in neural network pruning.
method Meta-analysis of 81 papers, controlled conditions, ShrinkBench framework.
result Neural network pruning community lacks standardized benchmarks and metrics.
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
problem Improving pruning of large language models to reduce size without sacrificing performance.
method AlphaPruning uses HT-SR theory to allocate layerwise sparsity ratios more theoretically.
result AlphaPruning prunes LLaMA-7B to 80% sparsity with reasonable perplexity.
Bayesian inference improves neural network pruning efficiency.
problem Reducing computational and memory demands of large neural networks.
method Utilizes Bayesian inference to calculate Bayes factors for iterative pruning.
result Achieves desired levels of sparsity while maintaining competitive accuracy.
This work characterizes the fundamental limit of network pruning using statistical dimension and convex geometry.
problem The fundamental limit of network pruning is still lacking, especially for deep neural networks.
method Directly imposing sparsity constraint on the loss function and using statistical dimension in convex geometry.
result Characterizes the sharp phase transition point as the fundamental limit of pruning ratio.