Synaptic pruning reduces CNNs by 96% on CIFAR-10.
problem Memory and computation constraints in CNNs for mobile devices.
method Synaptic Pruning: data-driven method to prune connections based on Synaptic Strength.
result Significant size reduction and computation saving with up to 96% pruning on CIFAR-10.
ANPyC combats forgetting by pruning and consolidating neural parameters.
problem Catastrophic forgetting in neural networks, especially with long-term tasks.
method Adversarial Neural Pruning and Synaptic Consolidation (ANPyC) to balance task-relevant and irrelevant parameters.
result ANPyC prevents forgetting while enabling efficient learning of multiple tasks.
New algorithm finds important synapses without training data.
problem Finding important synapses in neural networks without data.
method Iterative Synaptic Flow Pruning (SynFlow) based on conservation law.
result Algorithm consistently outperforms existing pruning methods.
We introduce Delay Pruning, a simple yet powerful technique to regularize dynamic Boltzmann machines (DyBM). The recently introduced DyBM provides a particularly structured Boltzmann machine, as a generative model of a multi-dimensional time-series. This Boltzmann machine can have infinitely many layers of units but al…
Optimizes neural networks for solving problems with pruning and ensembles of minimal structures.
problem Improving neural network performance and interpretability.
method Pruning neural networks based on the principle of controlling training and pruning, using sensitivity indicators and logically transparent NN.
result Ensemble of minimal neural networks provides diverse forecasting algorithms and identifies areas for further data collection.
Brain uses synaptic failure to sample from posterior distributions.
problem Bayesian inference in the brain's probabilistic computations.
method Adapting synaptic failure to sample posterior predictive distributions.
result Synaptic failure enables sampling of complete posterior predictive distributions.
Synaptic strength can be seen as probability to propagate impulse, and according to synaptic plasticity, function could exist from propagation activity to synaptic strength. If the function satisfies constraints such as continuity and monotonicity, neural network under external stimulus will always go to fixed point, a…
The study reveals how synaptic correlations promote dimension reduction in neural networks.
problem Understanding how synaptic correlations affect neural correlations and dimension reduction in deep neural networks.
method A simplified model of dimension reduction considering pairwise correlations among synapses, using mathematical self-consistency for both binary and continuous synapses.
result Weakly-correlated synapses encourage dimension reduction compared to orthogonal synapses, and they also slow down the decorrelation process.
Locally learned synaptic failure enables complete Bayesian inference.
problem Bayesian inference in neural networks.
method Biologically constrained neural network with synaptic failure and lateral inhibition.
result Synaptic failure allows sampling from both epistemic and aleatoric distributions.
New synaptic model derived from reinforcement learning for spiking neurons.
problem Learning optimal actions in complex systems.
method Derives a synaptic update rule from reinforcement learning algorithms.
result Synaptic strengths lead to locally optimal reward values.
Learning and memory in the brain are implemented by complex, time-varying changes in neural circuitry. The computational rules according to which synaptic weights change over time are the subject of much research, and are not precisely understood. Until recently, limitations in experimental methods have made it challen…
Natural gradient learning improves synaptic plasticity in spiking neurons.
problem Parametrization dependence leads to inconsistencies in classical synaptic plasticity theories.
method Proposes natural gradient descent in Riemannian geometry for spiking neurons.
result Derives a synaptic learning rule that explains biological phenomena.
Expanding neural networks improves their learning from noisy data.
problem Improving neural network performance in noisy conditions.
method Sparse expansion of neural network inputs, followed by pruning, enhances generalization.
result Sparse expansion of neural networks improves generalization performance, even after pruning.
There has been significant recent interest towards achieving highly efficient deep neural network architectures. A promising paradigm for achieving this is the concept of evolutionary deep intelligence, which attempts to mimic biological evolution processes to synthesize highly-efficient deep neural networks over succe…
A promising paradigm for achieving highly efficient deep neural networks is the idea of evolutionary deep intelligence, which mimics biological evolution processes to progressively synthesize more efficient networks. A crucial design factor in evolutionary deep intelligence is the genetic encoding scheme used to simula…
Enhances SNNs for spatio-temporal feature extraction.
problem Insufficient temporal dependencies in existing SNN synaptic structures.
method Integrates temporal convolution and attention mechanisms into synaptic connections.
result Improves SNN performance on classification tasks.
Learning in neural networks poses peculiar challenges when using discretized rather then continuous synaptic states. The choice of discrete synapses is motivated by biological reasoning and experiments, and possibly by hardware implementation considerations as well. In this paper we extend a previous large deviations a…
New model reveals balance crucial for robust neural coding.
problem Efficient neural coding in noisy, chaotic networks.
method Analytical model of balanced predictive coding with dissociated balance and weight disorder.
result Superclassical scaling in coding accuracy, independent of balance and weight disorder.
In this work, we perform an exploratory study on synthesizing deep neural networks using biological synaptic strength distributions, and the potential influence of different distributions on modelling performance particularly for the scenario associated with small data sets. Surprisingly, a CNN with convolutional layer…
Proposes a method to prevent neural networks from forgetting learned tasks.
problem Catastrophic forgetting in neural networks.
method Attention-based selective plasticity of synapses inspired by the cholinergic neuromodulatory system.
result Competitive performance on benchmark tasks compared to state-of-the-art methods.
A new neural model evolves to learn at the synaptic level.
problem Lack of biologically realistic neural models in deep learning.
method Evolve individual neuron and synaptic models using ENUs.
result Evolved neural network learns complex tasks like a T-maze.
A neuron is a basic physiological and computational unit of the brain. While much is known about the physiological properties of a neuron, its computational role is poorly understood. Here we propose to view a neuron as a signal processing device that represents the incoming streaming data matrix as a sparse vector of …
Dynamics and function of neuronal networks are determined by their synaptic connectivity. Current experimental methods to analyze synaptic network structure on the cellular level, however, cover only small fractions of functional neuronal circuits, typically without a simultaneous record of neuronal spiking activity. H…
This paper suggests a learning-theoretic perspective on how synaptic plasticity benefits global brain functioning. We introduce a model, the selectron, that (i) arises as the fast time constant limit of leaky integrate-and-fire neurons equipped with spiking timing dependent plasticity (STDP) and (ii) is amenable to the…
Biological neural network mimics CCA for multi-channel data.
problem Implementing CCA in a biologically plausible neural network.
method Derive an online CCA algorithm with local synaptic updates for multi-compartmental neurons.
result The derived neural network architecture and synaptic updates resemble cortical pyramidal neuron behavior.
Proposes a new method to improve deep neural network performance.
problem Improving generalization in deep neural networks on unseen data.
method Post-synaptic potential regularization (PSP).
result PSP achieves comparable performance to sophisticated learning strategies and improves generalization over ℓ2 regularization. Improved KAN model explains brain dynamics through edge learning and synaptic strength.
problem Explaining brain dynamics and frequencies in different brain regions.
method ELKAN (Edge Learning KNN) model with edge learning and trimming, inspired by brain science.
result ELKAN model outperforms KAN in explaining brain frequencies and dynamics.
A model explains credit assignment in deep learning networks.
problem Understanding how deep learning assigns credit to parameters.
method Mean-field learning model with ensembles of sub-networks.
result Synaptic connections can be categorized into three types.
Elite ONNs learn better with synaptic plasticity, improving performance over CNNs.
problem Limited heterogeneity in ONNs due to fixed operator sets.
method Synaptic plasticity-based search for optimal operator sets.
result Elite ONNs achieve superior learning performance compared to conventional methods.
Bayesian method reconstructs neural network memories from connectivity.
problem Reconstructing memories from neural network connectivity.
method Bayesian inference using statistical physics principles.
result Algorithm successfully reconstructs stored patterns from synaptic connectivity.
Lookahead pruning extends single-layer optimization to multi-layer, outperforming magnitude-based pruning.
problem Pruning neural networks to reduce computational cost and memory usage.
method Developed a multi-layer optimization approach extending the single-layer optimization of magnitude-based pruning.
result Consistently outperforms magnitude-based pruning on various networks, especially in high sparsity.
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
ICE-Pruning accelerates deep neural network pruning by 9.61x.
problem Efficiently pruning deep neural networks while maintaining accuracy.
method Iterative pruning with automatic fine-tuning steps, freezing strategy, and custom learning rate scheduler.
result Significantly reduces pruning time by up to 9.61x.
DSA efficiently allocates sparsity across layers for budgeted pruning.
problem Efficiently distributing resources (sparsity) across layers in pruning under resource constraints.
method DSA uses differentiable pruning to find continuous layer-wise pruning ratios via gradient-based optimization.
result DSA achieves superior performance and significantly reduces the time cost of pruning.
Study examines effects of pruning techniques on deep learning models.
problem Understanding the impact of pruning methods on deep learning model structure and dynamics.
method Investigated differences in connectivity and learning dynamics of pruned models using various iterative pruning techniques.
result Emergence of structure in pruned models through magnitude-based unstructured pruning and weight rewinding.
Pruning neural networks can improve test accuracy even with significant parameter reduction.
problem The tradeoff between generalization and stability in neural network pruning.
method Analysis of pruning behavior over training, focusing on instability and its relation to generalization.
result Pruning's benefit to generalization increases with its instability.
Drop Pruning uses stochastic optimization to prune and recover weights, reducing model size and improving performance.
problem Complexity and inefficiency in pruning deep neural networks.
method Introduces stochastic optimization with 'drop away' and 'drop back' strategies to prune and recover weights.
result Achieves competitive compression performance and accuracy compared to state-of-the-art approaches.
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.
problem Flaws in existing pruning criteria for CNNs.
method Empirical experiments and Convolutional Weight Distribution Assumption.
result The Convolutional Weight Distribution Assumption improves filter selection in pruning.
Analyzes unsupervised neural networks using statistical mechanics and Monte Carlo simulations.
problem Understanding computational capabilities of unsupervised neural networks.
method Statistical mechanics approach and Monte Carlo simulations.
result Obtained a phase diagram summarizing network performance.
Gibbs pruning optimizes neural networks by combining physics and regularization.
problem Large neural networks are impractical for many applications.
method Combines statistical physics and stochastic regularization to train and prune networks simultaneously.
result Gibbs pruning achieves state-of-the-art performance on ResNet-56.
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
New method prunes neural networks at initialization, improving performance.
problem Improving neural network compression at initialization.
method Formally characterizes initialization conditions for reliable pruning based on connection sensitivity.
result Improved neural network performance on image classification tasks.
This paper introduces blind adversarial pruning to balance accuracy, efficiency, and robustness in neural networks.
problem Balancing accuracy, efficiency, and robustness in neural networks with limited resources.
method Adversarial pruning with a cutoff-scale strategy to dynamically adjust the strength of adversarial examples.
result Blind adversarial pruning improves the overall AER of pruned models compared to adversarial pruning.
Deep CNN model predicts neuronal cell health from images.
problem Predicting the biological activity of chemical compounds on neuronal cells.
method Deep convolutional neural network (CNN) with residual connections.
result Achieved 99.6% accuracy in distinguishing treated from untreated cells.
Olshausen and Field (OF) proposed that neural computations in the primary visual cortex (V1) can be partially modeled by sparse dictionary learning. By minimizing the regularized representation error they derived an online algorithm, which learns Gabor-filter receptive fields from a natural image ensemble in agreement …
Speeds up training and inference by pruning entire channels before training.
problem Training and inference speed in deep neural networks.
method Structured pruning applied before training, focusing on removing entire channels and hidden units.
result 2x speedup in training and 3x speedup in inference.
Paper models Pavlov's classical conditioning using stochastic processes and Langevin equations.
problem Lack of modeling for Pavlov's classical conditioning.
method Modeling neural and synaptic dynamics via Langevin equations.
result Pavlov's mechanism spontaneously leads to synaptic weights similar to Hebb's.