Learning can be efficient with binary synapses, needing only a few bits of precision.
problem Learning in neural networks with discrete synapses.
method Extending large deviations analysis to synapses with multiple states, showing efficiency and robustness.
result Only a few bits of synaptic precision are needed for near-optimal performance, consistent with biological findings.
Proposes an alternative method to train RBMs with binary synapses using Bayesian learning rule.
problem Training RBMs with binary synapses is challenging due to discrete nature of synapses.
method Proposes an alternative optimization method using the Bayesian learning rule, updating natural parameters instead of expectation parameters.
result No additional clipping is needed as natural parameters take values in the entire real domain.
We show that discrete synaptic weights can be efficiently used for learning in large scale neural systems, and lead to unanticipated computational performance. We focus on the representative case of learning random patterns with binary synapses in single layer networks. The standard statistical analysis shows that this…
Unified theory for training neural networks with binary synapses.
problem Discrete nature of synapses and complex interactions in neural networks.
method Variational mean-field theory decomposing learning into maximization and expectation steps.
result Unified framework for unsupervised learning in neural networks.
Stochastic binary synapses in neural networks lead to robust solutions.
problem Training low-precision neural networks with stochastic weights.
method Gradient descent on a probability distribution of binary synapses.
result Binary solutions are robust and generalize well, while typical solutions are isolated.
The study reveals how synaptic correlations promote dimension reduction in neural networks.
problem Understanding how synaptic correlations affect neural correlations and dimension reduction in deep neural networks.
method A simplified model of dimension reduction considering pairwise correlations among synapses, using mathematical self-consistency for both binary and continuous synapses.
result Weakly-correlated synapses encourage dimension reduction compared to orthogonal synapses, and they also slow down the decorrelation process.
Synapse arbitrates TSFMs to improve time series forecasting performance.
problem TSFMs vary in performance across different forecasting tasks, domains, and horizons.
method Synapse dynamically assigns and adjusts predictive weights based on TSFM performance.
result Synapse consistently outperforms other ensembling techniques and individual TSFMs.
Stochastic DNNs with memristive synapses perform well despite limited device variability.
problem Performance of DNNs with memristive synapses under limited dynamic range and variability.
method Stochastic training of DNNs with memristive synapses, focusing on variability optimization.
result Stochastic memristive DNNs suffer less than 3% loss in accuracy compared to floating point software baseline.
A model studies deep neural networks with binary synapses under connection removal.
problem Understanding the mechanism of deep learning from a theoretical perspective.
method Random active path model with diluted binary synapses under removal perturbation.
result A critical value of perturbation separates spin glass and paramagnetic phases, with the latter having poor generalization performance.
Updated VESICLE-CNN for faster synapse detection.
problem Slow patch-based synapse detection.
method Fully convolutional approach using dilated convolutions.
result 600x speedup at test time with no loss in accuracy.
Intelligent synapses help neural networks learn continuously.
problem Continual learning in changing data environments.
method Intelligent synapses that accumulate and reuse task-relevant information.
result Significant reduction in forgetting with improved computational efficiency.
New training algorithm enhances SNNs for temporal signal processing.
problem Lack of robust training algorithms for large-scale SNNs.
method Formulated SNN as IIR filters, proposed training algorithm for optimal synapse filter kernels and weights.
result Model and training algorithm outperform state-of-the-art approaches in accuracy.
Paper proposes Memory Aware Synapses for selective knowledge preservation in lifelong learning.
problem Selective preservation of knowledge in artificial learning systems.
method Unsupervised, online computation of parameter importance based on sensitivity to changes.
result State-of-the-art performance and ability to adapt parameter importance based on unlabeled data.
A neuromorphic unit models complex synapses efficiently.
problem Efficiently simulating complex synaptic response functions in neural networks.
method Digital neuromorphic architecture, Spiking Temporal Processing Unit (STPU), modeling arbitrary complex synaptic response functions.
result Demonstrates flexibility and efficiency of STPU for instantiating neural algorithms.
A neuromemristive HTM architecture boosts robustness and performance.
problem Improving robustness and performance of HTM algorithms.
method Developed a neuromemristive crossbar architecture for HTM, incorporating memristors and neurogenesis.
result Enhanced robustness and performance of HTM through neuromemristive architecture.
SIM-CE models C. elegans neural circuits for behavioral analysis.
problem Understanding the neural basis of C. elegans behavior.
method User-friendly Simulink platform with detailed neuron and synapse models.
result SIM-CE enables detailed multi-scale simulations of C. elegans behavior.
A central problem in neuroscience is reconstructing neuronal circuits on the synapse level. Due to a wide range of scales in brain architecture such reconstruction requires imaging that is both high-resolution and high-throughput. Existing electron microscopy (EM) techniques possess required resolution in the lateral p…
QSD enhances deep network performance through biologically plausible dropout.
problem Overfitting in deep networks.
method Quantal Synaptic Dilution (QSD) model based on neuronal synapses.
result QSD outperforms standard dropout in various deep network architectures.
Synaptic cluster-driven evolution improves deep neural networks by reducing synapses and clusters.
problem Efficiently synthesizing deep neural networks with fewer synapses and clusters.
method Synaptic cluster-driven genetic encoding scheme.
result Significantly smaller number of synapses and clusters in offspring networks.
High-speed model accurately simulates neuromorphic devices.
problem Accurately modeling stochastic synapses in large-scale neuromorphic systems.
method Generative vector autoregressive model based on resistive memory cell data.
result Fast, high-throughput model reproduces synaptic parameters and correlations.
New method for continual learning without task boundaries.
problem Traditional continual learning is task-based and impractical for real-world applications.
method Developed an online continual learning system using Memory Aware Synapses.
result Valid approach demonstrated in self-supervised learning and robot collision avoidance.
Networks capture our intuition about relationships in the world. They describe the friendships between Facebook users, interactions in financial markets, and synapses connecting neurons in the brain. These networks are richly structured with cliques of friends, sectors of stocks, and a smorgasbord of cell types that go…
EP algorithm improved for CNNs and real-time learning.
problem EP's long simulation times and non-local learning rule limitations.
method Discrete-time formulation, continual weight updates, local time information.
result C-EP achieves best performance on MNIST with CNNs.
A new neural model evolves to learn at the synaptic level.
problem Lack of biologically realistic neural models in deep learning.
method Evolve individual neuron and synaptic models using ENUs.
result Evolved neural network learns complex tasks like a T-maze.
We study the computational capacity of a model neuron, the Tempotron, which classifies sequences of spikes by linear-threshold operations. We use statistical mechanics and extreme value theory to derive the capacity of the system in random classification tasks. In contrast to its static analog, the Perceptron, the Temp…
New algorithm finds important synapses without training data.
problem Finding important synapses in neural networks without data.
method Iterative Synaptic Flow Pruning (SynFlow) based on conservation law.
result Algorithm consistently outperforms existing pruning methods.
Neural network memorizes external stimuli through synaptic strength changes.
problem Memory and classification in neural networks.
method One-to-one mapping between stimulus and synaptic strength under synaptic plasticity constraints.
result Neural network can memorize external stimuli through synaptic changes.
New loss function connects learning rate and momentum.
problem Finding optimal learning rate and momentum empirically.
method Proposes a new information-theoretical loss function.
result Loss, learning rate, and momentum are closely connected.
EP learns like BPTT but with local weight updates.
problem Existing EP lacks a local time learning rule.
method C-EP updates weights simultaneously with neuron dynamics.
result C-EP follows BPTT gradients and performs well.
Improved KAN model explains brain dynamics through edge learning and synaptic strength.
problem Explaining brain dynamics and frequencies in different brain regions.
method ELKAN (Edge Learning KNN) model with edge learning and trimming, inspired by brain science.
result ELKAN model outperforms KAN in explaining brain frequencies and dynamics.
Neural connectomics has begun producing massive amounts of data, necessitating new analysis methods to discover the biological and computational structure. It has long been assumed that discovering neuron types and their relation to microcircuitry is crucial to understanding neural function. Here we developed a nonpara…
Paper proves multiplicative weight updates can train neural networks without learning rate tuning.
problem Vanishing and exploding gradients in gradient descent for compositional functions.
method Proves descent lemma for compositional functions using multiplicative weight updates and derives Madam optimizer.
result Madam optimizer trains state-of-the-art neural networks without learning rate tuning.
Binary perceptron's instability linked to replica symmetry breaking.
problem Understanding the relationship between algorithmic instability and replica symmetry breaking in binary perceptron learning.
method Established the connection between algorithmic instability and replica symmetry breaking by comparing the instability condition around the fixed point to the instability for breaking the replica symmetric solution of the free energy function.
result The instability condition around the algorithmic fixed point is identical to the instability for breaking the replica symmetric saddle point solution of the free energy function.
Unsupervised learning for evolving data streams with STAM architecture.
problem Learning from non-stationary, unlabeled data streams over time.
method Self-Taught Associative Memory (STAM) architecture with online clustering, novelty detection, and feature storage.
result STAM architecture improves clustering and classification tasks compared to existing continual learning models.
Proposes a method to prevent neural networks from forgetting learned tasks.
problem Catastrophic forgetting in neural networks.
method Attention-based selective plasticity of synapses inspired by the cholinergic neuromodulatory system.
result Competitive performance on benchmark tasks compared to state-of-the-art methods.
We apply neuromodulation to deep neural networks to improve learning.
problem Static learning parameters in neural networks limit adaptability.
method Evolved neuromodulatory dynamics modify learning parameters over training.
result Evolution found dynamic, location-specific learning strategies.
A new type of neural network variable called 'fast weights' improves sequence models by storing recent past memories.
problem Sequence models benefit from attention to the past, but current neural networks lack a mechanism to store recent past information efficiently.
method Introduce 'fast weights' that change faster than neural activities but slower than standard weights, allowing for temporary memory storage of recent past.
result Fast weights enable efficient implementation of past attention in sequence models without needing to store neural activity patterns.
Natural gradient learning improves synaptic plasticity in spiking neurons.
problem Parametrization dependence leads to inconsistencies in classical synaptic plasticity theories.
method Proposes natural gradient descent in Riemannian geometry for spiking neurons.
result Derives a synaptic learning rule that explains biological phenomena.
Theory reconstructs network connectivity from event timings.
problem Reconstructing network connectivity from incomplete continuous-time data.
method Linearizes event space mapping to reveal direct influences.
result Reveals synapse presence and inhibitory/activating nature.
High-conductance neurons sample from target distributions in stochastic inference.
problem Understanding stochastic inference in neocortical circuits.
method Analytical derivation of neural activation function, simulation of spiking networks, Bayesian inference.
result Ensemble of spiking neurons can sample from a target distribution.
New dropout technique reduces training time by 20-77%.
problem Efficient training of deep neural networks consumes excessive time and energy.
method Approximate Random Dropout replaces random neuron/synapse dropout with regular patterns to reduce computation and data access.
result Reduces training time by 20-77% with minimal accuracy loss.
Log-Normal Multiplicative Dynamics improves low-precision training of neural networks.
problem Training large neural networks with low precision is unstable.
method Derive a Bayesian learning rule with log-normal posterior distributions and multiplicative updates.
result LMD achieves stable and accurate training for Vision Transformer and GPT-2.
Unsupervised learning by hidden units with biological plausibility.
problem Training neural networks without labeled data.
method Global inhibition in hidden layer to learn feature detectors.
result Learned feature detectors enable supervised training of higher layers.
Synthetic neural networks with fewer connections perform better.
problem Creating highly efficient deep neural networks.
method Synaptic cluster-driven genetic encoding mimics biological evolution.
result Synthetic networks achieve state-of-the-art performance with significantly fewer connections.
Paper models Pavlov's classical conditioning using stochastic processes and Langevin equations.
problem Lack of modeling for Pavlov's classical conditioning.
method Modeling neural and synaptic dynamics via Langevin equations.
result Pavlov's mechanism spontaneously leads to synaptic weights similar to Hebb's.
The study learns neural update rules by remembering past experiences.
problem Developing efficient online learning rules for neural networks.
method Representing neurons with vectors, using meta-neural networks for updates, and training for remembering past experiences.
result The approach reveals insights into learning rules and could be used for complex tasks like episodic memory.
New synaptic model derived from reinforcement learning for spiking neurons.
problem Learning optimal actions in complex systems.
method Derives a synaptic update rule from reinforcement learning algorithms.
result Synaptic strengths lead to locally optimal reward values.
Proposes a method to retrain neural networks incrementally for continuous data flow.
problem Continuous data flow and the challenges of catastrophic forgetting and efficient retraining.
method Incremental retraining using multi-armed bandits to select important samples and weights, and a new regularization term for synapse and neuron importance.
result Mitigates catastrophic forgetting and boosts model performance.