Unified approach for neural networks with multi-compartmental neurons and non-Hebbian plasticity.
problem Limited computational power of existing neural network models for multi-compartmental neurons and non-Hebbian plasticity.
method Unified extension of similarity matching approach to derive neural networks with multi-compartmental neurons and local, non-Hebbian learning rules.
result Unified approach facilitates understanding of multi-compartmental neuronal structures and non-Hebbian plasticity.
Neural network models of early sensory processing typically reduce the dimensionality of streaming input data. Such networks learn the principal subspace, in the sense of principal component analysis (PCA), by adjusting synaptic weights according to activity-dependent learning rules. When derived from a principled cost…
Enhanced BCPNN learns hidden representations without labels.
problem Unsupervised learning of hierarchical representations in neural networks.
method Extended BCPNN architecture with local Hebbian learning mechanisms.
result Demonstrated capability for unsupervised learning of salient hidden representations.
In this work we propose Hebbian-descent as a biologically plausible learning rule for hetero-associative as well as auto-associative learning in single layer artificial neural networks. It can be used as a replacement for gradient descent as well as Hebbian learning, in particular in online learning, as it inherits the…
Combines Hebbian and DQN for better POMDP problem solving.
problem Difficult POMDP problems with TD errors.
method Modulated Hebbian plus Q network architecture (MOHQA) integrating Hebbian and DQN.
result Improved DQN performance and outperformed other algorithms on some POMDPs.
Biological neurons learn tensor decompositions of higher-order correlations using nonlinear Hebbian plasticity.
problem Learning higher-order correlations in biological neurons.
method Introduce and study generalized nonlinear Hebbian learning rules.
result Neurons can learn tensor eigenvectors of higher-order input correlation tensors.
A new graph embedding method using Hebbian learning for improved vector representations.
problem Creating accurate vector representations for nodes in graphs.
method Hebbian learning with non-convex Gaussian mixture model for node embeddings.
result The method outperforms state-of-the-art methods on benchmark data sets and generates relevant recommendations.
Local unsupervised learning outperforms end-to-end training for image classification.
problem Local unsupervised learning is believed to be inferior to end-to-end training.
method Designing a local algorithm to learn convolutional filters at scale on large image datasets using local Hebbian learning.
result Convolutional networks with patch normalization significantly outperform standard networks on image classification tasks.
Hebbian learning derived from maximum entropy principles.
problem Storing and retrieving information in neural networks.
method Maximum entropy extremization to derive Hebbian learning rules.
result Hebbian learning rules converge to Hopfield model in the big data limit.
Despite our extensive knowledge of biophysical properties of neurons, there is no commonly accepted algorithmic theory of neuronal function. Here we explore the hypothesis that single-layer neuronal networks perform online symmetric nonnegative matrix factorization (SNMF) of the similarity matrix of the streamed data. …
We show that deep networks can be trained using Hebbian updates yielding similar performance to ordinary back-propagation on challenging image datasets. To overcome the unrealistic symmetry in connections between layers, implicit in back-propagation, the feedback weights are separate from the feedforward weights. The f…
Neural network learns manifold structure for semi-supervised learning.
problem Semi-supervised learning on manifolds with explicit graph representations.
method Feed-forward neural network with two layers, learning manifold structure and classification.
result Neural network achieves semi-supervised learning on non-trivial manifolds.
A model retains learned knowledge for longer by adding a plastic component to neural networks.
problem Catastrophic forgetting in neural networks when learning new tasks.
method Differentiable Hebbian Consolidation model with a DHP Softmax layer.
result Reduces forgetting in benchmarks like Permuted MNIST and Vision Datasets Mixture.
We unify recent neural approaches to one-shot learning with older ideas of associative memory in a model for metalearning. Our model learns jointly to represent data and to bind class labels to representations in a single shot. It builds representations via slow weights, learned across tasks through SGD, while fast wei…
Proposes a new neural network approach to credit assignment.
problem Credit assignment problem in deep neural networks.
method Contrastive similarity matching objective function.
result Deep networks learn to match similarity between layers.
Olshausen and Field (OF) proposed that neural computations in the primary visual cortex (V1) can be partially modeled by sparse dictionary learning. By minimizing the regularized representation error they derived an online algorithm, which learns Gabor-filter receptive fields from a natural image ensemble in agreement …
Sleep-based regularization stabilizes STDP in recurrent neural networks.
problem Pathological weight dynamics in recurrent SNNs.
method Periodic offline phases with stochastic decay and spontaneous activity.
result Sleep-based renormalization prevents weight saturation and preserves learned structure.
New derivation shows how a three-factor learning rule is derived from Oja's rule.
problem Deriving a three-factor learning rule from Oja's rule.
method Using frame theory to systematically derive EGHR-PCA from Oja's rule.
result A principled derivation of a biologically plausible learning rule.
Neural networks are commonly trained to make predictions through learning algorithms. Contrastive Hebbian learning, which is a powerful rule inspired by gradient backpropagation, is based on Hebb's rule and the contrastive divergence algorithm. It operates in two phases, the forward (or free) phase, where the data are …
Bio-inspired neural networks use predictive coding for efficient weight updates.
problem Training artificial neural networks efficiently and biologically plausibly.
method Predictive Coding (PC) updates weights locally using only local information.
result PC provides theoretical advantages like automatic gradient scaling.
Paper introduces supervised and unsupervised TAM models for binary neurons.
problem Learning and retrieval of structured triplets of patterns in neural networks.
method Extends Hebbian paradigm to supervised and unsupervised protocols, using glassy statistical mechanical techniques.
result Obtained self-consistency equations for critical dataset sizes and retrieval performance.
New learning rules from information bottleneck improve deep learning without precise labels.
problem Training deep neural networks with backpropagation is biologically implausible.
method Kernelized information bottleneck principle with 3-factor Hebbian structure.
result The new learning rules perform nearly as well as backpropagation on image classification tasks.
A new learning framework mimics biological STDP for neural networks.
problem To create a neural network that learns like biological systems.
method Developed MSTDP framework using Spike-timing dependent plasticity rules.
result Framework can learn and generate patterns without additional supervision.
Recently, a novel family of biologically plausible online algorithms for reducing the dimensionality of streaming data has been derived from the similarity matching principle. In these algorithms, the number of output dimensions can be determined adaptively by thresholding the singular values of the input data matrix. …
The study reveals how synaptic correlations promote dimension reduction in neural networks.
problem Understanding how synaptic correlations affect neural correlations and dimension reduction in deep neural networks.
method A simplified model of dimension reduction considering pairwise correlations among synapses, using mathematical self-consistency for both binary and continuous synapses.
result Weakly-correlated synapses encourage dimension reduction compared to orthogonal synapses, and they also slow down the decorrelation process.
In a physical neural system, where storage and processing are intimately intertwined, the rules for adjusting the synaptic weights can only depend on variables that are available locally, such as the activity of the pre- and post-synaptic neurons, resulting in local learning rules. A systematic framework for studying t…
BDH model learns like the brain, rivaling Transformer performance.
problem Leveraging brain-like properties for machine learning.
method Scale-free biologically inspired network of neuron particles.
result BDH model achieves Transformer-like performance with interpretability.
Biological neural network mimics CCA for multi-channel data.
problem Implementing CCA in a biologically plausible neural network.
method Derive an online CCA algorithm with local synaptic updates for multi-compartmental neurons.
result The derived neural network architecture and synaptic updates resemble cortical pyramidal neuron behavior.
New synaptic model derived from reinforcement learning for spiking neurons.
problem Learning optimal actions in complex systems.
method Derives a synaptic update rule from reinforcement learning algorithms.
result Synaptic strengths lead to locally optimal reward values.
The paper relaxes constraints on predictive coding models, making them more biologically plausible.
problem Neurophysiological models of predictive coding are not fully biologically plausible.
method The paper relaxes constraints on standard predictive coding algorithms by removing neurally implausible features.
result The removal of neurally implausible features does not significantly affect learning performance.
Oja's rule improves neural network training without engineered tricks.
problem Training deep neural networks with biological constraints.
method Incorporating Oja's plasticity rule into error-driven training.
result Stable, efficient learning in feedforward and recurrent architectures.
New algorithm shows neural networks can learn without full backpropagation.
problem Stochastic gradient descent with backpropagation is non-biologically plausible.
method Random and fixed backpropagation weights in a feedback alignment algorithm.
result Error converges to zero exponentially fast in overparameterized networks.
The study revisits Hopfield's associative memory model and calculates its capacity for two specific pattern basins.
problem Determining the capacity of a Hebbian-Hopfield network for storing binary patterns.
method Using fully lifted random duality theory and numerical analysis, the study calculates the capacity for two specific pattern basins.
result Explicit characterizations of the capacity for the AGS and NLT pattern basins, with remarkable fast lifting convergence.
Bayesian Predictive Coding improves deep learning uncertainty quantification.
problem Limitations of maximum a posteriori and maximum likelihood estimates in predictive coding.
method Developed Bayesian Predictive Coding (BPC) that estimates a posterior distribution over network parameters.
result BPC offers comparable uncertainty quantification to existing methods in Bayesian deep learning and improves convergence properties.
Develops a new method for solving generalized eigenvalue problems efficiently.
problem Efficiently solving generalized eigenvalue problems for large datasets.
method Inspired by the Generalized Hebbian Algorithm, develops a game-theory inspired approach to solving GEPs.
result Demonstrates state-of-the-art performance for optimizing Deep CCA.
New neural network can learn multiple patterns at once.
problem Learning multiple patterns simultaneously in neural networks.
method Multitasking Hebbian Network (MHN) for parallel learning of sparse data-sets.
result MHN can handle multiple patterns hierarchically or in parallel, depending on dilution.
Enhances sequence memory capacity in neural networks.
problem Limited sequence capacity in Hopfield-like neural networks.
method Introducing a nonlinear interaction term and a generalized pseudoinverse rule.
result Significantly increased sequence capacity with novel scaling laws.
Catastrophic forgetting/interference is a critical problem for lifelong learning machines, which impedes the agents from maintaining their previously learned knowledge while learning new tasks. Neural networks, in particular, suffer plenty from the catastrophic forgetting phenomenon. Recently there has been several eff…
Recently a daily routine for associative neural networks has been proposed: the network Hebbian-learns during the awake state (thus behaving as a standard Hopfield model), then, during its sleep state, optimizing information storage, it consolidates pure patterns and removes spurious ones: this forces the synaptic matr…
VSML unifies meta learning concepts and enables simple backpropagation.
problem Improving and unifying meta learning concepts for neural networks.
method Unified approach using variable shared meta learning and simple weight-sharing.
result Simple backpropagation can be implemented and meta learned without gradient calculation.
We propose a new cognitive framework for option price modelling, using quantum neural computation formalism. Briefly, when we apply a classical nonlinear neural-network learning to a linear quantum Schrödinger equation, as a result we get a nonlinear Schrödinger equation (NLS), performing as a quantum stochastic filter…
How can we build agents that keep learning from experience, quickly and efficiently, after their initial training? Here we take inspiration from the main mechanism of learning in biological brains: synaptic plasticity, carefully tuned by evolution to produce efficient lifelong learning. We show that plasticity, just li…
Thanks to the availability of large scale digital datasets and massive amounts of computational power, deep learning algorithms can learn representations of data by exploiting multiple levels of abstraction. These machine learning methods have greatly improved the state-of-the-art in many challenging cognitive tasks, s…
A federated model learns shared archetypes from heterogeneous clients in continual learning.
problem Federated learning struggles with client heterogeneity and streaming distribution shifts.
method Clients encode their data as low-rank Hebbian operators, which are sent to a central server for aggregation and factorization into global archetypes.
result Improved global archetype reconstruction and associative retrieval in heterogeneous clients, drift, and novelty settings.
A dynamic Boltzmann machine (DyBM) has been proposed as a model of a spiking neural network, and its learning rule of maximizing the log-likelihood of given time-series has been shown to exhibit key properties of spike-timing dependent plasticity (STDP), which had been postulated and experimentally confirmed in the fie…
Analyzes unsupervised neural networks using statistical mechanics and Monte Carlo simulations.
problem Understanding computational capabilities of unsupervised neural networks.
method Statistical mechanics approach and Monte Carlo simulations.
result Obtained a phase diagram summarizing network performance.
Artifical Neural Networks are a particular class of learning systems modeled after biological neural functions with an interesting penchant for Hebbian learning, that is "neurons that wire together, fire together". However, unlike their natural counterparts, artificial neural networks have a close and stringent couplin…
A central problem to understanding intelligence is the concept of generalisation. This allows previously learnt structure to be exploited to solve tasks in novel situations differing in their particularities. We take inspiration from neuroscience, specifically the hippocampal-entorhinal system known to be important for…