Learning and memory in the brain are implemented by complex, time-varying changes in neural circuitry. The computational rules according to which synaptic weights change over time are the subject of much research, and are not precisely understood. Until recently, limitations in experimental methods have made it challen…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “synaptic weights”
Natural gradient learning improves synaptic plasticity in spiking neurons.
problem Parametrization dependence leads to inconsistencies in classical synaptic plasticity theories.
method Proposes natural gradient descent in Riemannian geometry for spiking neurons.
result Derives a synaptic learning rule that explains biological phenomena.
Adaptive Synaptic Failure Enables Sampling from Posterior Predictive Distributions in the Brainq-bio.NC
Brain uses synaptic failure to sample from posterior distributions.
problem Bayesian inference in the brain's probabilistic computations.
method Adapting synaptic failure to sample posterior predictive distributions.
result Synaptic failure enables sampling of complete posterior predictive distributions.
A neuron is a basic physiological and computational unit of the brain. While much is known about the physiological properties of a neuron, its computational role is poorly understood. Here we propose to view a neuron as a signal processing device that represents the incoming streaming data matrix as a sparse vector of …
A model explains credit assignment in deep learning networks.
problem Understanding how deep learning assigns credit to parameters.
method Mean-field learning model with ensembles of sub-networks.
result Synaptic connections can be categorized into three types.
Current algorithms for deep learning probably cannot run in the brain because they rely on weight transport, where forward-path neurons transmit their synaptic weights to a feedback path, in a way that is likely impossible biologically. An algorithm called feedback alignment achieves deep learning without weight transp…
New model reveals balance crucial for robust neural coding.
problem Efficient neural coding in noisy, chaotic networks.
method Analytical model of balanced predictive coding with dissociated balance and weight disorder.
result Superclassical scaling in coding accuracy, independent of balance and weight disorder.
Variational mean-field theory for training restricted Boltzmann machines with binary synapsesstat.ML
Unsupervised learning requiring only raw data is not only a fundamental function of the cerebral cortex, but also a foundation for a next generation of artificial neural networks. However, a unified theoretical framework to treat sensory inputs, synapses and neural activity together is still lacking. The computational …
This paper suggests a learning-theoretic perspective on how synaptic plasticity benefits global brain functioning. We introduce a model, the selectron, that (i) arises as the fast time constant limit of leaky integrate-and-fire neurons equipped with spiking timing dependent plasticity (STDP) and (ii) is amenable to the…
Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes. Here we show that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a numb…
Catastrophic forgetting/interference is a critical problem for lifelong learning machines, which impedes the agents from maintaining their previously learned knowledge while learning new tasks. Neural networks, in particular, suffer plenty from the catastrophic forgetting phenomenon. Recently there has been several eff…
Neural networks are commonly trained to make predictions through learning algorithms. Contrastive Hebbian learning, which is a powerful rule inspired by gradient backpropagation, is based on Hebb's rule and the contrastive divergence algorithm. It operates in two phases, the forward (or free) phase, where the data are …
Olshausen and Field (OF) proposed that neural computations in the primary visual cortex (V1) can be partially modeled by sparse dictionary learning. By minimizing the regularized representation error they derived an online algorithm, which learns Gabor-filter receptive fields from a natural image ensemble in agreement …
This paper presents a supervised learning algorithm, namely, the Synaptic Efficacy Function with Meta-neuron based learning algorithm (SEF-M) for a spiking neural network with a time-varying weight model. For a given pattern, SEF-M uses the learning algorithm derived from meta-neuron based learning algorithm to determi…
Ongoing studies have identified similarities between neural representations in biological networks and in deep artificial neural networks. This has led to renewed interest in developing analogies between the backpropagation learning algorithm used to train artificial networks and the synaptic plasticity rules operative…
Learning weights in a spiking neural network with hidden neurons, using local, stable and online rules, to control non-linear body dynamics is an open problem. Here, we employ a supervised scheme, Feedback-based Online Local Learning Of Weights (FOLLOW), to train a network of heterogeneous spiking neurons with hidden l…
Soft diamond regularizers improve deep learning performance and sparsity.
problem Improving deep learning performance and sparsity of trained weights.
method New soft diamond synaptic weight priors based on thick-tailed symmetric alpha stable probability curves.
result Soft diamond regularizers outperform state-of-the-art methods in deep learning tasks.
New algorithm shows neural networks can learn without full backpropagation.
problem Stochastic gradient descent with backpropagation is non-biologically plausible.
method Random and fixed backpropagation weights in a feedback alignment algorithm.
result Error converges to zero exponentially fast in overparameterized networks.
Sleep-based regularization stabilizes STDP in recurrent neural networks.
problem Pathological weight dynamics in recurrent SNNs.
method Periodic offline phases with stochastic decay and spontaneous activity.
result Sleep-based renormalization prevents weight saturation and preserves learned structure.
Improved KAN model explains brain dynamics through edge learning and synaptic strength.
problem Explaining brain dynamics and frequencies in different brain regions.
method ELKAN (Edge Learning KNN) model with edge learning and trimming, inspired by brain science.
result ELKAN model outperforms KAN in explaining brain frequencies and dynamics.
Pavlov Learning Machinescond-mat.dis-nn
Paper models Pavlov's classical conditioning using stochastic processes and Langevin equations.
problem Lack of modeling for Pavlov's classical conditioning.
method Modeling neural and synaptic dynamics via Langevin equations.
result Pavlov's mechanism spontaneously leads to synaptic weights similar to Hebb's.
Synaptic strength can be seen as probability to propagate impulse, and according to synaptic plasticity, function could exist from propagation activity to synaptic strength. If the function satisfies constraints such as continuity and monotonicity, neural network under external stimulus will always go to fixed point, a…
Deep neural networks (DNNs) often require good regularizers to generalize well. Currently, state-of-the-art DNN regularization techniques consist in randomly dropping units and/or connections on each iteration of the training algorithm. Dropout and DropConnect are characteristic examples of such regularizers, that are …
Convolutional Neural Networks(CNNs) are both computation and memory intensive which hindered their deployment in mobile devices. Inspired by the relevant concept in neural science literature, we propose Synaptic Pruning: a data-driven method to prune connections between input and output feature maps with a newly propos…
Bio-inspired neural networks use predictive coding for efficient weight updates.
problem Training artificial neural networks efficiently and biologically plausibly.
method Predictive Coding (PC) updates weights locally using only local information.
result PC provides theoretical advantages like automatic gradient scaling.
Information in neural networks is represented as weighted connections, or synapses, between neurons. This poses a problem as the primary computational bottleneck for neural networks is the vector-matrix multiply when inputs are multiplied by the neural network weights. Conventional processing architectures are not well…
Low-rank structure emerges in neural networks during learning.
problem Understanding the evolution of synaptic connectivity over learning.
method Investigated the rank of 3-tensor formed by weight matrices throughout learning.
result Inferred weights are low-tensor-rank and evolve in a fixed low-dimensional subspace.
The study reveals how synaptic correlations promote dimension reduction in neural networks.
problem Understanding how synaptic correlations affect neural correlations and dimension reduction in deep neural networks.
method A simplified model of dimension reduction considering pairwise correlations among synapses, using mathematical self-consistency for both binary and continuous synapses.
result Weakly-correlated synapses encourage dimension reduction compared to orthogonal synapses, and they also slow down the decorrelation process.
Locally learned synaptic failure enables complete Bayesian inference.
problem Bayesian inference in neural networks.
method Biologically constrained neural network with synaptic failure and lateral inhibition.
result Synaptic failure allows sampling from both epistemic and aleatoric distributions.
We describe a novel spiking neural network (SNN) for automated, real-time handwritten digit classification and its implementation on a GP-GPU platform. Information processing within the network, from feature extraction to classification is implemented by mimicking the basic aspects of neuronal spike initiation and prop…
Neural network models of early sensory processing typically reduce the dimensionality of streaming input data. Such networks learn the principal subspace, in the sense of principal component analysis (PCA), by adjusting synaptic weights according to activity-dependent learning rules. When derived from a principled cost…
Identifies learning rules from neural network observables.
problem Determine the underlying plasticity rules governing learning in biological systems.
method Simulated idealized neuroscience experiments with artificial neural networks to generate a dataset of learning trajectories. Used linear and non-linear classifiers to identify learning rules from aggregate statistics of weights, activations, and activity changes.
result Different classes of learning rules can be separated solely on the basis of aggregate statistics of the weights, activations, or instantaneous layer-wise activity changes.
Significant success has been reported recently using deep neural networks for classification. Such large networks can be computationally intensive, even after training is over. Implementing these trained networks in hardware chips with a limited precision of synaptic weights may improve their speed and energy efficienc…
There has been significant recent interest towards achieving highly efficient deep neural network architectures. A promising paradigm for achieving this is the concept of evolutionary deep intelligence, which attempts to mimic biological evolution processes to synthesize highly-efficient deep neural networks over succe…
Despite our extensive knowledge of biophysical properties of neurons, there is no commonly accepted algorithmic theory of neuronal function. Here we explore the hypothesis that single-layer neuronal networks perform online symmetric nonnegative matrix factorization (SNMF) of the similarity matrix of the streamed data. …
Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapsescond-mat.dis-nn
We show that discrete synaptic weights can be efficiently used for learning in large scale neural systems, and lead to unanticipated computational performance. We focus on the representative case of learning random patterns with binary synapses in single layer networks. The standard statistical analysis shows that this…
Evolution in Groups: A deeper look at synaptic cluster driven evolution of deep neural networkscs.NE
A promising paradigm for achieving highly efficient deep neural networks is the idea of evolutionary deep intelligence, which mimics biological evolution processes to progressively synthesize more efficient networks. A crucial design factor in evolutionary deep intelligence is the genetic encoding scheme used to simula…
Enhances SNNs for spatio-temporal feature extraction.
problem Insufficient temporal dependencies in existing SNN synaptic structures.
method Integrates temporal convolution and attention mechanisms into synaptic connections.
result Improves SNN performance on classification tasks.
Learning may need only a few bits of synaptic precisioncond-mat.dis-nn
Learning in neural networks poses peculiar challenges when using discretized rather then continuous synaptic states. The choice of discrete synapses is motivated by biological reasoning and experiments, and possibly by hardware implementation considerations as well. In this paper we extend a previous large deviations a…
QSD enhances deep network performance through biologically plausible dropout.
problem Overfitting in deep networks.
method Quantal Synaptic Dilution (QSD) model based on neuronal synapses.
result QSD outperforms standard dropout in various deep network architectures.
Synthesizing Deep Neural Network Architectures using Biological Synaptic Strength Distributionscs.NE
In this work, we perform an exploratory study on synthesizing deep neural networks using biological synaptic strength distributions, and the potential influence of different distributions on modelling performance particularly for the scenario associated with small data sets. Surprisingly, a CNN with convolutional layer…
This paper introduces a hierarchical associative memory model with multiple layers.
problem Limitations of traditional associative memory models with only one hidden layer.
method Develops a fully recurrent model with arbitrary layers, including locally connected ones, and a corresponding energy function.
result The model can dynamically assemble memories using weights from lower layers and higher layers' rules.
High-speed model accurately simulates neuromorphic devices.
problem Accurately modeling stochastic synapses in large-scale neuromorphic systems.
method Generative vector autoregressive model based on resistive memory cell data.
result Fast, high-throughput model reproduces synaptic parameters and correlations.
ANPyC combats forgetting by pruning and consolidating neural parameters.
problem Catastrophic forgetting in neural networks, especially with long-term tasks.
method Adversarial Neural Pruning and Synaptic Consolidation (ANPyC) to balance task-relevant and irrelevant parameters.
result ANPyC prevents forgetting while enabling efficient learning of multiple tasks.
In this paper, we derive a new model of synaptic plasticity, based on recent algorithms for reinforcement learning (in which an agent attempts to learn appropriate actions to maximize its long-term average reward). We show that these direct reinforcement learning algorithms also give locally optimal performance for the…
We study the computational capacity of a model neuron, the Tempotron, which classifies sequences of spikes by linear-threshold operations. We use statistical mechanics and extreme value theory to derive the capacity of the system in random classification tasks. In contrast to its static analog, the Perceptron, the Temp…
Study on random matrices in deep neural networks with IID entries.
problem Distribution of singular values in product of random matrices for deep neural networks.
method Random matrix theory with a streamlined approach for non-Gaussian data.
result Generalization of macroscopic universality property to non-Gaussian data.
FP uses random projections to train networks without feedback, achieving comparable performance to backpropagation.
problem Training neural networks without feedback from downstream layers.
method Forward Projection (FP) method that uses randomised nonlinear projections and closed-form regression.
result FP achieves comparable generalisation to backpropagation methods with a single forward pass, offering significant speedup.