New artificial neuron doubles weight for improved deep learning accuracy.
problem Improving deep learning models' accuracy.
method Introducing a double-weight neuron, tested on MNIST and CIFAR-10 datasets.
result Significant improvement in classification accuracy for MNIST and CIFAR-10 datasets.
Novel 'strong neuron' improves deep learning efficiency and robustness.
problem Improving deep learning efficiency and robustness against adversarial attacks.
method Introducing a novel 'strong neuron' model and a constructive training algorithm.
result Achieved 10x-100x reduction in operations count and hardware requirements.
RNNs trained on head direction task mimic brain's compass and shifter neurons.
problem Modeling brain's head direction system using neural networks.
method Optimized recurrent neural networks trained on angular velocity integration.
result RNNs naturally emerge with compass and shifter neuron-like properties.
BEAN models neuronal correlations to create interpretable representations.
problem Hard interpretation of dense-layer representations in DNNs.
method Inspired by neuroscience, BEAN models neuronal correlations and dependencies.
result BEAN enables formation of interpretable neuronal clusters without sacrificing model performance.
New method learns neural network activation functions from data.
problem Learning activation functions for neural networks.
method Model each neuron's activation function as a small neural network.
result Learned activation functions improve network performance.
Study enhances neural network interpretability through statistical methods.
problem Complexity and interpretability challenges in neural networks.
method Theoretical framework, statistical tests, dimensionality reduction algorithms.
result Developed bootstrapping technique and statistical tests for ANN performance.
Single neuron with ADA learns XOR and outperforms other functions.
problem Classifying linearly non-separable data.
method Proposed a new artificial neuron with apical dendrite activation.
result ADA function achieves 100% accuracy on XOR and superior performance on benchmark datasets.
AKOrN uses synchronized neurons to improve AI tasks.
problem Improving AI performance through better neural representations.
method AKOrN introduces synchronized neurons to replace threshold units.
result AKOrN improves performance across various AI tasks.
The artificial neural network is a popular framework in machine learning. To empower individual neurons, we recently suggested that the current type of neurons could be upgraded to 2nd order counterparts, in which the linear operation between inputs to a neuron and the associated weights is replaced with a nonlinear qu…
Visualizes deep neural networks for speech recognition using learned topographic filter maps.
problem Unintuitive internal structure of deep neural networks complicates activation visualization.
method Trains a convolutional speech recognition model with filters arranged in a 2D grid, highlighting similar filters.
result Topographic filter maps visualize artificial neuron activations more intuitively.
New learning algorithm mimics biological neural networks.
problem Biologically implausible backpropagation for directed neural networks.
method Introduces new neuronal dynamics and learning rule for arbitrary architectures, sparsity-inducing pruning method, and dynamical-systems characterization.
result Prunes irrelevant connections and improves learning efficiency.
A current research trend in neurocomputing involves the design of novel artificial neural networks incorporating the concept of time into their operating model. In this paper, a novel architecture that employs stigmergy is proposed. Computational stigmergy is used to dynamically increase (or decrease) the strength of a…
We describe a mechanism by which artificial neural networks can learn rapid adaptation - the ability to adapt on the fly, with little data, to new tasks - that we call conditionally shifted neurons. We apply this mechanism in the framework of metalearning, where the aim is to replicate some of the flexibility of human …
Automated method finds meaningful directions in neural network activations.
problem Mixed selectivity in neurons makes interpretation challenging.
method Automated quantification of interpretability and discovery of meaningful directions.
result Meaningful directions in neural network activations are more interpretable than individual neurons.
Spatially positioned neurons in neural networks mimic biological systems.
problem Creating neural networks that can perform multiple tasks efficiently.
method Added spatial positions and proximity penalties to artificial neurons.
result Neurons naturally cluster, each responsible for a specific task.
Law derived for neural networks with sparse connections.
problem Understanding the behavior of neural networks with sparse connections.
method Law of large numbers for empirical distribution of parameters derived.
result Law for neural networks with sparse connections derived.
Despite our extensive knowledge of biophysical properties of neurons, there is no commonly accepted algorithmic theory of neuronal function. Here we explore the hypothesis that single-layer neuronal networks perform online symmetric nonnegative matrix factorization (SNMF) of the similarity matrix of the streamed data. …
Quadratic autoencoder improves low-dose CT image denoising.
problem Low-dose CT image denoising.
method Quadratic autoencoder architecture applied to CT denoising.
result Quadratic autoencoder achieves superior denoising performance and efficiency.
Neural networks learn to mimic brain neurons with two-input activation functions, improving performance and robustness.
problem Training neural networks to mimic the complex interactions of brain neurons.
method Developed a network-in-network architecture with two-input activation functions, optimized hyperparameters, and compared to conventional ReLU networks.
result Two-input activation functions can learn soft XOR functions, improving network performance and robustness.
Named-entity recognition (NER) aims at identifying entities of interest in a text. Artificial neural networks (ANNs) have recently been shown to outperform existing NER systems. However, ANNs remain challenging to use for non-expert users. In this paper, we present NeuroNER, an easy-to-use named-entity recognition tool…
New estimator corrects bias in CKA for sparsely sampled neurons.
problem Bias in CKA for sparsely sampled neurons.
method Novel estimator that corrects for input and feature sampling.
result Reliable model-to-brain alignment with sparsely sampled neurons.
This paper presents a new artificial neuron model capable of learning its receptive field in the topological domain of inputs. The model provides adaptive and differentiable local connectivity (plasticity) applicable to any domain. It requires no other tool than the backpropagation algorithm to learn its parameters whi…
Graph Convolutional Networks improve prosthetic sensation interpretation.
problem Improving neuroprosthetic performance and sensory information stability.
method Applied Graph Convolutional Networks (GCNs) to interpret neuronal spiking activity.
result GCN model achieved 73.5% performance on ordinal regression task.
Novel method decorrelates neurons for better deep learning model generalization.
problem High correlations between neurons limit deep learning model generalization.
method Regularization terms from minimum spanning tree of neuron cliques, using correlation dissimilarities.
result Our regularizers outperform existing methods and minimize neuron redundancies.
Single neurons can perform as well as dense networks in binary and multi-class recognition tasks.
problem Designing efficient neural networks for recognition tasks.
method Investigated the use of single or multiple neurons in neural networks for binary and multi-class recognition tasks.
result Sparse networks can be as efficient as dense networks in both binary and multi-class tasks.
Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural ne…
Novel chaotic neurons improve AI with minimal training data.
problem Limited training data for AI algorithms.
method Intrinsically chaotic neurons inspired by chaos theory.
result Classification accuracy up to 95.8% with just 2 training samples per class.
This brief note highlights some basic concepts required toward understanding the evolution of machine learning and deep learning models. The note starts with an overview of artificial intelligence and its relationship to biological neuron that ultimately led to the evolution of todays intelligent models.
Proposes a new neural network architecture inspired by biology to improve learning and information flow.
problem Improving artificial neural networks to match biological neuron properties like multidirectional propagation and probabilistic modeling.
method Extends KAN approach with joint distribution neurons that can propagate values and distributions, including variance and higher-order moments.
result Proposed architecture can predict and propagate distributions, including expected values and variances.
RNNs are suboptimal at compressing past sensory inputs for future prediction.
problem RNNs do not optimally compress past sensory inputs for future prediction.
method Investigated RNNs trained with maximum likelihood and found they extract unnecessary information. Injected noise into hidden states to improve performance.
result Injecting noise into RNN hidden states improves predictive information, sample quality, likelihood, and classification performance.
New method improves neural network robustness with brain-like learning.
problem Traditional methods struggle with robustness in adversarial attacks.
method Generalized likelihood ratio method with brain-like learning mechanisms.
result Significant improvement in robustness of neural networks.
Introspects convolutional speech recognition models using Gradient-adjusted Neuron Activation Profiles.
problem Lack of interpretability in deep learning ASR models.
method Gradient-adjusted Neuron Activation Profiles (GradNAPs) for feature and representation visualization.
result Gains insight into how data is processed in convolutional ASR models.
New method decomposes sensory information from neurons into specific stimuli and features.
problem Understanding how much and what specific information neurons encode.
method Introduced axioms for meaningful stimulus-wise decomposition and derived a tractable solution using diffusion models.
result Can efficiently estimate contributions of specific stimuli and features to encoded information.
ChaosNet uses chaotic neural networks for classification with high accuracy on limited data.
problem Classification with limited training data.
method ChaosNet architecture with GLS neurons and a novel learning algorithm exploiting topological transitivity.
result ChaosNet achieves high accuracy (73.89% - 98.33%) with as few as 7 training samples per class.
A new neural network model using weighted Lehmer means and multiplets.
problem Creating a neural network that can emulate different generalized mean cases.
method Replacing dot product with weighted Lehmer mean and using multiplets of neurons.
result The network can emulate the exclusive-or problem and approximate functions.
Perceptual capabilities of artificial systems have come a long way since the advent of deep learning. These methods have proven to be effective, however they are not as efficient as their biological counterparts. Visual attention is a set of mechanisms that are employed in biological visual systems to ease computationa…
The paper examines how few neuron failures affect neural network performance.
problem Understanding the impact of random neuron failures on neural network output.
method Probabilistic analysis of neural networks with small random crashes using Taylor expansion in the continuous limit.
result Provable bounds on error in neural network output under small random crashes.
This paper provides an overview of activation functions in neural networks.
problem Confusion in activation function selection and properties in deep learning.
method Analytic review of popular activation functions.
result Clarification of activation function properties and selection.
Fault-tolerant neural networks inspired by biological error correction codes.
problem Achieving reliable computation with unreliable neurons.
method Using biological error correction codes from grid cells in the mammalian cortex to develop a fault-tolerant neural network.
result Noisy biological neurons operate below a fault-tolerance threshold, suggesting a mechanism for reliable computation in the brain.
A new neural model evolves to learn at the synaptic level.
problem Lack of biologically realistic neural models in deep learning.
method Evolve individual neuron and synaptic models using ENUs.
result Evolved neural network learns complex tasks like a T-maze.
Paper introduces supervised and unsupervised TAM models for binary neurons.
problem Learning and retrieval of structured triplets of patterns in neural networks.
method Extends Hebbian paradigm to supervised and unsupervised protocols, using glassy statistical mechanical techniques.
result Obtained self-consistency equations for critical dataset sizes and retrieval performance.
Abstract: Investigates the role of activation functions in neural networks and their physical basis.
problem Understanding the role of activation functions in neural networks and their physical basis.
method Formalizes the use of activation functions in neural inference by relating them to phase transitions in statistical physics.
result Reveals the physical justification for the performance of typical activation functions in neural networks.
Gravitational-wave data analysis is rapidly absorbing techniques from deep learning, with a focus on convolutional networks and related methods that treat noisy time series as images. We pursue an alternative approach, in which waveforms are first represented as weighted sums over reduced bases (reduced-order modeling)…
New method converts ANN gates to SNNs with AMOS neurons for improved image classification.
problem Efficiently converting ANN gates to SNNs for neuromorphic hardware.
method Introducing AMOS conversion for gates in ANNs, improving accuracy and throughput.
result Improved accuracy of SNNs for ImageNet from 74.60% to 80.97%.
Artificial Neural Networks (ANNs) are currently being used as function approximators in many state-of-the-art Reinforcement Learning (RL) algorithms. Spiking Neural Networks (SNNs) have been shown to drastically reduce the energy consumption of ANNs by encoding information in sparse temporal binary spike streams, hence…
To compare entities of differing types and structural components, the artificial neural network paradigm was used to cross-compare structural components between heterogeneous documents. Trainable weighted structural components were input into machine-learned activation functions of the neurons. The model was used for m…
New algorithm shows neural networks can learn without full backpropagation.
problem Stochastic gradient descent with backpropagation is non-biologically plausible.
method Random and fixed backpropagation weights in a feedback alignment algorithm.
result Error converges to zero exponentially fast in overparameterized networks.
A new framework for flexible neural network receptive fields.
problem Adaptive and flexible receptive fields in neural networks.
method Density-embedding layers that replace affine transformations with scalar products of input and density functions.
result Density-embedding layers can adaptively tune receptive fields and are computationally efficient.