Solves internal covariate shift and dying neurons with linked neurons.
problem Internal covariate shift and dying neurons in deep learning.
method Defining linked neurons with two constraints: shared operating point and non-zero gradient.
result Linked neurons effectively solve internal covariate shift and improve training efficiency.
The seemingly stochastic transient dynamics of neocortical circuits observed in vivo have been hypothesized to represent a signature of ongoing stochastic inference. In vitro neurons, on the other hand, exhibit a highly deterministic response to various types of stimulation. We show that an ensemble of deterministic le…
The highly variable dynamics of neocortical circuits observed in vivo have been hypothesized to represent a signature of ongoing stochastic inference but stand in apparent contrast to the deterministic response of neurons measured in vitro. Based on a propagation of the membrane autocorrelation across spike bursts, we …
Automated method finds meaningful directions in neural network activations.
problem Mixed selectivity in neurons makes interpretation challenging.
method Automated quantification of interpretability and discovery of meaningful directions.
result Meaningful directions in neural network activations are more interpretable than individual neurons.
Efficiently estimates neural connectivity using ensemble stimulation.
problem Estimating functional connectivity in large, behaving neural populations.
method Noisy group testing combined with convex optimization and Bayesian inference.
result Connectivity can be inferred with logarithmic growth in tests, even for large networks.
Tool visualizes Transformer model attention for better understanding.
problem Understanding complex attention mechanisms in deep learning models.
method Developed an open-source tool to visualize attention at three levels.
result Visualization helps interpret and analyze Transformer models.
Method reconstructs neuron models from spike times efficiently.
problem Reconstructing neuron models from spike times in degenerate populations.
method Combining deep learning with DICs to map spike times to DIC densities and generate degenerate CBM populations.
result Fast and scalable reconstruction of degenerate populations from spike recordings.
CNNs reveal retinal ganglion cell features, linking visual processing to neuroscience.
problem Understanding what CNNs learn about retinal neuronal circuits.
method Trained CNNs on white noise images to predict neural responses from salamander retinas.
result CNN filters resemble biological retinal components and ganglion cell receptive fields.
New method makes neural networks transparent, revealing learning modes.
problem Lack of interpretability in neural networks.
method Weight pathway analysis (WPA) to decompose neural networks into subnetworks.
result Neural networks store and utilize information holographically, with linear and nonlinear learning modes.
Gradient descent learns a single neuron without knowing the relationship between inputs and labels.
problem Learning a single neuron without knowing the relationship between inputs and labels.
method Using gradient descent to minimize empirical risk over i.i.d. samples, with a nonconvex and nonsmooth optimization problem.
result Gradient descent achieves near-optimal population risk in polynomial time and sample complexity.
A neural network learns to control a two-link arm with non-linear dynamics.
problem Training spiking neural networks to control complex, non-linear systems.
method Feedback-based Online Local Learning Of Weights (FOLLOW) to train a network of spiking neurons with hidden layers.
result The network learns an inverse model of the arm's dynamics and uses it to generate a motor command for control.
EP learns like BPTT but with local weight updates.
problem Existing EP lacks a local time learning rule.
method C-EP updates weights simultaneously with neuron dynamics.
result C-EP follows BPTT gradients and performs well.
BDH model learns like the brain, rivaling Transformer performance.
problem Leveraging brain-like properties for machine learning.
method Scale-free biologically inspired network of neuron particles.
result BDH model achieves Transformer-like performance with interpretability.
Deep P-Spline automates DNN structure selection for complex regression problems.
problem Challenges in selecting optimal network structures for DNNs.
method Linking neuron selection to knot placement in basis expansion techniques, introducing a difference penalty for automated knot selection.
result Deep P-Spline extends model class and forms a latent variable modeling framework with theoretical guarantees.
Neural networks (NN) have achieved state-of-the-art performance in various applications. Unfortunately in applications where training data is insufficient, they are often prone to overfitting. One effective way to alleviate this problem is to exploit the Bayesian approach by using Bayesian neural networks (BNN). Anothe…
Neuron Shapley identifies key neurons in deep networks, improving model accuracy and fairness.
problem Identifying responsible neurons in deep networks for better model performance and fairness.
method Neuron Shapley framework quantifies neuron contributions, accounting for interactions.
result Removing just 30 critical filters can destroy model accuracy, revealing network function.
New q-neurons improve neural network performance.
problem Improving neural network activation functions.
method Introducing q-neurons based on Jackson's q-derivatives with stochastic parameters. result Consistently improved performance over state-of-the-art activation functions.
Describes explaining neurons in deep representations using compositional logical concepts.
problem Interpreting neuron behavior in deep neural networks.
method Identifying compositional logical concepts that closely approximate neuron behavior.
result Compositional explanations provide insights into model performance and allow for adversarial example creation.
SeReNe prunes neurons with low sensitivity to reduce network size.
problem Large neural networks consume too many resources on resource-constrained devices.
method Exploits neural sensitivity as a regularizer to prune neurons with low sensitivity.
result Pruning neurons with low sensitivity achieves competitive compression ratios.
Under-parameterized networks can either copy or average teacher weights, leading to universal optimal solutions.
problem Approximating a teacher network with an under-parameterized student network.
method Analyzing shallow neural networks with erf activation function and unitary teacher weights, proving copy-average configurations are critical points and finding the optimal solution.
result The optimal solution for under-parameterized networks has a universal structure, whether copying or averaging teacher neurons.
This research investigates selectively pruning hyper and hypo neurons to improve neural network generalization.
problem Improving neural network generalization to unseen data.
method Investigates pruning hyper and hypo neurons selectively in fully connected layers of CNNs.
result Selective pruning of hyper and hypo neurons improves model performance on out-of-domain data.
Modeling hidden neurons in SNNs using mesoscopic approximations.
problem Underconstrained problem of modeling unobserved neurons in SNNs.
method Coarse-graining and mean-field approximations to derive neuLVM.
result neuLVM can efficiently model large SNNs and recover connectivity parameters.
Deep networks exhibit permutation saddles and valleys between equivalent minima.
problem Understanding the structure of loss landscapes in deep neural networks.
method Geometric approach to constructing paths between equivalent minima and saddle points.
result Existence of permutation saddles and valleys in deep neural networks.
Topological methods improve neuron analysis and tracer injection summary.
problem Traditional methods fail to capture the tree-like structure of neurons.
method Discrete Morse (DM) Theory for neuron skeletonization and consensus tree summarization.
result Significant performance improvements over non-topological methods.
Developed a BP algorithm for training neural networks with 2nd order neurons.
problem Training neural networks with nonlinear quadratic operations.
method Created a general backpropagation algorithm.
result Validated the generalized BP algorithm through numerical studies.
Quadratic autoencoder improves low-dose CT image denoising.
problem Low-dose CT image denoising.
method Quadratic autoencoder architecture applied to CT denoising.
result Quadratic autoencoder achieves superior denoising performance and efficiency.
Neural network robustness evaluated in the presence of failing neurons.
problem Evaluating robustness of neural networks in the face of neuron failures.
method Leveraging Lipschitz continuity of activation functions, calculating Forward Error Propagation.
result Tight bounds on the number of failing neurons before network accuracy is compromised.
We developed a faster method for calculating neuron importance in neural networks.
problem Assigning importance to individual neurons in deep learning models.
method We developed Neuron Integrated Gradients, a scalable implementation of Total Conductance.
result Neuron Integrated Gradients is faster and empirically stronger than DeepLIFT.
BEAN models neuronal correlations to create interpretable representations.
problem Hard interpretation of dense-layer representations in DNNs.
method Inspired by neuroscience, BEAN models neuronal correlations and dependencies.
result BEAN enables formation of interpretable neuronal clusters without sacrificing model performance.
SpaRCe optimizes reservoir computing by learning neuron thresholds to improve performance and prevent forgetting.
problem Improving performance and preventing forgetting in reservoir computing networks.
method Integrates neuron-specific learnable thresholds to optimize sparsity without altering dynamics, learning read-out weights and thresholds via gradient rule.
result Threshold learning improves performance and alleviates catastrophic forgetting.
It will be shown that according to theorems of K. Menger, every neuron grid if identified with a curve is able to preserve the adopted qualitative structure of a data space. Furthermore, if this identification is made, the neuron grid structure can always be mapped to a subset of a universal neuron grid which is constr…
This work improves DNN interpretability by reducing neuron ambiguity.
problem Lack of interpretability in DNNs, especially in healthcare applications.
method Developed a metric to evaluate neuron consistency, used adversarial examples to identify ambiguous features, and proposed adversarial training to improve consistency.
result Reduced ambiguity of neurons in DNNs, improving interpretability.
A new method to understand neural networks by sampling the 'inverse set' of a neuron.
problem Understanding the internal representation of neurons in neural networks.
method Optimization-based sampling approach to characterize the input space that excites a neuron.
result Inspection of samples reveals regularities that help understand the neuron's representation.
Theory explains how deep nets learn features from data.
problem Understanding how deep neural networks learn features from data.
method Developed a noise-nonlinearity phase diagram and a mechanical theory.
result Links feature learning across layers to generalization.
Neural networks learn to mimic brain neurons with two-input activation functions, improving performance and robustness.
problem Training neural networks to mimic the complex interactions of brain neurons.
method Developed a network-in-network architecture with two-input activation functions, optimized hyperparameters, and compared to conventional ReLU networks.
result Two-input activation functions can learn soft XOR functions, improving network performance and robustness.
CHANI learns classification tasks with local transformations inspired by biology.
problem Proving neural networks can learn classification tasks with local transformations.
method CHANI uses spiking neurons modeled by Hawkes processes with expert aggregation for local learning.
result CHANI can learn and encode multiple classes, forming assemblies of neurons.
BinaryGAN uses binary neurons for GAN training.
problem Training GANs with binary outputs.
method End-to-end backpropagation with sigmoid-adjusted straight-through estimators.
result BinaryGAN can generate binary-valued predictions.
Researchers develop methods to learn neuron dynamics from colored noise.
problem Learning nonlocal stochastic neuron dynamics from colored noise.
method Proposed two methods for closing Fokker-Planck equations: nonlocal large-eddy-diffusivity closure and data-driven sparse regression.
result Mutual information and total correlation between stimulus and neuron states calculated for FHN neuron.
Single neuron with ADA learns XOR and outperforms other functions.
problem Classifying linearly non-separable data.
method Proposed a new artificial neuron with apical dendrite activation.
result ADA function achieves 100% accuracy on XOR and superior performance on benchmark datasets.
We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences.…
Despite our extensive knowledge of biophysical properties of neurons, there is no commonly accepted algorithmic theory of neuronal function. Here we explore the hypothesis that single-layer neuronal networks perform online symmetric nonnegative matrix factorization (SNMF) of the similarity matrix of the streamed data. …
Stable unactivated neurons reduce expressiveness in ReLU networks.
problem Reducing expressiveness in ReLU neural networks due to stably unactivated neurons.
method Investigated the probability of neurons being stably unactivated in ReLU networks with symmetric weight and bias distributions.
result Proved the probability of a neuron being stably unactivated in the second hidden layer of a ReLU network.
Novel chaotic neurons improve AI with minimal training data.
problem Limited training data for AI algorithms.
method Intrinsically chaotic neurons inspired by chaos theory.
result Classification accuracy up to 95.8% with just 2 training samples per class.
Single-spike neurons can approximate as well as multi-spike neurons.
problem Limitation of single-spike neurons in spiking neural networks.
method Comparison of single-spike and multi-spike neural networks.
result Single-spike and multi-spike neural networks are equivalent in approximation capabilities.
Optimal neuron activation functions improve neural network performance.
problem Limited expressive power of standard neuron activation functions in neural networks.
method Additive Gaussian process regression to construct individual neuron activation functions.
result Optimal neuron activation functions lead to better performance and reduced overfitting.
New neuron model learns and adapts its receptive field.
problem Learning and focusing on informative inputs.
method Adaptive locally connected neuron model using backpropagation.
result Focusing neurons outperform dense layers in classification tasks.
Novel 'strong neuron' improves deep learning efficiency and robustness.
problem Improving deep learning efficiency and robustness against adversarial attacks.
method Introducing a novel 'strong neuron' model and a constructive training algorithm.
result Achieved 10x-100x reduction in operations count and hardware requirements.
Deep CNN model predicts neuronal cell health from images.
problem Predicting the biological activity of chemical compounds on neuronal cells.
method Deep convolutional neural network (CNN) with residual connections.
result Achieved 99.6% accuracy in distinguishing treated from untreated cells.