We propose a new generic type of stochastic neurons, called -neurons, that considers activation functions based on Jackson's -derivatives with stochastic parameters . Our generalization of neural network architectures with -neurons is shown to be both scalable and very easy to implement. We demonstrate expe…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “stochastic neurons”
Researchers develop methods to learn neuron dynamics from colored noise.
problem Learning nonlocal stochastic neuron dynamics from colored noise.
method Proposed two methods for closing Fokker-Planck equations: nonlocal large-eddy-diffusivity closure and data-driven sparse regression.
result Mutual information and total correlation between stimulus and neuron states calculated for FHN neuron.
The seemingly stochastic transient dynamics of neocortical circuits observed in vivo have been hypothesized to represent a signature of ongoing stochastic inference. In vitro neurons, on the other hand, exhibit a highly deterministic response to various types of stimulation. We show that an ensemble of deterministic le…
A machine learning framework predicts self-induced stochastic resonance in neurons.
problem Predicting coherent oscillations in slow-fast excitable systems driven by noise.
method Physics-informed machine learning with a Noise-Augmented State Predictor architecture and Kramers' escape theory constraints.
result Trained PINN accurately predicts spike-train coherence on noise intensity, excitability, and timescale separation.
In this paper, a simple, general method of adding auxiliary stochastic neurons to a multi-layer perceptron is proposed. It is shown that the proposed method is a generalization of recently successful methods of dropout (Hinton et al., 2012), explicit noise injection (Vincent et al., 2010; Bishop, 1995) and semantic has…
The highly variable dynamics of neocortical circuits observed in vivo have been hypothesized to represent a signature of ongoing stochastic inference but stand in apparent contrast to the deterministic response of neurons measured in vitro. Based on a propagation of the membrane autocorrelation across spike bursts, we …
Persistent Neuronscs.LG
Persistent neurons improve neural network optimization by leveraging previous solutions.
problem Improving neural network optimization under different initialization and data distributions.
method Persistent neurons use information from previous converged solutions to explore new landscapes and avoid local minima.
result Persistent neurons converge to more optimal solutions and improve model performance under various initializations.
Binary Stochastic Filtering (BSF), the algorithm for feature selection and neuron pruning is proposed in this work. The method defines filtering layer which penalizes amount of the information involved in the training process. This information could be the input data or output of the previous layer, which directly lead…
This work makes vision networks more interpretable by identifying key neurons.
problem Complex deep networks are hard to interpret, hindering safety-critical applications.
method Uses stochastic local competition and multimodal models to dissect and interpret neurons.
result The method generates understandable descriptions for a few active neurons, revealing network decision-making.
OPT framework improves neural network generalization by learning an orthogonal transformation.
problem Improving neural network generalization.
method Orthogonal over-parameterized training (OPT) framework that minimizes hyperspherical energy.
result OPT framework provably minimizes hyperspherical energy and improves empirical generalization.
We present two Bayesian procedures to infer the interactions and external currents in an assembly of stochastic integrate-and-fire neurons from the recording of their spiking activity. The first procedure is based on the exact calculation of the most likely time courses of the neuron membrane potentials conditioned by …
Detection and skeletonization of single neurons and tracer injections using topological methodscs.CV
Topological methods improve neuron analysis and tracer injection summary.
problem Traditional methods fail to capture the tree-like structure of neurons.
method Discrete Morse (DM) Theory for neuron skeletonization and consensus tree summarization.
result Significant performance improvements over non-topological methods.
Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scal…
Improved Bayesian neural network inference by selectively removing redundant modes.
problem Redundant modes in Bayesian neural network posteriors complicate approximate inference.
method Structured partial stochasticity and deterministic subset selection of weights.
result Improved performance of approximate inference schemes with simplified posterior distribution.
Pavlov Learning Machinescond-mat.dis-nn
Paper models Pavlov's classical conditioning using stochastic processes and Langevin equations.
problem Lack of modeling for Pavlov's classical conditioning.
method Modeling neural and synaptic dynamics via Langevin equations.
result Pavlov's mechanism spontaneously leads to synaptic weights similar to Hebb's.
Study on neuron dynamics for XOR classification with zero-margin.
problem Understanding neural network training dynamics in zero-margin classification problems.
method Analysis of Gaussian XOR problem, focusing on neuron block dynamics and generalization without margin assumptions.
result Neurons cluster into four directions and block-level signals evolve coherently, essential for reliable prediction in the Gaussian setting.
Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities? In this work, we uncover a phenomenon in which the behavior of these complex networks -- …
Inference of stochastic time series with missing dataphysics.data-an
Method infers dynamics from incomplete time series data.
problem Challenges in inferring stochastic dynamics from time series with missing data.
method Expectation Maximization (EM) algorithm that iterates between E-step and M-step.
result The EM algorithm effectively recovers missing data points and infers underlying network models from real neuronal activities.
Artificial Neural Networks (ANNs) are currently being used as function approximators in many state-of-the-art Reinforcement Learning (RL) algorithms. Spiking Neural Networks (SNNs) have been shown to drastically reduce the energy consumption of ANNs by encoding information in sparse temporal binary spike streams, hence…
Analyzes minima of deep linear networks with weight decay.
problem Understanding the loss landscape of deep neural networks.
method Analytical solutions for global minima with weight decay and stochastic neurons.
result The origin is a special point with qualitatively different minima in networks with more than 1 hidden layer.
Asymptotic properties of one-layer artificial neural networks with sparse connectivitycond-mat.dis-nn
Law derived for neural networks with sparse connections.
problem Understanding the behavior of neural networks with sparse connections.
method Law of large numbers for empirical distribution of parameters derived.
result Law for neural networks with sparse connections derived.
Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generationcs.LG
It has been shown recently that deep convolutional generative adversarial networks (GANs) can learn to generate music in the form of piano-rolls, which represent music by binary-valued time-pitch matrices. However, existing models can only generate real-valued piano-rolls and require further post-processing, such as ha…
One conjecture in both deep learning and classical connectionist viewpoint is that the biological brain implements certain kinds of deep networks as its back-end. However, to our knowledge, a detailed correspondence has not yet been set up, which is important if we want to bridge between neuroscience and machine learni…
Improves conditions for mode connectivity in deep neural networks.
problem Ensuring mode connectivity in deep neural networks under practical conditions.
method Exploiting feature quality and mild over-parameterization conditions.
result Connectivity of solutions found by stochastic gradient descent confirmed under new conditions.
We propose stochastic, non-parametric activation functions that are fully learnable and individual to each neuron. Complexity and the risk of overfitting are controlled by placing a Gaussian process prior over these functions. The result is the Gaussian process neuron, a probabilistic unit that can be used as the basic…
We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences.…
Stochasticity is key for machine learning's robustness and generalizability.
problem Machine learning's need for robustness and generalizability.
method Review of ML literature and biological intelligence.
result Stochasticity is a critical ingredient for intelligent systems in ML.
A new SNN model explains decision-making with learning and spiking neurons.
problem Lack of learning mechanism in existing models for decision-making.
method Proposes a Spiking Neural Network (SNN) model that incorporates a learning mechanism and uses multivariate Hawkes processes.
result Shows a coupling between DDM and Poisson counter models and derives a DDM from a Hawkes network of spiking neurons.
Fast reconstruction of degenerate populations of conductance-based neuron models from spike timesq-bio.NC
Method reconstructs neuron models from spike times efficiently.
problem Reconstructing neuron models from spike times in degenerate populations.
method Combining deep learning with DICs to map spike times to DIC densities and generate degenerate CBM populations.
result Fast and scalable reconstruction of degenerate populations from spike recordings.
Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networkscs.LG
The optimization of multilayer neural networks typically leads to a solution with zero training error, yet the landscape can exhibit spurious local minima and the minima can be disconnected. In this paper, we shed light on this phenomenon: we show that the combination of stochastic gradient descent (SGD) and over-param…
The apparent stochasticity of in-vivo neural circuits has long been hypothesized to represent a signature of ongoing stochastic inference in the brain. More recently, a theoretical framework for neural sampling has been proposed, which explains how sample-based inference can be performed by networks of spiking neurons.…
Dropout has been witnessed with great success in training deep neural networks by independently zeroing out the outputs of neurons at random. It has also received a surge of interest for shallow learning, e.g., logistic regression. However, the independent sampling for dropout could be suboptimal for the sake of conver…
Neural plasticity is an important functionality of human brain, in which number of neurons and synapses can shrink or expand in response to stimuli throughout the span of life. We model this dynamic learning process as an -norm regularized binary optimization problem, in which each unit of a neural network (e.g., …
Neuron Shapley identifies key neurons in deep networks, improving model accuracy and fairness.
problem Identifying responsible neurons in deep networks for better model performance and fairness.
method Neuron Shapley framework quantifies neuron contributions, accounting for interactions.
result Removing just 30 critical filters can destroy model accuracy, revealing network function.
Describes explaining neurons in deep representations using compositional logical concepts.
problem Interpreting neuron behavior in deep neural networks.
method Identifying compositional logical concepts that closely approximate neuron behavior.
result Compositional explanations provide insights into model performance and allow for adversarial example creation.
New method for online learning in interacting particle systems.
problem Parameter estimation in stochastic interacting particle systems.
method Stochastic approximation of gradient of asymptotic log likelihood using continuous observations.
result Convergence to stationary points of asymptotic log-likelihood under suitable assumptions.
SeReNe prunes neurons with low sensitivity to reduce network size.
problem Large neural networks consume too many resources on resource-constrained devices.
method Exploits neural sensitivity as a regularizer to prune neurons with low sensitivity.
result Pruning neurons with low sensitivity achieves competitive compression ratios.
Under-parameterized networks can either copy or average teacher weights, leading to universal optimal solutions.
problem Approximating a teacher network with an under-parameterized student network.
method Analyzing shallow neural networks with erf activation function and unitary teacher weights, proving copy-average configurations are critical points and finding the optimal solution.
result The optimal solution for under-parameterized networks has a universal structure, whether copying or averaging teacher neurons.
Modeling hidden neurons in SNNs using mesoscopic approximations.
problem Underconstrained problem of modeling unobserved neurons in SNNs.
method Coarse-graining and mean-field approximations to derive neuLVM.
result neuLVM can efficiently model large SNNs and recover connectivity parameters.
Inspired by complexity and diversity of biological neurons, our group proposed quadratic neurons by replacing the inner product in current artificial neurons with a quadratic operation on input data, thereby enhancing the capability of an individual neuron. Along this direction, we are motivated to evaluate the power o…
New framework models neural systems with random architecture on manifolds.
problem Complex, uncertain systems with non-Gaussian outputs.
method Latent random field on compact manifold generates neural architecture and weights.
result Synthetic neural systems can produce stochastic outputs for deterministic inputs.
A Note on Topology Preservation in Classification, and the Construction of a Universal Neuron Gridcs.NE
It will be shown that according to theorems of K. Menger, every neuron grid if identified with a curve is able to preserve the adopted qualitative structure of a data space. Furthermore, if this identification is made, the neuron grid structure can always be mapped to a subset of a universal neuron grid which is constr…
ISAAC Newton uses input-based curvature for efficient training.
problem Efficient training in small-batch stochastic regimes.
method ISAAC Newton conditions gradients using selected second-order information based on input.
result Effective training even in small-batch stochastic regimes, competitive to first-order and second-order methods.
Neural networks learn to mimic brain neurons with two-input activation functions, improving performance and robustness.
problem Training neural networks to mimic the complex interactions of brain neurons.
method Developed a network-in-network architecture with two-input activation functions, optimized hyperparameters, and compared to conventional ReLU networks.
result Two-input activation functions can learn soft XOR functions, improving network performance and robustness.
Unseen or out-of-domain data can seriously degrade the performance of a neural network model, indicating the model's failure to generalize to unseen data. Neural net pruning can not only help to reduce a model's size but can improve the model's generalization capacity as well. Pruning approaches look for low-salient ne…
CHANI learns classification tasks with local transformations inspired by biology.
problem Proving neural networks can learn classification tasks with local transformations.
method CHANI uses spiking neurons modeled by Hawkes processes with expert aggregation for local learning.
result CHANI can learn and encode multiple classes, forming assemblies of neurons.
Study shows how SGD in large neural networks behaves as neurons increase.
problem Understanding SGD behavior in overparameterized neural networks.
method Probabilistic approach to continuous-time dynamics of SGD, focusing on particle interactions.
result Particles' interactions asymptotically vanish, leading to a mean-field limit.
Despite our extensive knowledge of biophysical properties of neurons, there is no commonly accepted algorithmic theory of neuronal function. Here we explore the hypothesis that single-layer neuronal networks perform online symmetric nonnegative matrix factorization (SNMF) of the similarity matrix of the streamed data. …