Biological neurons learn tensor decompositions of higher-order correlations using nonlinear Hebbian plasticity.
problem Learning higher-order correlations in biological neurons.
method Introduce and study generalized nonlinear Hebbian learning rules.
result Neurons can learn tensor eigenvectors of higher-order input correlation tensors.
Neural networks learn to mimic brain neurons with two-input activation functions, improving performance and robustness.
problem Training neural networks to mimic the complex interactions of brain neurons.
method Developed a network-in-network architecture with two-input activation functions, optimized hyperparameters, and compared to conventional ReLU networks.
result Two-input activation functions can learn soft XOR functions, improving network performance and robustness.
The artificial neural network is a popular framework in machine learning. To empower individual neurons, we recently suggested that the current type of neurons could be upgraded to 2nd order counterparts, in which the linear operation between inputs to a neuron and the associated weights is replaced with a nonlinear qu…
T-SVM improves learning in spiking neurons by maximizing dynamical margin.
problem Finding robust solutions in spiking neuronal networks with temporal correlations.
method Introduces Temporal Support Vector Machine (T-SVM) to maximize dynamical margin.
result T-SVM enables learning of tasks requiring nonlinear spatial integration.
Neural networks learn task-specific features, influenced by nonlinearity.
problem Understanding the nature of task-dependent feature learning in neural networks.
method Investigation of fully-connected, wide neural networks using Bayesian framework.
result The nature of internal representations depends on neuronal nonlinearity, leading to analog, redundant, or sparse coding schemes.
Paper improves neural interaction modeling using nonlinear Hawkes processes.
problem Inability of classic Hawkes process to model inhibitory interactions.
method Augmented auxiliary latent variables and EM algorithm for efficient inference.
result Demonstrates accurate and efficient estimation of neural interaction dynamics.
Optimistic estimate predicts best fitting performance of nonlinear models.
problem Evaluating the potential of nonlinear models in fitting.
method Proposes an optimistic estimate to quantify the smallest sample size for fitting nonlinear models.
result Predicts specific subsets of targets that can be fitted at overparameterization.
One conjecture in both deep learning and classical connectionist viewpoint is that the biological brain implements certain kinds of deep networks as its back-end. However, to our knowledge, a detailed correspondence has not yet been set up, which is important if we want to bridge between neuroscience and machine learni…
Classical models describe primary visual cortex (V1) as a filter bank of orientation-selective linear-nonlinear (LN) or energy models, but these models fail to predict neural responses to natural stimuli accurately. Recent work shows that models based on convolutional neural networks (CNNs) lead to much more accurate p…
Optimization geometry affects deep learning performance.
problem The impact of optimization geometry on deep learning performance.
method Analysis of pseudogradient methods for learning generalized linear models.
result Non-asymptotic bounds on generalization error characterize model performance.
HNHN learns from hypergraphs with hyperedge neurons for better classification.
problem Learning from hypergraphs with complex relationships.
method Hypergraph convolution network with hyperedge neurons and adaptive normalization.
result Improved classification accuracy and speed compared to state-of-the-art methods.
Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation units (DEUs), an improvement to modern neural networks, which enables each neuron to learn a particular nonlinear activation functio…
Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation units (DEUs), an improvement to modern neural networks, which enables each neuron to learn a particular nonlinear activation functio…
Develops CLDS models to model neural activity with nonlinear dynamics.
problem Complex, nonlinear dynamics in neural population activity.
method Conditionally Linear Dynamical System (CLDS) models using Gaussian Process (GP) priors.
result CLDS models can perform well even in data-limited conditions.
In order to classify linearly non-separable data, neurons are typically organized into multi-layer neural networks that are equipped with at least one hidden layer. Inspired by some recent discoveries in neuroscience, we propose a new model of artificial neuron along with a novel activation function enabling the learni…
New method makes neural networks transparent, revealing learning modes.
problem Lack of interpretability in neural networks.
method Weight pathway analysis (WPA) to decompose neural networks into subnetworks.
result Neural networks store and utilize information holographically, with linear and nonlinear learning modes.
It has been widely assumed that a neural network cannot be recovered from its outputs, as the network depends on its parameters in a highly nonlinear way. Here, we prove that in fact it is often possible to identify the architecture, weights, and biases of an unknown deep ReLU network by observing only its output. Ever…
PINNs solve neuronal parameter and state estimation problems with limited data.
problem Estimating parameters and hidden state variables from noisy partial data in multiscale neuronal models.
method Physics-informed neural networks (PINNs) for joint state and parameter estimation.
result PINNs deliver robust and accurate parameter inference and state reconstruction, even with limited data.
Study on neuron dynamics for XOR classification with zero-margin.
problem Understanding neural network training dynamics in zero-margin classification problems.
method Analysis of Gaussian XOR problem, focusing on neuron block dynamics and generalization without margin assumptions.
result Neurons cluster into four directions and block-level signals evolve coherently, essential for reliable prediction in the Gaussian setting.
Neuronal circuits formed in the brain are complex with intricate connection patterns. Such complexity is also observed in the retina as a relatively simple neuronal circuit. A retinal ganglion cell receives excitatory inputs from neurons in previous layers as driving forces to fire spikes. Analytical methods are requir…
Information in neural networks is represented as weighted connections, or synapses, between neurons. This poses a problem as the primary computational bottleneck for neural networks is the vector-matrix multiply when inputs are multiplied by the neural network weights. Conventional processing architectures are not well…
Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scal…
This paper explores adaptive neural activation in RNNs for better learning.
problem Fixed neural activation functions limit the performance and adaptability of RNNs.
method Developed a novel parametric family of nonlinear activation functions inspired by biological neurons.
result Adaptive neural activation improves learning speed and performance in RNNs.
In this paper, we introduce the notion of liquid time-constant (LTC) recurrent neural networks (RNN)s, a subclass of continuous-time RNNs, with varying neuronal time-constant realized by their nonlinear synaptic transmission model. This feature is inspired by the communication principles in the nervous system of small …
Theory explains how deep nets learn features from data.
problem Understanding how deep neural networks learn features from data.
method Developed a noise-nonlinearity phase diagram and a mechanical theory.
result Links feature learning across layers to generalization.
Bayesian method improves predictions in overparameterized nonlinear regression.
problem Understanding overparameterization in nonlinear regression models.
method Bayesian framework with adaptive prior considering data spectral structure.
result Posterior contraction established for generalized linear and single-neuron models, demonstrating prediction consistency.
Morphological neurons, that is morphological operators such as dilation and erosion with learnable structuring elements, have intrigued researchers for quite some time because of the power these operators bring to the table despite their simplicity. These operators are known to be powerful nonlinear tools, but for a gi…
The recent discovered spatial-temporal information processing capability of bio-inspired Spiking neural networks (SNN) has enabled some interesting models and applications. However designing large-scale and high-performance model is yet a challenge due to the lack of robust training algorithms. A bio-plausible SNN mode…
Analyzes minima of deep linear networks with weight decay.
problem Understanding the loss landscape of deep neural networks.
method Analytical solutions for global minima with weight decay and stochastic neurons.
result The origin is a special point with qualitatively different minima in networks with more than 1 hidden layer.
Generalizes memory and forecasting capacities for nonlinear recurrent networks with dependent inputs.
problem Understanding memory and forecasting capabilities in networks with dependent inputs.
method Formulated bounds for memory and forecasting capacities in terms of network size and input properties.
result Proved that memory capacity for linear recurrent networks with independent inputs is given by the rank of the controllability matrix.
Study uses machine learning to predict nonlinear seismic brace behavior.
problem Predicting nonlinear seismic response of structural braces.
method State-of-the-art machine learning techniques, specifically LSTM, were used.
result LSTM method effectively captures nonlinear brace behavior.
Learning and memory in the brain are implemented by complex, time-varying changes in neural circuitry. The computational rules according to which synaptic weights change over time are the subject of much research, and are not precisely understood. Until recently, limitations in experimental methods have made it challen…
We propose a general statistical framework for clustering multiple time series that exhibit nonlinear dynamics into an a-priori-unknown number of sub-groups. Our motivation comes from neuroscience, where an important problem is to identify, within a large assembly of neurons, subsets that respond similarly to a stimulu…
Estimates nonlinear Hawkes processes using RKHSs with ReLU rectification.
problem Nonlinear multivariate Hawkes processes with complex interaction functions.
method Nonparametric estimation using RKHSs with approximations for ReLU and integral operators.
result Proposes an estimation method with bounds on approximation errors.
This work analyzes autoencoders to understand feature learning, revealing different phases of learning.
problem Understanding the feature extraction mechanism in weight-tied autoencoders.
method Mean field analysis of two-layer weight-tied autoencoders trained with stochastic gradient descent.
result The training dynamics of autoencoders exhibit different phases corresponding to learning different principal subspaces of the data.
This paper presents a new family of backpropagation-free neural architectures, Gated Linear Networks (GLNs). What distinguishes GLNs from contemporary neural networks is the distributed and local nature of their credit assignment mechanism; each neuron directly predicts the target, forgoing the ability to learn feature…
NAST generalizes scattering transform for non-stationary time series analysis.
problem Analyzing non-stationary time series data.
method Neural activation of scattering transform with various activation functions and high pass filters.
result Central and non-central limit theorems for NAST of Gaussian processes.
Paper accelerates nonlinear mapping in online systems with lower time complexity.
problem Speeding up nonlinear mapping in online systems.
method Integrates an acceleration module into Dendrite Net (DD) to reduce time complexity.
result DD with AC has lower time complexity while maintaining nonlinear mapping and system identification properties.
Deep P-Spline automates DNN structure selection for complex regression problems.
problem Challenges in selecting optimal network structures for DNNs.
method Linking neuron selection to knot placement in basis expansion techniques, introducing a difference penalty for automated knot selection.
result Deep P-Spline extends model class and forms a latent variable modeling framework with theoretical guarantees.
Framework analyzes neural network dynamics for better understanding and optimization.
problem Understanding the fundamental mechanisms of deep neural networks.
method Dynamical systems theory, transformation units, attraction basins.
result Different transformation modes lead to distinct learning phases and network performance.
The paper designs neural networks with assurance for controlling nonlinear systems.
problem Designing neural networks with assurance for nonlinear system control.
method Bounding the number of affine functions needed for a CPWA function, connecting it to a TLL NN architecture.
result The TLL NN architecture is parameterized by the number of affine functions in the CPWA function it realizes.
Bayesian neural networks improve macroeconomic forecasting and model nonlinearities.
problem Handling small T, big K macroeconomic datasets with temporal dependence.
method Developed Bayesian neural networks with mixture activation functions, shrinkage priors, and stochastic volatility.
result BNNs produce precise density forecasts, often better than other methods.
New method for nonlinear Granger causality improves predictive relationships.
problem Challenges in applying Granger causality to nonlinear data.
method Permutation of covariate set, artificial neural networks, consistent variance estimation.
result Permutation method outperforms other techniques in predicting nonlinear relationships.
Echo state networks are computationally lightweight reservoir models inspired by the random projections observed in cortical circuitry. As interest in reservoir computing has grown, networks have become deeper and more intricate. While these networks are increasingly applied to nontrivial forecasting tasks, there is a …
Study shows how activation functions impact the storage capacity of treelike neural networks.
problem Understanding the role of activation functions in neural network expressive power.
method Analysis of treelike two-layer networks with various activation functions in the infinite-width limit.
result Activation functions affect storage capacity and robustness, with nonlinearity increasing capacity and decreasing robustness.
Panda predicts chaotic systems without retraining, showing emergent properties.
problem Predicting chaotic systems with small errors.
method Trained on a synthetic dataset of chaotic dynamical systems using evolutionary algorithms.
result Panda predicts unseen chaotic systems with zero-shot learning.
Describes explaining neurons in deep representations using compositional logical concepts.
problem Interpreting neuron behavior in deep neural networks.
method Identifying compositional logical concepts that closely approximate neuron behavior.
result Compositional explanations provide insights into model performance and allow for adversarial example creation.
SeReNe prunes neurons with low sensitivity to reduce network size.
problem Large neural networks consume too many resources on resource-constrained devices.
method Exploits neural sensitivity as a regularizer to prune neurons with low sensitivity.
result Pruning neurons with low sensitivity achieves competitive compression ratios.