The paper examines how spike strengths and alignments affect overfitting in linear regression models.
problem The impact of spike strengths and alignments on overfitting in linear regression models.
method Characterization of generalization error through exact expressions and analysis of spike strengths, aspect ratio, and target alignment.
result Increasing spike strength can lead to catastrophic overfitting before benign overfitting, especially in well-specified aligned problems.
Develops generic spike-and-slab priors for high-dimensional linear regression.
problem Bayesian high-dimensional linear regression challenges.
method Proposes a class of generic spike-and-slab priors and a unified framework for theoretical assessment.
result Achieves nearly-optimal posterior contraction rate and model selection consistency under general conditions.
Paper proposes a new method for Bayesian linear regression using spike-and-slab priors.
problem Identifying predictors with similar relationships in linear regression models.
method Hierarchical Bayesian models with spike-and-slab priors and a Gibbs sampler.
result The proposed method outperforms previous methods in simulations and real data analysis.
New algorithms improve Bayesian linear regression with spike-and-slab priors.
problem Efficiently sampling from Bayesian linear regression models with sparsity-inducing priors.
method Design of two sampling algorithms: Gibbs sampling and Stochastic Localization.
result Stochastic Localization sampler shows significant advantage for poorly designed data matrices.
Spike-and-slab priors are improved for high-dimensional Bayesian regression.
problem Prohibitive computational costs for existing samplers in high-dimensional settings.
method Proposes Scalable Spike-and-Slab (S3) for high-dimensional Bayesian regression. result Improves computational cost to max{n2pt,np} per iteration, demonstrating significant speed-ups and quality gains. Improved VB algorithm for high-dimensional logistic regression with theoretical guarantees.
problem Sparse high-dimensional logistic regression model selection.
method Spike and slab variational Bayes approximation.
result Optimal convergence rates in ℓ2 and prediction loss for sparse truths. The study explains delayed spikes in batch-normalized models.
problem Delayed spikes in batch-normalized models.
method Analyzing batch-normalized linear models, deriving conditions for delayed onset and waiting time.
result Explicit conditions for delayed-onset and waiting time in whitened square-loss linear regression.
This paper analyzes generalization for linear models with spiked covariance structures.
problem Understanding the generalization performance of linear models with spiked covariance structures.
method Derives the generalization error for two simple models with spiked covariances using random matrix theory.
result The eigenvector and eigenvalue corresponding to the spike significantly influence the generalization error.
Bayesian neural network achieves nearly optimal performance in Besov space.
problem Bayesian neural networks in Besov space.
method Spike-and-slab prior and shrinkage prior for posterior convergence rate.
result The posterior convergence rate is nearly minimax and adaptive to unknown smoothness.
Over the past decade, deep neural networks (DNNs) have demonstrated remarkable performance in a variety of applications. As we try to solve more advanced problems, increasing demands for computing and power resources has become inevitable. Spiking neural networks (SNNs) have attracted widespread interest as the third-g…
Unified framework for training SNNs using EP, faster convergence.
problem Training spiking neural networks with Expectation-Propagation.
method Message-passing framework for learning marginal distributions of SNN parameters.
result Faster convergence compared to gradient-based methods.
New algorithms sample spike-and-slab priors efficiently in high dimensions.
problem Sampling from spike-and-slab priors in high-dimensional settings.
method Provably efficient algorithms for posterior sampling with sublinear measurement count.
result First provable algorithms for spike-and-slab posterior sampling without strong SNR assumptions.
A new method discovers equations from data using Bayesian and kernel techniques.
problem Discovering equations from data is hard due to sparsity and noise.
method Kernel regression for function estimation and Bayesian spike-and-slab prior for uncertainty quantification.
result KBASS method outperforms state-of-the-art methods on benchmark tasks.
A novel Bayesian method for dynamic sparsity in Gaussian dynamic linear regression.
problem Variable selection and shrinkage in time-varying regression models.
method Time-varying sparsity via Markov switching priors for coefficients' variances, extending spike-and-slab priors.
result Induces smoothness or shrinkage towards zero at each time point, leading to improved model performance.
Self-distillation optimally improves model performance in spiked covariance models.
problem Improving model performance in spiked covariance models.
method Developed spectral shrinkage estimators and analyzed self-distillation.
result Self-distillation achieves optimal performance among spectral shrinkage estimators for spiked covariance matrices.
Efficiently identifies important variables in binary outcomes using variational Bayes.
problem Bayesian variable selection for binary outcomes with computational challenges.
method Mean-field variational Bayes approximation with closed-form updates and efficient inference algorithm.
result Successfully identifies important variables and is orders of magnitude faster than MCMC.
Bayesian model predicts phenotype effects from multi-environmental factors.
problem Predict phenotype effects from multi-environmental trials.
method Bayesian tensor regression with spike-and-slab structure.
result Model outperforms previous methods in simulation and real-world data.
Single-spike neurons can approximate as well as multi-spike neurons.
problem Limitation of single-spike neurons in spiking neural networks.
method Comparison of single-spike and multi-spike neural networks.
result Single-spike and multi-spike neural networks are equivalent in approximation capabilities.
In this paper a new Bayesian model for sparse linear regression with a spatio-temporal structure is proposed. It incorporates the structural assumptions based on a hierarchical Gaussian process prior for spike and slab coefficients. We design an inference algorithm based on Expectation Propagation and evaluate the mode…
A dynamic Boltzmann machine (DyBM) has been proposed as a model of a spiking neural network, and its learning rule of maximizing the log-likelihood of given time-series has been shown to exhibit key properties of spike-timing dependent plasticity (STDP), which had been postulated and experimentally confirmed in the fie…
Much of studies on neural computation are based on network models of static neurons that produce analog output, despite the fact that information processing in the brain is predominantly carried out by dynamic neurons that produce discrete pulses called spikes. Research in spike-based computation has been impeded by th…
Exact inference in the linear regression model with spike and slab priors is often intractable. Expectation propagation (EP) can be used for approximate inference. However, the regular sequential form of EP (R-EP) may fail to converge in this model when the size of the training set is very small. As an alternative, we …
Neurons perform computations, and convey the results of those computations through the statistical structure of their output spike trains. Here we present a practical method, grounded in the information-theoretic analysis of prediction, for inferring a minimal representation of that structure and for characterizing its…
RFMs transition from linear to nonlinear under specific input-label correlation.
problem Understanding the transition from linear to nonlinear behavior in RFMs.
method Analyzing RFMs under spiked covariance designs, characterizing the interaction between anisotropy and input-label correlation.
result The RFM generalization error is governed by the strength of input-label correlation, leading to a clear nonlinear advantage above a specific boundary.
SNNs enhance high-frequency price spike forecasting in HFT environments.
problem Conventional financial models fail to capture fine temporal structure in high-frequency price spikes.
method Application of Spiking Neural Networks (SNNs) with hyperparameter tuning via Bayesian Optimization (BO).
result SNN models optimized with PSA achieve significantly higher cumulative returns in backtesting.
SGD converges with perturbed forward-backward passes, explained by geometric amplification.
problem Analyzing convergence of SGD with perturbed forward-backward passes in composite optimization.
method Characterized propagation and amplification of perturbations, derived convergence guarantees for non-convex and PL objectives.
result Perturbations cascade through the computational graph, affecting convergence order under specific conditions.
SNNs can represent complex functions efficiently.
problem Understanding the representational power of SNNs.
method Viewed as sequence-to-sequence processors, analyzed using spike train functions.
result SNNs have the universal representation property for certain functions.
Third-generation neural networks, or Spiking Neural Networks (SNNs), aim at harnessing the energy efficiency of spike-domain processing by building on computing elements that operate on, and exchange, spikes. In this paper, the problem of training a two-layer SNN is studied for the purpose of classification, under a Ge…
Anisotropic data structure affects learning dynamics and generalization error in linear networks.
problem Understanding the impact of data anisotropy on learning dynamics and generalization error in linear networks.
method Examined a spiked covariance structure as a model of anisotropy in a two-layer linear network in a linear regression setting.
result Learning dynamics proceed in two phases: initially driven by input-output correlation, then by other principal directions of the data structure. Derived an analytical expression for the generalization error.
Affine spiking neural networks learn efficiently and generalize well.
problem Learning with spiking neural networks, especially with positive weights.
method Affine encoders and decoders, continuous parameter dependence, gradient-based training.
result Affine spiking neural networks can approximate shallow ReLU networks and generalize well.
Developing electrophysiological recordings of brain neuronal activity and their analysis provide a basis for exploring the structure of brain function and nervous system investigation. The recorded signals are typically a combination of spikes and noise. High amounts of background noise and possibility of electric sign…
Extracting and detecting spike activities from the fluorescence observations is an important step in understanding how neuron systems work. The main challenge lies in that the combination of the ambient noise with dynamic baseline fluctuation, often contaminates the observations, thereby deteriorating the reliability o…
New method improves neural spike train models by minimizing divergence directly, leading to better performance.
problem Poor performance and divergence issues in spike train models using maximum likelihood estimation.
method Directly minimize maximum mean discrepancy using spike train kernels and stochastic optimization.
result The proposed method generates well-behaved models with better control over feature trade-offs.
New method detects spike-and-wave epileptiform discharges using Kendall's Tau-b.
problem Detecting spike-and-wave epileptiform discharges in EEG signals.
method Proposes a new method based on Kendall's Tau-b coefficient.
result High Specificity and rule in (SpPIn) for spike-and-wave discharge detection.
Bayesian inference corrected for bias in high-dimensional models.
problem Bayesian inference for high-dimensional regression models often produces biased credible sets.
method Debiasing approach based on Bernstein-von Mises theorem.
result Frequentist validity of debiased Bayesian posterior.
T2FSNN improves deep SNNs by reducing spikes and latency.
problem Inefficiency in spiking neural networks due to lack of scalable training algorithms.
method Introduces time-to-first-spike coding with kernel-based dynamic threshold and dendrite, combined with gradient-based optimization and early firing methods.
result Reduces inference latency and spike count by 22% and less than 1% compared to burst coding.
A central problem of random matrix theory is to understand the eigenvalues of spiked random matrix models, introduced by Johnstone, in which a prominent eigenvector (or "spike") is planted into a random matrix. These distributions form natural statistical models for principal component analysis (PCA) problems throughou…
Improved SNNs with quantized activations outperform traditional networks.
problem Maintaining SotA accuracy in SNNs with limited bit precision.
method Interpolating between non-spiking and spiking regimes using signal processing tools.
result First hybrid SNN outperforms traditional RNNs in accuracy with reduced bit precision.
A learning rule for first-spike times in neural networks reduces energy consumption and reaction times.
problem Energy efficiency and reaction time in neuromorphic systems.
method Derivation of a learning rule for first-spike times in leaky integrate-and-fire neurons, using only input and output spike times.
result Demonstrated that the approach can implement error backpropagation in hierarchical spiking networks and is capable of harnessing neuromorphic system's speed and energy characteristics.
New method uses surrogate gradients to train efficient spiking networks on neuromorphic hardware.
problem Training high-performing spiking networks on analog neuromorphic hardware is challenging due to device mismatch and lack of efficient algorithms.
method Introduces a general in-the-loop learning framework based on surrogate gradients.
result Learning self-corrects for device mismatch, resulting in competitive spiking network performance.
This research bridges binary and spiking neural networks for efficient on-chip AI.
problem Reducing compute requirements in machine learning frameworks.
method Training Spiking Neural Networks in extreme quantization regime and utilizing standard training techniques for conversion.
result Training Spiking Neural Networks in extreme quantization regime achieves near full precision accuracies.
A vast majority of computation in the brain is performed by spiking neural networks. Despite the ubiquity of such spiking, we currently lack an understanding of how biological spiking neural circuits learn and compute in-vivo, as well as how we can instantiate such capabilities in artificial spiking circuits in-silico.…
Study recovers spike order in noisy tensor estimation without SNR assumptions.
problem Estimating multiple signal vectors from noisy tensor observations.
method Gradient flow optimization of a nonconvex function.
result Determines sample complexity for efficient permutation recovery.
This paper suggests a learning-theoretic perspective on how synaptic plasticity benefits global brain functioning. We introduce a model, the selectron, that (i) arises as the fast time constant limit of leaky integrate-and-fire neurons equipped with spiking timing dependent plasticity (STDP) and (ii) is amenable to the…
Algorithm detects and estimates correlated signals in spiked matrices.
problem Detect and estimate correlated signals in spiked matrices.
method Proposes an efficient algorithm based on counting edge-decorated cycles.
result Algorithm succeeds under certain signal-to-noise ratio conditions.
Accurate statistical models of neural spike responses can characterize the information carried by neural populations. But the limited samples of spike counts during recording usually result in model overfitting. Besides, current models assume spike counts to be Poisson-distributed, which ignores the fact that many neur…
Spiking neural networks (SNNs) offer a promising alternative to current artificial neural networks to enable low-power event-driven neuromorphic hardware. Spike-based neuromorphic applications require processing and extracting meaningful information from spatio-temporal data, represented as series of spike trains over …
Spiking neural networks perform similarly to deep networks on occluded images.
problem Robust object recognition in partially occluded images.
method Developed a two-layer spiking neural network trained on natural scenes with a biologically plausible learning rule, compared to deep convolutional networks.
result Spiking neural networks achieve good accuracy and robustness on stepwise pixel erasement tasks.