Quadratic models explain neural network behavior during training.
problem Understanding neural network dynamics during training with large learning rates.
method Developed and tested Neural Quadratic Models.
result Neural Quadratic Models exhibit the 'catapult phase' similar to neural networks.
Use simplified layerwise linear models to understand neural dynamics.
problem Complex neural network dynamics are hard to grasp.
method Apply simplified layerwise linear models to explain neural phenomena.
result Simplified models explain neural collapse, emergence, etc.
Analyzes neural networks using linear models to understand their behavior.
problem Understanding multi-layer neural networks through linear models.
method Recalls and reviews four models: linear regression with concentrated features, kernel ridge regression, random feature model, and neural tangent model.
result Highlights limitations of linear theory and discusses approaches to overcome them.
Convolutional Neural Processes improve data efficiency in neural processes.
problem Improving data efficiency in neural processes for small datasets.
method Convolutional Neural Processes (ConvNPs) improve data efficiency by leveraging translation equivariance and convolutional neural networks.
result ConvNPs enhance the performance of neural processes in small-data problems.
Probabilistic deep learning uses neural networks and models to handle uncertainty.
problem Handling uncertainty in deep learning models.
method Two approaches: probabilistic neural networks and deep probabilistic models.
result TensorFlow Probability library supports both approaches.
Paper introduces new neural network models and theories.
problem Understanding neural networks beyond over-parameterized regime.
method Develops two exact models and a novel representor theory.
result Provides insights into neural network training and kernel evolution.
Neural GARCH models financial time series with time-varying coefficients.
problem Modeling conditional heteroskedasticity in financial time series.
method Neural network adaptation of GARCH and BEKK models with time-varying coefficients parameterized by a recurrent neural network.
result Neural Students t model consistently outperforms other models on financial time series.
Improved neural topic model for semi-supervised learning.
problem Representing textual data in an interpretable manner with limited labeled data.
method Label-Indexed Neural Topic Model (LI-NTM) that combines deep generative models with semi-supervised learning.
result LI-NTM outperforms existing models in document reconstruction and classifier performance.
DeepWeightFlow generates diverse neural network weights efficiently.
problem Generating complete neural network weights efficiently and accurately.
method Flow Matching in weight space with Git Re-Basin and TransFusion.
result DeepWeightFlow generates high-accuracy neural networks without fine-tuning.
Neural ODEs provide a framework for studying the training dynamics of neural networks.
problem Training dynamics of neural networks
method Dynamical mean field theory
result Derive learning curves in the high-dimensional limit
Quantum neural network and tensor network models outperform classical models in Japanese stock market predictions.
problem Improving stock return predictions using quantum and quantum-inspired machine learning.
method Evaluation of quantum neural network and tensor network models against classical models like linear and neural networks.
result Tensor network model outperforms classical models in Japanese stock market, including linear and neural network models.
Neural networks can learn relationships that traditional models cannot.
problem Identifying factors that differentiate neural networks from traditional models.
method Proving non-identifiability of neural networks compared to smooth parametric models.
result Neural networks can learn nontrivial relationships that traditional models cannot.
Monotonic neural additive models simplify machine learning for credit scoring.
problem Complex machine learning methods make models less transparent and fair.
method Introducing monotonic neural additive models that simplify neural networks while maintaining regulatory compliance.
result Monotonic neural additive models achieve similar accuracy to complex neural networks but are more transparent and fair.
Deep learning models outperform classical methods in forecasting neural activity.
problem Improving forecasting of neural activity using deep learning models.
method Systematic evaluation of eight probabilistic deep learning models against classical statistical models and baseline methods.
result Several deep learning models consistently outperform classical approaches in forecasting neural activity.
New method identifies latent variables in cognitive models using neural networks.
problem Inference of latent variables in complex cognitive models is limited.
method Recurrent neural networks and simulation-based inference for latent variable sequences.
result Extends neural Bayes estimation to broader classes of cognitive models.
Symmetry-regularized Neural ODEs improve model stability and interpretability.
problem Improving the stability and physical interpretability of Neural ODEs.
method Integrating Lie symmetries and conservation laws into the loss function.
result Symmetry-regularized Neural ODEs enhance model stability and interpretability.
Neural population activity often exhibits rich variability and temporal structure. This variability is thought to arise from single-neuron stochasticity, neural dynamics on short time-scales, as well as from modulations of neural firing properties on long time-scales, often referred to as "non-stationarity". To better …
Improved neural population modeling using shared features and ensemble detection.
problem Missing shared coding properties in neural latent variable models.
method Feature sharing across tuning curves and soft clustering of neurons.
result More interpretable and better-performing neural population models.
New algorithm learns switching dynamics from multiple neural signals.
problem Learning accurate switching dynamical system models from multimodal neural data.
method Unsupervised learning algorithm for multiscale switching dynamical system models.
result Switching multiscale dynamical system models outperform single-scale models in behavior decoding.
Neural SDEs model continuous sequences using neural networks.
problem Modeling continuous-time dynamics in sequence data.
method Interprets time-series as samples from a continuous dynamical system, parameterized by Neural SDE.
result Demonstrates superior performance in diverse sequence modeling tasks.
This paper uses ODE to improve RNN models for time series data.
problem Improving RNN models for irregularly sampled time series data.
method Extending RNNs with Neural Ordinary Differential Equations (ODEs).
result New ODE-based RNN models reduce training and evaluation time.
New neural network models for functional data.
problem Handling non-linear functional data.
method Functional Direct Neural Network (FDNN) and Functional Basis Neural Network (FBNN) with gradient-based optimization.
result Demonstrated effectiveness in complex functional models.
Stochastic neural networks with infinite width become deterministic, reducing training variance.
problem Understanding how stochasticity in neural networks affects learning and regularization.
method Theoretical analysis of stochastic neural networks with infinite width.
result As the width of an optimized stochastic neural network increases, its predictive variance on the training set decreases to zero.
New method sparsifies hybrid neural ODEs for better performance and stability.
problem Excessive latent states and interactions from mechanistic models lead to training inefficiency and over-fitting.
method Automatic state selection and structure optimization combining domain-informed graph modifications with data-driven regularization.
result Improved predictive performance and robustness with desired sparsity.
Gaussian process models simplify neural network behavior for easier understanding.
problem Understanding and predicting the behavior of deep learning systems.
method Constructing surrogate models using Gaussian processes from finite neural networks.
result Surrogate models capture phenomena like spectral bias and predict generalization well.
Neural model outperforms ETAS in forecasting Central Apennines earthquakes.
problem Short-term seismicity forecasting with incomplete data.
method Extended a neural network model to the magnitude domain, using it to forecast earthquakes above a target magnitude threshold.
result Neural model outperforms ETAS at lower magnitude thresholds, due to its robustness to missing data.
RCNPs extend equivariant neural processes to higher dimensions, improving performance on tasks with inherent symmetries.
problem Inherently equivariant tasks in spatio-temporal modeling, Bayesian Optimization, and continuous control.
method Relational Conditional Neural Processes (RCNPs) that extend equivariances to higher dimensions.
result Empirically competitive performance on tasks with equivariances.
NAMLSS models provide interpretable neural regression for location, scale, and shape.
problem Lack of interpretability in deep learning models for complex data distributions.
method Combines classical statistical methods with DNNs for distributional regression.
result Achieves visual interpretability and predictive power of deep learning models.
Deep neural-kernel models combine neural networks and kernel machines for scalable large datasets.
problem Combining neural networks and kernel machines for efficient large-scale learning.
method Hybrid neural-kernel architecture using explicit feature mapping and pooling layers.
result The deep neural-kernel models are effective and scalable on benchmark datasets.
How can we efficiently propagate uncertainty in a latent state representation with recurrent neural networks? This paper introduces stochastic recurrent neural networks which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model. The c…
Proposes a neural network for contextual regression.
problem Improving model efficiency and interpretability in regression with contextual features.
method Simple contextual neural network (SCtxtNN) that separates context identification from context-specific regression.
result SCtxtNN achieves lower excess mean squared error and more stable performance than feed-forward neural networks.
We propose two neural network based mixture models in this article. The proposed mixture models are explicit in nature. The explicit models have analytical forms with the advantages of computing likelihood and efficiency of generating samples. Computation of likelihood is an important aspect of our models. Expectation-…
GNP models predictive correlations and outperforms NPs.
problem Training and understanding of Neural Processes.
method Proposed a new model, Gaussian Neural Process (GNP), which incorporates translation equivariance and provides universal approximation guarantees.
result Demonstrates encouraging performance and provides universal approximation guarantees.
Neural SVEs model complex systems with memory, outperforming traditional methods.
problem Modeling systems with memory effects and irregular behavior.
method Introducing neural stochastic Volterra equations as a physics-inspired architecture.
result Neural SVEs outperform neural SDEs and DeepONets in various applications.
Extends Neural ODEs to model discrete changes in continuous systems.
problem Lack of explicit termination time in existing Neural ODE formulations.
method Introduces neural event functions to implicitly define termination criteria.
result Models discrete changes in continuous systems without prior knowledge.
Neural jump model improves option pricing accuracy.
problem Jump risk in option pricing.
method Neural jump stochastic differential equation model with Gumbel-Softmax gradient learning.
result Neural jump components significantly improve option pricing accuracy.
DRIFT uses neural flows to replace distributional regression models.
problem Lack of neural network representations for distributional regression models.
method Inverse flow transformations (DRIFT) for distributional regression.
result Neural representations in DRIFT match classical statistical methods in performance.
Abstract: Investigates the role of activation functions in neural networks and their physical basis.
problem Understanding the role of activation functions in neural networks and their physical basis.
method Formalizes the use of activation functions in neural inference by relating them to phase transitions in statistical physics.
result Reveals the physical justification for the performance of typical activation functions in neural networks.
Study compares neural and statistical models for Parkinson's disease progression from voice data.
problem Difficult statistical analysis of longitudinal voice biomarkers due to subject correlation, small cohorts, and varied disease trajectories.
method Evaluated Neural Mixed Effects (NME), Generalized Neural Network Mixed Models (GNMMs), and semi-parametric Generalized Additive Mixed Models (GAMMs).
result GAMMs achieve stronger predictive performance and retain interpretable smooth effects and subject-level structure.
The paper connects neural networks to Mahalanobis distance for interpretability.
problem Lack of interpretability in neural networks.
method Establishes a connection between neural network linear layers and Mahalanobis distance.
result Provides a foundation for more interpretable neural network models.
We develop a neural network model to classify liver cancer patients into high-risk and low-risk groups using genomic data. Our approach provides a novel technique to classify big data sets using neural network models. We preprocess the data before training the neural network models. We first expand the data using wavel…
Research on neural networks has gained significant momentum over the past few years. Because training is a resource-intensive process and training data cannot always be made available to everyone, there has been a trend to reuse pre-trained neural networks. As such, neural networks themselves have become research data.…
Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous ne…
New model predicts neural network performance from early training epochs, incorporating architecture impact.
problem Predicting neural network performance from early training epochs, neglecting architecture impact.
method Architecture-aware graph ordinary differential equation model.
result Model outperforms state-of-the-art methods for MLP and CNN learning curves.
Paper explains neural collapse in neural networks using a new model.
problem Understanding neural collapse in neural networks during training.
method Introducing the unconstrained layer-peeled model (ULPM) to prove gradient flow convergence to critical points of a minimum-norm separation problem.
result Proves that all critical points are strict saddle points except the global minimizers exhibiting neural collapse.
Deep neural networks have been increasingly used in software engineering and program analysis tasks. They usually take a program and make some predictions about it, e.g., bug prediction. We call these models neural program analyzers. The reliability of neural programs can impact the reliability of the encompassing anal…
Bayesian neural networks help quantify prediction uncertainties in neural models.
problem Uncertainty in neural network predictions due to model randomness and lack of knowledge.
method Bayesian statistical framework to categorize uncertainty.
result Errors in neural network predictions can be obtained and characterized.
Smartfluidnet accelerates Eulerian fluid simulation with neural networks.
problem Current neural network methods for Eulerian fluid simulation lack flexibility and generalization.
method Smartfluidnet automates model generation and dynamic switching to meet user requirements.
result Smartfluidnet achieves 1.46x and 590x speedup compared to state-of-the-art models, with better simulation quality.