Analyzes neural networks using linear models to understand their behavior.
problem Understanding multi-layer neural networks through linear models.
method Recalls and reviews four models: linear regression with concentrated features, kernel ridge regression, random feature model, and neural tangent model.
result Highlights limitations of linear theory and discusses approaches to overcome them.
Use simplified layerwise linear models to understand neural dynamics.
problem Complex neural network dynamics are hard to grasp.
method Apply simplified layerwise linear models to explain neural phenomena.
result Simplified models explain neural collapse, emergence, etc.
DeConvNet, Guided BackProp, LRP, were invented to better understand deep neural networks. We show that these methods do not produce the theoretically correct explanation for a linear model. Yet they are used on multi-layer networks with millions of parameters. This is a cause for concern since linear models are simple …
New neural network models for functional data.
problem Handling non-linear functional data.
method Functional Direct Neural Network (FDNN) and Functional Basis Neural Network (FBNN) with gradient-based optimization.
result Demonstrated effectiveness in complex functional models.
Neural networks improve geospatial data analysis by relaxing linearity assumptions.
problem Traditional geospatial analysis assumes linear models, limiting flexibility.
method Embedding neural networks within traditional geostatistical models for non-linear mean functions.
result NN-GLS algorithm provides consistent and scalable predictions for irregular spatial data.
Paper proposes LANN to measure model complexity of neural networks with curve activation functions.
problem Measuring model complexity of neural networks with curve activation functions.
method Proposes LANN, a piecewise linear framework to approximate curve activation functions, and derives complexity measure based on the number of linear regions.
result Demonstrates positive correlation between overfitting and model complexity during training.
Wide neural networks become linear, with constant tangent kernel, due to Hessian scaling.
problem Understanding the linearity of large non-linear models and the tangent kernel.
method Analyzing the scaling properties of the Hessian matrix of neural networks as their width increases.
result The constancy of the tangent kernel is due to the scaling properties of the Hessian matrix.
In recent years we see a rapidly growing line of research which shows learnability of various models via common neural network algorithms. Yet, besides a very few outliers, these results show learnability of models that can be learned using linear methods. Namely, such results show that learning neural-networks with gr…
DebiNet uses over-parameterized neural networks to improve linear model performance and debiasing.
problem Improving linear model performance and debiasing in high-dimensional settings.
method Incorporates over-parameterized neural networks into semi-parametric models to estimate parameters consistently.
result DebiNet offers valid inference and accurate prediction by leveraging neural networks' universal approximation and linear model's interpretability.
A body of recent work in modeling neural activity focuses on recovering low-dimensional latent features that capture the statistical structure of large-scale neural populations. Most such approaches have focused on linear generative models, where inference is computationally tractable. Here, we propose fLDS, a general …
LUNO linearizes neural operators to quantify their predictive uncertainty.
problem Quantifying the predictive error of neural operators for high-stakes simulations.
method Model linearization to push weight-space uncertainty forward to predictions.
result LUNO provides a practical and theoretically sound way to apply Bayesian methods to neural operators.
Develops CLDS models to model neural activity with nonlinear dynamics.
problem Complex, nonlinear dynamics in neural population activity.
method Conditionally Linear Dynamical System (CLDS) models using Gaussian Process (GP) priors.
result CLDS models can perform well even in data-limited conditions.
Quadratic models explain neural network behavior during training.
problem Understanding neural network dynamics during training with large learning rates.
method Developed and tested Neural Quadratic Models.
result Neural Quadratic Models exhibit the 'catapult phase' similar to neural networks.
Improved electricity price forecasting model combining linear and non-linear structures.
problem Day-ahead electricity price forecasting in energy systems.
method Recurrent neural networks with embedded linear structures.
result Approximately 11% higher accuracy than state-of-the-art models.
A new complexity measure for neural networks improves upon classical methods.
problem Lack of a refined complexity measure for comparing different neural network architectures, especially permutation-invariant ones.
method Introduced an equivalence relation among linear functions and counted them relative to this relation.
result The new complexity measure clearly distinguishes between different models and increases exponentially with depth.
Proposes a new linearity-based neural network compression method.
problem Reduction of neural network model size while maintaining accuracy.
method Integrates linearity-based intuition with ReLU activation functions to merge layers.
result Achieves up to 75% reduction in model size without loss of accuracy.
We study the neural-linear bandit model for solving sequential decision-making problems with high dimensional side information. Neural-linear bandits leverage the representation power of deep neural networks and combine it with efficient exploration mechanisms, designed for linear contextual bandits, on top of the last…
Deep networks become equivalent to linear models in large data regimes.
problem Understanding the behavior of deep neural networks in large data regimes.
method Information-theoretic analysis of fully-trained neural networks in proportional scaling regime.
result Proves deep Gaussian equivalence principle, showing deep networks can be simplified to linear models.
A new estimator learns sparse linear models with context-dependent coefficients.
problem Sparse linear models lack flexibility compared to deep neural networks for handling feature groups.
method Contextual lasso estimator using a deep neural network with lasso regularization.
result Learned models can be sparser than standard lasso without sacrificing predictive power.
DREAM model improves computational efficiency for non-linear effects in relational event models.
problem Efficiently modeling non-linear effects in dynamic relational networks.
method Introduces Deep Relational Event Additive Model (DREAM) using Neural Additive Models.
result Demonstrates superior computational efficiency compared to traditional REM approaches.
Study clarifies Bayesian generalization error in CBM for 3-layered linear neural networks.
problem Understanding the generalization error in concept bottleneck models.
method Mathematical analysis of Bayesian generalization error and free energy in CBM for 3-layered linear neural networks.
result CBM significantly alters the parameter region and Bayesian generalization error compared to standard models.
Recurrent neural networks can learn complex transduction problems that require maintaining and actively exploiting a memory of their inputs. Such models traditionally consider memory and input-output functionalities indissolubly entangled. We introduce a novel recurrent architecture based on the conceptual separation b…
Quantum neural network and tensor network models outperform classical models in Japanese stock market predictions.
problem Improving stock return predictions using quantum and quantum-inspired machine learning.
method Evaluation of quantum neural network and tensor network models against classical models like linear and neural networks.
result Tensor network model outperforms classical models in Japanese stock market, including linear and neural network models.
This paper explains double descent in linear neural networks, identifying new factors.
problem Understanding double descent in linear neural networks.
method Gradient flow derivation and necessary conditions for double descent.
result Singular values of input-output covariance matrix are important for double descent in two-layer models.
This paper improves Bayesian neural nets by using local linearization.
problem Underfitting in Bayesian neural networks.
method Local linearization of Bayesian neural networks to create a generalized linear model (GLM) for predictions.
result The GLM predictive resolves common underfitting problems of the Laplace approximation.
Sparse linear models improve neural network debuggability.
problem Improving neural network interpretability and debugging.
method Using sparse linear models over learned deep feature representations.
result The approach leads to more debuggable and accurate neural networks.
NGSLL combines DNN accuracy with linear model interpretability.
problem Combining high accuracy of DNNs with interpretability of linear models.
method Neural generators of sparse local linear models (NGSLL) using DNNs to approximate non-linear functions.
result Effective in real-world datasets, achieving high predictive performance and interpretability.
The neural linear model is a simple adaptive Bayesian linear regression method that has recently been used in a number of problems ranging from Bayesian optimization to reinforcement learning. Despite its apparent successes in these settings, to the best of our knowledge there has been no systematic exploration of its …
KalmanNet uses neural networks to improve state estimation in systems with unknown dynamics.
problem State estimation of systems with non-linear dynamics and partial information.
method KalmanNet integrates a recurrent neural network with the Kalman filter to handle non-linearities and model mismatches.
result KalmanNet outperforms classic filtering methods in systems with both mismatched and accurate domain knowledge.
New neural network models for complex functional data analysis.
problem Complex relations between functional predictors and responses.
method Function-on-Function regression models using neural networks with continuous hidden layers.
result Demonstrated power and flexibility in handling complex functional models.
Estimates the upper bound of linear regions in spheres centered at specific data points in ReLU neural networks.
problem Bounding the number of linear regions in specific areas of neural networks using ReLU activations.
method Developed a method to estimate the upper bound of linear regions in any sphere within the input space of a ReLU neural network.
result The boundaries of linear regions move away from training data points during training, and spheres centered at these points contain more regions than arbitrary points.
FastCPH efficiently predicts survival times using neural networks.
problem Improving efficiency and accuracy of Cox proportional hazards models in neural networks.
method Developed FastCPH, a linear-time method supporting Breslow and Efron methods for tied events.
result Outperforms existing CoxPH approaches and selects useful covariates.
CANN models improve insurance claim count predictions using telematics data.
problem Improving insurance claim count predictions with telematics data.
method Combining classical actuarial models with neural networks for telematics data.
result CANN models outperform traditional models in predicting insurance claims.
Analyzes minima of deep linear networks with weight decay.
problem Understanding the loss landscape of deep neural networks.
method Analytical solutions for global minima with weight decay and stochastic neurons.
result The origin is a special point with qualitatively different minima in networks with more than 1 hidden layer.
Solla discusses neural processing using statistical physics and Bayesian methods.
problem Understanding neural information processing through statistical physics.
method Bayesian inference, Gibbs description, Generalized Linear Models, dimensionality reduction.
result Connection between neural processing and statistical physics.
New algorithms improve contextual bandits with neural networks and energy models.
problem Inefficient exploration in non-linear models for contextual bandits.
method Maximum entropy exploration using neural networks and energy models.
result Both techniques outperform standard algorithms, with energy models best overall.
Weak correlations explain linear dynamics in deep learning models.
problem Understanding the linear structure in gradient-based learning algorithms.
method Characterization of weak correlations between derivatives and parameters.
result Weak correlations are the underlying principle for linearization in deep learning models.
Kernel methods and MLPs perform similarly to linear models in high dimensions.
problem Understanding the performance of kernel methods and MLPs in high-dimensional settings.
method Analysis of kernel methods and MLPs in a high-dimensional regime with proportional asymptotics.
result Linear models are optimal in high-dimensional settings when data is generated by kernel models with nonlinear relationships.
The paper introduces Causal Neural Operators to approximate operators in stochastic analysis.
problem Leveraging temporal structure in non-linear operators for deep learning models.
method Designing a deep learning model framework for infinite-dimensional linear metric spaces.
result Causal Neural Operators can uniformly approximate Hölder or smooth trace class operators.
PiNets provide faithful explanations for neural networks.
problem Lack of true explanations for neural network predictions.
method Pointwise-interpretable Networks (PiNets) that form linear models instance-wise.
result PiNets offer explanations that are meaningful, aligned, robust, and sufficient.
Develops a method to model neural dynamics with flexible yet interpretable latent states.
problem Capturing complex nonlinear dynamics in neural time series while maintaining interpretability.
method Gaussian Process Switching Linear Dynamical System (gpSLDS) that balances expressiveness and interpretability.
result Favorable performance in comparison to rSLDS on synthetic and real neuroscience data.
We present a novel approach to modelling and learning vector fields from physical systems using neural networks that explicitly satisfy known linear operator constraints. To achieve this, the target function is modelled as a linear transformation of an underlying potential field, which is in turn modelled by a neural n…
A scalable method for Bayesian inference in large linear models.
problem High computational cost in Bayesian linear models for large networks.
method Sample-based inference and g-prior for hyperparameter selection.
result Linearised neural network inference on large datasets (ResNet-18, ResNet-50, U-Net).
Paper investigates Lipschitz constants of self-attention modules in neural networks.
problem Lipschitz constants of self-attention modules in neural networks.
method Proved standard dot-product self-attention is not Lipschitz for unbounded input domain. Proposed L2 self-attention that is Lipschitz. Derived upper bound on L2 self-attention's Lipschitz constant.
result Proved standard self-attention is not Lipschitz for unbounded input domain and proposed an alternative L2 self-attention that is Lipschitz.
New findings show DNC is not optimal for deep models, revealing a low-rank bias.
problem Theoretical limitations of DNC in non-linear models and multi-class classification.
method Analysis of non-linear models of arbitrary depth in multi-class classification.
result DNC stops being optimal for DUFM when going beyond two layers or two classes, due to a low-rank bias.
Neural networks with DAGs show linearity as width increases.
problem Understanding linearity in neural networks with arbitrary DAG structures.
method Analyzing the transition to linearity in networks with arbitrary DAGs, characterizing width by minimum in-degree.
result General neural networks with DAGs exhibit linearity as width approaches infinity.
Paper compresses deep neural networks by eliminating redundant neurons.
problem Challenges in deploying deep learning models due to high parameter count.
method Exploits non-linear redundancy to compress neural networks without loss.
result Reduces network size by up to 99% with minimal performance loss.
Enhances non-life insurance pricing models using transformer models.
problem Improving predictive power of non-life insurance pricing models.
method Enhances actuarial non-life models with transformer models for tabular data.
result Transformer models outperform benchmark models in claim frequency prediction.