Neural-net-induced Gaussian process (NNGP) regression inherits both the high expressivity of deep neural networks (deep NNs) as well as the uncertainty quantification property of Gaussian processes (GPs). We generalize the current NNGP to first include a larger number of hyperparameters and subsequently train the model…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
25 results for “NNGPs”
The NNGP kernel's predictions closely match those of the Matern kernel under certain conditions.
problem Comparing NNGP kernels to Matern kernels in practical applications.
method Demonstrated the necessity of normalization for NNGP kernels, explored numerical challenges, and compared predictions and performance.
result NNGP kernel predictions closely match Matern kernel predictions under specific circumstances.
Recent work has established the equivalence between deep neural networks and Gaussian processes (GPs), resulting in so-called neural network Gaussian processes (NNGPs). The behaviour of these models depends on the initialisation of the corresponding network. In this work, we consider the impact of noise regularisation …
Proposes scale mixture of NNGPs for more flexible stochastic processes.
problem Limited focus on broadening the class of stochastic processes from NNGPs.
method Scale mixture of NNGPs with scale priors on last-layer parameters.
result Turns neural networks into a richer class of stochastic processes.
The Theory and Practice of Highly Scalable Gaussian Process Regression with Nearest Neighboursstat.ML
Develops a theoretical framework for scalable Gaussian Process regression methods.
problem Limited scalability of Gaussian Process regression for large datasets.
method Introduces and analyzes Nearest Neighbour Gaussian Process (NNGP) and scalable GPnn methods.
result Derives almost sure pointwise limits for predictive criteria and proves risk minimax rates.
Fast variational Bayes methods improve geospatial data analysis speed and accuracy.
problem Inaccurate and slow variational Bayes methods for large geospatial data.
method Combination of calculus of variations, closed-form gradient updates, and linear response corrections.
result Comparable accuracy to spNNGP with reduced computational costs and faster speed.
Graph convolutional deep kernel machine learns representations for graph tasks.
problem Limited representation learning in infinite-width neural networks.
method Developed a graph convolutional deep kernel machine as an infinite-width limit.
result Representation learning improves performance for heterophilous node classification tasks.
Paper connects neural networks to Gaussian processes for understanding double-descent.
problem Understanding the double-descent phenomenon in neural networks.
method Uses techniques from random matrix theory and Gaussian processes.
result Establishes a connection between NNGP and random matrix theory for neural networks.
A new method combines deep kernels with Gaussian processes to avoid overfitting.
problem Losing Bayesian benefits in deep kernel learning due to kernel optimization.
method Using Infinite-width neural networks and Neural Network Gaussian Process (NNGP) as a guide for DKL optimization.
result Robustness to overfitting and good predictive performance on various datasets.
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limitstat.ML
Better uncertainty estimates for neural networks using Gaussian process priors.
problem Poor uncertainty estimates in neural networks, especially on out-of-distribution data.
method Characterize the function-space prior of an ensemble of infinitely-wide neural networks as a Gaussian process and use it to build a probabilistic model.
result The approach improves calibration of neural networks, especially under distributional shift.
Bayesian neural networks explore rare fluctuations for better feature learning.
problem Understanding rare but dominant fluctuations in Bayesian neural networks.
method Large-deviation theory and joint optimization over predictors and internal kernels.
result Posterior rate function optimization reveals data-dependent kernel selection.
This paper explores how kernel methods can explain data effects on neural collapse.
problem Understanding how data affects neural collapse in neural networks.
method Formulating NC1 as a function of kernel, specializing to NNGP and NTK, and exploring a data-aware Gaussian Process kernel.
result The NTK does not represent more collapsed features than the NNGP for Gaussian data, highlighting the limitations of data-independent kernels.
The hedonic approach based on a regression model has been widely adopted for the prediction of real estate property price and rent. In particular, a spatial regression technique called Kriging, a method of interpolation that was advanced in the field of spatial statistics, are known to enable high accuracy prediction i…
There has recently been much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width. However, these results do not apply to architectures that require one or more of the hidden layers to remain n…
Wide neural networks converge to Gaussian processes, improving generalization.
problem Understanding the generalization of wide neural networks, especially deep equilibrium models.
method Investigation of deep equilibrium models (DEQs) with infinite-depth layers, focusing on their convergence to Gaussian processes as width and depth approach infinity.
result Wide DEQs converge to Gaussian processes, maintaining generalization performance.
Attention mechanisms in deep learning become Gaussian process-like as the number of heads increases.
problem Understanding the behavior of attention mechanisms in deep learning models.
method Extending the equivalence between wide neural networks and Gaussian processes to attention architectures.
result Multi-head attention architectures behave as Gaussian processes as the number of heads tends to infinity.
Empirical study compares wide neural networks to kernel methods, resolving open questions.
problem Understanding the relationship between wide neural networks and kernel methods.
method Large-scale empirical study using various neural network architectures and kernel methods.
result Wide neural networks outperform fully-connected finite-width networks in some cases, but underperform convolutional finite-width networks.
Analyzes DNNs trained with noisy gradients, finding FWCs negligible for large n.
problem Analyzing DNNs trained with noisy gradients.
method Introduced analytical framework to analyze non-Gaussian stochastic process.
result FWCs negligible for large n, improving CNN performance.
New method speeds up neural kernel computations for various activations.
problem Inefficient computation of neural kernels for general activations.
method Fast sketching method using truncated Hermite expansion.
result 106x speedup for approximate CNTK computation on CIFAR-10.
Wide Bayesian neural networks have a simple weight posterior: theory and accelerated samplingstat.ML
Wide Bayesian neural networks have a simpler weight posterior, leading to faster MCMC sampling.
problem Sampling from the posterior of wide Bayesian neural networks is challenging.
method Introducing repriorisation, a data-dependent reparameterisation that simplifies the posterior distribution.
result The repriorisation map accelerates MCMC sampling, achieving up to 50x higher effective sample size.
RFAD uses random features to speed up dataset distillation.
problem Efficiently compress large datasets for reduced storage and computation.
method Random feature approximation of the Neural Network Gaussian Process kernel.
result At least 100-fold speedup over KIP with competitive accuracy.
Convolutional DKMs improve kernel methods on MNIST, CIFAR-10, and CIFAR-100.
problem Improving kernel methods for image classification.
method Developed a novel inter-domain inducing point approximation and introduced various techniques to extend DKMs to convolutional networks.
result Achieved state-of-the-art performance on image classification benchmarks.
New theory explains deep learning's success in transforming inputs.
problem Standard theoretical approaches eliminate representation learning.
method Developed a new infinite width limit for representation learning.
result Deep Gaussian processes (DGPs) have multivariate Gaussian posteriors.
New approach predicts generalization of deep neural networks in proportional-width regime.
problem Predicting generalization of deep neural networks in proportional-width regime.
method Equivalent Wishart Ansatz for hierarchical empirical kernels, renormalized NNGP kernel.
result Renormalized NNGP kernel captures dominant stochastic fluctuations in deep neural networks.