Deep Gaussian Processes are reinterpreted as deep trigonometric networks for tractable inference.
problem Challenging inference in DGPs due to intractable marginalization in latent function space.
method Viewing DGPs as deep trigonometric networks with Bochner's theorem, and using the wide limit with a bottleneck to translate DGPs into deep trigonometric networks.
result The weight space view yields the same effective covariance functions as obtained in function space, and varying prior distributions over network parameters is equivalent to employing different kernels.
The correlation length-scale next to the noise variance are the most used hyperparameters for the Gaussian processes. Typically, stationary covariance functions are used, which are only dependent on the distances between input points and thus invariant to the translations in the input space. The optimization of the hyp…
Optimal classifiers derived from GMMs are approximated by deep neural networks.
problem Binary classification of high-dimensional overlapping Gaussian mixtures.
method Closed-form expressions for Bayes optimal decision boundaries derived from GMMs' eigenstructure. Empirical validation through synthetic and real-world data.
result Deep neural networks approximate optimal classifiers for GMMs, with decision thresholds related to covariance eigenvectors.
Proposes DGCN with trajectory sampling for data-efficient policy search in MBRL.
problem Improving data efficiency in model-based reinforcement learning.
method Combines trajectory sampling and DGCN for uncertainty propagation in probabilistic world models.
result Improves sample-efficiency over other uncertainty propagation methods and probabilistic models.
A-BLINK speeds up Gaussian process covariance estimation.
problem Slow covariance matrix inversion in Gaussian processes.
method Two pre-trained neural networks learn Kriging weights and spatial variance.
result Significant computational speedups and posterior inference.
Recursive KalmanNet combines neural networks with Kalman filters for precise state estimation.
problem State estimation in systems with noisy measurements and non-Gaussian noise.
method Recursive KalmanNet uses a recurrent neural network to estimate states with consistent error covariance, optimizing for Gaussian negative log-likelihood.
result Recursive KalmanNet outperforms conventional Kalman filters and deep learning-based estimators in non-Gaussian noise conditions.
New kernels from ELU and GELU networks reveal non-trivial fixed points.
problem Understanding fixed-point dynamics in deep neural networks with ELU and GELU activations.
method Deriving covariance functions and analyzing fixed-point dynamics of ELU and GELU networks.
result ELU and GELU networks exhibit non-trivial fixed-point dynamics, explaining implicit regularization in overparameterized models.
Wide deep neural networks with Gaussian weights approximate Gaussian processes closely.
problem Understanding the approximation of deep neural networks with Gaussian weights to Gaussian processes.
method Established novel rates for the Gaussian approximation of random deep neural networks with Gaussian parameters and Lipschitz activation functions in the wide limit.
result The distance between the network output and the Gaussian approximation scales inversely with the width of the network.
Theoretical limits of deep residual networks show consistent covariance structures.
problem Understanding the limits of deep residual networks.
method Analyzing the behavior of deep residual networks with skip connections as width and depth approach infinity.
result Theoretical analysis confirms that the covariance structure remains consistent regardless of the order of width and depth.
The study proves Gaussian universality of deep random features learning.
problem Understanding the test error in deep random features learning.
method Proving Gaussian universality of test error in ridge regression and arbitrary convex losses.
result Sharp asymptotic formula for test error in ridge regression setting.
Compared with global average pooling in existing deep convolutional neural networks (CNNs), global covariance pooling can capture richer statistics of deep features, having potential for improving representation and generalization abilities of deep CNNs. However, integration of global covariance pooling into deep CNNs …
Novel neural GP kernels learn stable, flexible covariance structures.
problem Scalable and flexible covariance kernels for Gaussian processes.
method Directly learn kriging coefficients and conditional standard deviations using deep neural architectures exploiting permutation-equivariant structure.
result Improved training stability and data efficiency with expressive, non-stationary kernels.
Characterizes test error in learning with deep, structured feature maps.
problem Characterizing test error in learning with deep, structured feature maps.
method Asymptotic analysis of feature covariance and population covariance.
result Closed-form formula for feature covariance in Gaussian rainbow neural networks.
Study on random matrices in deep neural networks using Gaussian data.
problem Distribution of singular values in product of random matrices in deep learning.
method Free probability theory combined with standard techniques of random matrix theory.
result Justification for applying free probability theory to non-independent random data matrices.
Bayesian deep neural networks converge to processes with α-stable marginals under infinite variance weights.
problem Representation learning in deep kernel processes is hindered by deterministic covariance kernels.
method Showed convergence to α-stable processes with conditionally Gaussian representations in infinite-width networks.
result Conditional random covariance kernels can be recursively linked, even if the process is α-stable.
Choosing appropriate architectures and regularization strategies for deep networks is crucial to good predictive performance. To shed light on this problem, we analyze the analogous problem of constructing useful priors on compositions of functions. Specifically, we study the deep Gaussian process, a type of infinitely…
Recent work has established the equivalence between deep neural networks and Gaussian processes (GPs), resulting in so-called neural network Gaussian processes (NNGPs). The behaviour of these models depends on the initialisation of the corresponding network. In this work, we consider the impact of noise regularisation …
Compact Gaussian model approximates deep ensemble predictions.
problem Efficiently approximating deep ensemble models for image prediction.
method Sparse-structured multivariate Gaussian with Cholesky parameterization trained to match pre-trained ensemble outputs.
result Compact representation captures uncertainty and structured correlations explicitly.
The paper calibrates the G2++ model using deep learning for interest rates.
problem Calibrating interest rate models with deep learning.
method Calibrated G2++ model using Neural Networks trained on covariances and correlations of Zero-Coupon and Forward rates.
result Deep learning calibration outperforms classic methods.
STACI uses neural nets to estimate spatio-temporal fields with valid uncertainty quantification.
problem Scalable spatio-temporal deep learning models fail to capture underlying correlation structure.
method Variational Bayesian neural network approximation of non-stationary spatio-temporal Gaussian Process (GP) with conformal inference.
result STACI provides accurate prediction intervals for spatio-temporal processes, outperforming competing methods.
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of eva…
Deep kernel learning combines the non-parametric flexibility of kernel methods with the inductive biases of deep learning architectures. We propose a novel deep kernel learning model and stochastic variational inference procedure which generalizes deep kernel learning approaches to enable classification, multi-task lea…
We introduce a variational Bayesian neural network where the parameters are governed via a probability distribution on random matrices. Specifically, we employ a matrix variate Gaussian \cite{gupta1999matrix} parameter posterior distribution where we explicitly model the covariance among the input and output dimensions…
Neural network method estimates covariate-dependent graphical models with statistical guarantees.
problem Estimating graph structure from covariate-dependent data.
method Neural network approach that allows flexible functional dependency on covariates.
result Theoretical PAC guarantees for the method's performance.
Bayesian neural networks with dependent weights converge to Gaussian mixtures.
problem Limitations of standard Gaussian priors in neural networks.
method Posterior analysis with Gaussian likelihood for networks with dependent weights.
result Posterior distribution identified in the wide-width limit, ensuring invertibility of random covariance matrix.
Efficient inference for multimodal Gaussian mixture models of interacting dynamical systems.
problem Efficient inference for multimodal distributions in stochastic dynamical systems.
method Graph neural networks with moment matching for sample-free inference and structured covariance approximations.
result Sample-free inference with improved efficiency and stability compared to Monte Carlo alternatives.
This paper shows how infinitely wide Tensor Networks converge to Gaussian Processes.
problem Understanding the relationship between Tensor Networks and Gaussian Processes.
method Analyzing the infinite-width limit of Tensor Networks and comparing them to Gaussian Processes.
result Infinitely wide Tensor Networks converge to Gaussian Processes, proving their equivalence.
Proposes a transformer model with geostatistical inductive bias for spatio-temporal forecasting.
problem Combining probabilistic rigor of geostatistics with flexible deep learning representations.
method Spatially-informed transformer with learnable covariance kernel.
result Successfully recovers spatial decay parameters end-to-end via backpropagation.
Improves Gaussian process models for large datasets.
problem Selecting proper covariance functions in Gaussian processes.
method Nonparametric process convolutions and deep GP models.
result Improves performance on benchmarks for GPs, especially for larger datasets.
DeepKriging uses DNNs to predict spatial data with improved accuracy and scalability.
problem Predicting spatial processes with non-linear and non-Gaussian data.
method Adds an embedding layer of spatial coordinates with basis functions to DNNs.
result DeepKriging provides non-linear predictions with smaller approximation errors and is scalable for large datasets.
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …
Bayesian neural networks use ridgelet prior for uncertainty quantification.
problem Combining strong predictive performance with uncertainty quantification in Bayesian neural networks.
method Proposes a ridgelet prior that approximates a Gaussian process covariance function in the output space of the network.
result Establishes universality property allowing Bayesian neural networks to approximate any Gaussian process.
Novel deep Gaussian process improves predictive uncertainty.
problem Flexible probabilistic data representations with tractable inference.
method Structured Gaussian variational family with marginalisation.
result Improved accuracy and calibrated uncertainty estimates.
Study on random matrices in deep neural networks with IID entries.
problem Distribution of singular values in product of random matrices for deep neural networks.
method Random matrix theory with a streamlined approach for non-Gaussian data.
result Generalization of macroscopic universality property to non-Gaussian data.
A new model combines deep learning and Gaussian Processes with hyperdata learning.
problem Combining deep learning and Gaussian Processes for expressive and robust learning.
method Conditional Deep Gaussian Process (DGP) with hyperdata learning and approximate inference.
result Conditional DGP offers better expressiveness and robustness compared to existing methods.
Neural networks speed up covariance estimation in spatial Gaussian processes.
problem Efficiently estimating covariance parameters in spatial Gaussian processes.
method Training neural networks to approximate maximum likelihood estimates.
result Neural network estimates are as accurate as ML methods but much faster.
Physics-informed deep learning for PDEs solves forward and inverse problems efficiently.
problem Solving forward and inverse problems in parametric PDEs efficiently and accurately.
method Physics-informed deep latent variable model (PDDLVM) combining deep neural networks, probabilistic modelling, and variational inference.
result Achieves up to three orders of magnitude speed-up compared to traditional FEM while providing coherent uncertainty estimates.
SVDD and Deep SVDD improve radar target detection in clutter.
problem Clutter and thermal noise degrade classical radar detection methods.
method Support Vector Data Description (SVDD) and Deep SVDD for one-class learning.
result SVDD and Deep SVDD outperform traditional methods on simulated radar data.
Neural-net-induced Gaussian process (NNGP) regression inherits both the high expressivity of deep neural networks (deep NNs) as well as the uncertainty quantification property of Gaussian processes (GPs). We generalize the current NNGP to first include a larger number of hyperparameters and subsequently train the model…
We study the robustness of classifiers to various kinds of random noise models. In particular, we consider noise drawn uniformly from the ℓ_p ball for p∈[1,∞] and Gaussian noise with an arbitrary covariance matrix. We characterize this robustness to random noise in terms of the distance to the decisio…
This paper improves causal inference using deep neural networks for low-dimensional covariates.
problem Improving causal inference with deep learning for high-dimensional covariates.
method Doubly robust off-policy learning with deep neural networks on low-dimensional manifolds.
result Nonasymptotic regret bounds for finite- and continuous-action scenarios, converging at a fast rate depending on intrinsic manifold dimension.
We propose SWA-Gaussian (SWAG), a simple, scalable, and general purpose approach for uncertainty representation and calibration in deep learning. Stochastic Weight Averaging (SWA), which computes the first moment of stochastic gradient descent (SGD) iterates with a modified learning rate schedule, has recently been sho…
Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution. In such situations, existing methods usually resort to a diagonal approximation of the covariance matrix despite, the fact that these mat…
The paper provides non-asymptotic Edgeworth expansions for neural network outputs.
problem Approximating deviations of finite-width neural networks from their Gaussian limit.
method Multidimensional Edgeworth expansions of arbitrary order for neural network outputs.
result Established a bound on the total variation distance between neural network output and its Edgeworth approximation.
We establish large deviation principles for convolutional neural networks.
problem Understanding the behavior of convolutional neural networks in the infinite-channel limit.
method We establish large deviation principles for convolutional neural networks under Gaussian prior and posterior distributions.
result We provide a large deviation principle for the sequence of conditional covariance matrices and the posterior distribution.
ConvNets improve nonstationary covariance estimation for large-scale spatial data.
problem Estimating nonstationary spatial covariance functions on large scales.
method Convolutional Neural Networks (ConvNets) for subregion identification and selection.
result Enhanced accuracy in parameter estimation using ConvNet-based partitioning.
Inter-domain Deep Gaussian Processes improve inference for non-stationary data.
problem Inference limitations in Gaussian processes for non-stationary data.
method Combines inter-domain and deep Gaussian processes for scalable approximate inference.
result Outperforms inter-domain shallow GPs and conventional DGPs on non-stationary data.
Deep linear ResNets converge globally with certain transformations.
problem Global convergence of training deep linear ResNets.
method Gradient descent and stochastic gradient descent for training L-hidden-layer linear ResNets. result GD and SGD can converge to global minimum for deep linear ResNets with specific transformations.