arXiv research

Novel neural GP kernels learn stable, flexible covariance structures.

Our article considers a Gaussian variational approximation of the posterior density in a high-dimensional state space model. The variational parameters to be optimized are the mean vector and the covariance matrix of the approximation. The number of parameters in the covariance matrix grows as the square of the number …

Unified error analysis for low-rank approximation improves data assimilation performance.

New methods estimate covariance for matrix data without assuming fixed size or specific distributions.

Diagonal transformations preserve independence structures in non-Gaussian distributions.

New method stabilizes private LASSO for high-dimensional data with diverse covariate scales.

Compact Gaussian model approximates deep ensemble predictions.

Novel Fréchet regression method handles errors-in-variables with low-rank covariates.

Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution. In such situations, existing methods usually resort to a diagonal approximation of the covariance matrix despite, the fact that these mat…

EiGLasso speeds up sparse Kronecker-sum covariance estimation.

The paper analyzes how Gaussian kernel parameters affect posterior covariance in Gaussian processes.

Gaussian variational approximation is a popular methodology to approximate posterior distributions in Bayesian inference especially in high dimensional and large data settings. To control the computational cost while being able to capture the correlations among the variables, the low rank plus diagonal structure was in…

In this paper we consider the use of the space vs. time Kronecker product decomposition in the estimation of covariance matrices for spatio-temporal data. This decomposition imposes lower dimensional structure on the estimated covariance matrix, thus reducing the number of samples required for estimation. To allow a sm…

Paper solves a key problem in learning from high-dimensional covariance matrices.

New method prevents posterior collapse in iVAE models.

We propose a novel estimation approach for the covariance matrix based on the $l_1$-regularized approximate factor model. Our sparse approximate factor (SAF) covariance estimator allows for the existence of weak factors and hence relaxes the pervasiveness assumption generally adopted for the standard approximate factor…

Covariance pooling is a feature pooling method with good classification accuracy. Because covariance features consist of second-order statistics, the scale of the feature elements are varied. Therefore, normalizing covariance features using a matrix square root affects the performance improvement. When pooling methods …

Covariance shrinkage via stochastic interpolation

PACE-GGM uses Gaussian mechanism for private covariance estimation.

Optimal classifiers derived from GMMs are approximated by deep neural networks.

This work improves variational inference by reducing gradient variance.

Bayesian model averaging fails under covariate shift, affecting neural networks' performance.

The L1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix, or alternatively the underlying graph structure of a Gaussian Markov Random Field, from very limited samples. We propose a novel algorithm for solving the…

We propose a black-box variational inference method to approximate intractable distributions with an increasingly rich approximating class. Our method, termed variational boosting, iteratively refines an existing variational approximation by solving a sequence of optimization problems, allowing the practitioner to trad…

We conduct a study of the aliased spectral densities of Matérn covariance functions on a regular grid of points, providing clarity on the properties of a popular approximation based on stochastic partial differential equations; while others have shown that it can approximate the covariance function well, we find that i…

Many popular statistical models, such as factor and random effects models, give arise a certain type of covariance structures that is a summation of low rank and sparse matrices. This paper introduces a penalized approximation framework to recover such model structures from large covariance matrix estimation. We propos…

New iterative methods improve scalability of Gaussian process approximations for large data.

Treeging combines regression trees and kriging for spatial and space-time prediction.

A scalable algorithm for GP regression selects relevant covariates efficiently.

Paper develops an online covariance estimator for nonsmooth stochastic approximation problems.

This study approximates distances between Gaussian processes and covariance operators using RKHS.

Logistic Gaussian process (LGP) priors provide a flexible alternative for modelling unknown densities. The smoothness properties of the density estimates can be controlled through the prior covariance structure of the LGP, but the challenge is the analytically intractable inference. In this paper, we present approximat…

Standard sparse pseudo-input approximations to the Gaussian process (GP) cannot handle complex functions well. Sparse spectrum alternatives attempt to answer this but are known to over-fit. We suggest the use of variational inference for the sparse spectrum approximation to avoid both issues. We model the covariance fu…

Much recent work has concerned sparse approximations to speed up the Gaussian process regression from the unfavorable O(n3) scaling in computational time to O(nm2). Thus far, work has concentrated on models with one covariance function. However, in many practical situations additive models with multiple covariance func…

Two new algorithms improve robust PCA and Schatten packing.

In this paper, we present a general, multistage framework for graphical model approximation using a cascade of models such as trees. In particular, we look at the problem of covariance matrix approximation for Gaussian distributions as linear transformations of tree models. This is a new way to decompose the covariance…

The covariance matrix of a $p$-dimensional random variable is a fundamental quantity in data analysis. Given $n$ i.i.d. observations, it is typically estimated by the sample covariance matrix, at a computational cost of $O(np^{2})$ operations. When $n,p$ are large, this computation may be prohibitively slow. Moreover, …

Graphical Lasso (GL) is a popular method for learning the structure of an undirected graphical model, which is based on an $l_1$ regularization technique. The objective of this paper is to compare the computationally-heavy GL technique with a numerically-cheap heuristic method that is based on simply thresholding the s…

We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian. Previous works decomposed the Hessian into two components, attributing the outliers to one o…

Exact Gaussian Processes for massive datasets using non-stationary sparsity-discovering kernels.

High precision analytical approximation is proposed for variance-covariance based risk allocation in a portfolio of risky assets. A general case of a single-period multi-factor Merton-type model with stochastic recovery is considered. The accuracy of the approximation as well as its speed are compared to and shown to b…

Geometric families of low-rank covariances improve flexibility and tractability in high dimensions.

CaTs use DAGs with transformers to enforce causal constraints, improving neural network robustness.

We uncover scaling laws and statistical structure in complex datasets.

STACI uses neural nets to estimate spatio-temporal fields with valid uncertainty quantification.

We define a copula process which describes the dependencies between arbitrarily many random variables independently of their marginal distributions. As an example, we develop a stochastic volatility model, Gaussian Copula Process Volatility (GCPV), to predict the latent standard deviations of a sequence of random varia…

Gaussian graphical models are widely utilized to infer and visualize networks of dependencies between continuous variables. However, inferring the graph is difficult when the sample size is small compared to the number of variables. To reduce the number of parameters to estimate in the model, we propose a non-asymptoti…

A new debiasing method for high-dimensional regression with applications to PCR.