arXiv research

The paper deals with learning probability distributions of observed data by artificial neural networks. We suggest a so-called gradient conjugate prior (GCP) update appropriate for neural networks, which is a modification of the classical Bayesian update for conjugate priors. We establish a connection between the gradi…

Proposes an extension of BayesCG for solving multiple linear systems.

New method corrects deep neural network variance for outlier data.

BONG optimizes Bayesian inference online with natural gradient descent.

We derive and approximate the conjugate prior of Dirichlet and beta distributions.

Researchers compare different gradient methods for ridge regression, finding conjugate gradients have similar performance.

New method improves calibration of BayesCG for better uncertainty quantification.

Proves FR-NGD optimally approximates evolutionary dynamics and continuous Bayesian inference.

Thompson sampling has impressive empirical performance for many multi-armed bandit problems. But current algorithms for Thompson sampling only work for the case of conjugate priors since these algorithms require to infer the posterior, which is often computationally intractable when the prior is not conjugate. In this …

Unified analysis of conjugate gradients and accelerated methods using duality gap.

Making inferences from data streams is a pervasive problem in many modern data analysis applications. But it requires to address the problem of continuous model updating and adapt to changes or drifts in the underlying data generating distribution. In this paper, we approach these problems from a Bayesian perspective c…

The stochastic variational inference (SVI) paradigm, which combines variational inference, natural gradients, and stochastic updates, was recently proposed for large-scale data analysis in conjugate Bayesian models and demonstrated to be effective in several problems. This paper studies a family of Bayesian latent vari…

This note investigates a conjugate class for the Dirichlet distribution class in the exponential family.

The natural gradient method has been used effectively in conjugate Gaussian process models, but the non-conjugate case has been largely unexplored. We examine how natural gradients can be used in non-conjugate stochastic settings, together with hyperparameter learning. We conclude that the natural gradient can signific…

We address the challenge of effective exploration while maintaining good performance in policy gradient methods. As a solution, we propose diverse exploration (DE) via conjugate policies. DE learns and deploys a set of conjugate policies which can be conveniently generated as a byproduct of conjugate gradient descent. …

The study proves triviality and rigidity results for Ricci solitons and estimates their conjugate radius.

NCG methods improve shape optimization efficiency.

In this work, we present direction-of-arrival (DoA) estimation algorithms based on the Krylov subspace that effectively exploit prior knowledge of the signals that impinge on a sensor array. The proposed multi-step knowledge-aided iterative conjugate gradient (CG) (MS-KAI-CG) algorithms perform subtraction of the unwan…

Harmonic functions of two variables are exactly those that admit a conjugate, namely a function whose gradient has the same length and is everywhere orthogonal to the gradient of the original function. We show that there are also partial differential equations controlling the functions of three variables that admit a c…

New method uses KL-divergence to create non-informative priors for multivariate Gaussian.

A new method speeds up deep neural network training.

Improved kernel ridge regression using conjugate gradients.

Develops a fast variational approximation for high-dimensional empirical Bayes posteriors.

The computational and storage complexity of kernel machines presents the primary barrier to their scaling to large, modern, datasets. A common way to tackle the scalability issue is to use the conjugate gradient algorithm, which relieves the constraints on both storage (the kernel matrix need not be stored) and computa…

Improved Gaussian process regression with tighter log marginal likelihood bounds.

In a modern observational study based on healthcare databases, the number of observations and of predictors typically range in the order of $10^5$ ~ $10^6$ and of $10^4$ ~ $10^5$. Despite the large sample size, data rarely provide sufficient information to reliably estimate such a large number of parameters. Sparse reg…

New conjugate priors improve Bayesian inference for multinomial probit models.

This manuscript proposes a probabilistic framework for algorithms that iteratively solve unconstrained linear problems $Bx = b$ with positive definite $B$ for $x$. The goal is to replace the point estimates returned by existing methods with a Gaussian posterior belief over the elements of the inverse of $B$, which can …

Nonlinear conjugate gradient (NLCG) based optimizers have shown superior loss convergence properties compared to gradient descent based optimizers for traditional optimization problems. However, in Deep Neural Network (DNN) training, the dominant optimization algorithm of choice is still Stochastic Gradient Descent (SG…

Low-rank matrix estimation from incomplete measurements recently received increased attention due to the emergence of several challenging applications, such as recommender systems; see in particular the famous Netflix challenge. While the behaviour of algorithms based on nuclear norm minimization is now well understood…

We characterize conjugate nonparametric Bayesian models as projective limits of conjugate, finite-dimensional Bayesian models. In particular, we identify a large class of nonparametric models representable as infinite-dimensional analogues of exponential family distributions and their canonical conjugate priors. This c…

A method uses CG to create efficient channels for ideal observers.

Conjugate gradient methods improve efficiency for high-dimensional GLMMs.

New method for identifying autoregressive systems on manifolds.

SING improves state inference in latent SDE models for better drift function estimation.

This report works out the details of a closed-form, fully Bayesian, multiclass, openset, generative pattern classifier using multivariate Gaussian likelihoods, with conjugate priors. The generative model has a common within-class covariance, which is proportional to the between-class covariance in the conjugate prior. …

We propose a novel Bayesian approach to solve stochastic optimization problems that involve finding extrema of noisy, nonlinear functions. Previous work has focused on representing possible functions explicitly, which leads to a two-step procedure of first, doing inference over the function space and second, finding th…

A new network reduces MIMO detection complexity.

NMC improves MCMC convergence by analyzing gradients to determine optimal proposal densities.

Deep learning designs effective preconditioners for water engineering problems.

A new method reduces the cost of solving large-scale linear models.

We propose a conjugate gradient type optimization technique for the computation of the Karcher mean on the set of complex linear subspaces of fixed dimension, modeled by the so-called Grassmannian. The identification of the Grassmannian with Hermitian projection matrices allows an accessible introduction of the geometr…

NeuralIF uses neural networks to improve preconditioning for faster CG convergence.

Let $(M,g(t))$, $0\le t\le T$, $\partial M\neφ$, be a compact $n$-dimensional manifold, $n\ge 2$, with metric $g(t)$ evolving by the Ricci flow such that the second fundamental form of $\partial M$ with respect to the unit outward normal of $\partial M$ is uniformly bounded below on $\partial M\times [0,T]$. We will pr…

We establish a point-wise gradient estimate for $all$ positive solutions of the conjugate heat equation. This contrasts to Perelman's point-wise gradient estimate which works mainly for the fundamental solution rather than all solutions. Like Perelman's estimate, the most general form of our gradient estimate does not …

Proposes a non-conjugate model selection method for chain event graphs.

Conjugate pairs of distributions over infinite dimensional spaces are prominent in statistical learning theory, particularly due to the widespread adoption of Bayesian nonparametric methodologies for a host of models and applications. Much of the existing literature in the learning community focuses on processes posses…

Thompson sampling used for linear bandits with normal-gamma priors.