Sequential coordinate ascent is more robust in high-dimensional linear regression.
problem Behavior difference between sequential and parallel coordinate ascent in variational inference.
method Comparison of sequential and parallel coordinate ascent algorithms in high-dimensional linear regression.
result Sequential algorithm converges under more relaxed conditions than parallel algorithm.
Algorithm improves variational inference in Wasserstein distance.
problem Improving variational inference methods for complex models.
method Wasserstein contraction analysis of coordinate ascent.
result General and sharp convergence guarantees for various models.
Random scan CAVI converges linearly under log-concave assumptions.
problem Analyzing the convergence rate of random scan Coordinate Ascent Variational Inference (CAVI) under log-concave conditions.
method Building on previous work, we analyze the random scan version of CAVI using optimal transport geometry.
result We obtain tight linear convergence rates for the random scan version of CAVI.
Improved Bayesian uncertainty quantification using variational bagging.
problem Inefficient and underestimating uncertainty in mean-field variational Bayes.
method Integrates bagging with variational Bayes for improved inference.
result Bagged variational posterior provides proper uncertainty quantification.
The mean field variational Bayes method is becoming increasingly popular in statistics and machine learning. Its iterative Coordinate Ascent Variational Inference algorithm has been widely applied to large scale Bayesian inference. See Blei et al. (2017) for a recent comprehensive review. Despite the popularity of the …
Optimizes variational inference for dynamic network models.
problem Estimating pairwise inner products and intercepts in dynamic latent space models.
method Structured mean-field variational inference with block coordinate ascent algorithm.
result Variational risk attains minimax optimal rate with logarithmic factor under certain conditions.
Study variational Bayes for high-dimensional linear regression with sparse priors.
problem Sparse high-dimensional linear regression model selection.
method Mean-field spike and slab variational Bayes approximation, oracle inequalities, coordinate-ascent variational inference (CAVI), prioritized updating scheme.
result Mean-field VB approximation converges to the sparse truth at optimal rate and gives optimal prediction.
A new particle algorithm improves mean-field variational inference.
problem Efficiently approximating nonparametric posterior distributions in machine learning.
method Introduces PArticle VI (PAVI), a novel particle-based algorithm for nonparametric mean-field approximation.
result Obtains non-asymptotic error bounds for PArticle VI, providing the first end-to-end guarantee for particle-based MFVI.
Gradient-based optimization improves variational empirical Bayes regression.
problem Sparse, large-scale multiple regression models.
method Gradient-based optimization (GradVI) for variational empirical Bayes (VEB) regression.
result GradVI produces similar predictive performance to CAVI but converges faster and is faster in certain settings.
A scalable method for estimating spatial data using VREML.
problem Costly computation of REML for large, sparse precision matrices in spatial data.
method Proposes VREML framework approximating marginal likelihood with Gaussian variational distribution and deriving a coordinate-ascent algorithm.
result Empirically shows VREML outperforms MLE and INLA.
Efficiently identifies important variables in binary outcomes using variational Bayes.
problem Bayesian variable selection for binary outcomes with computational challenges.
method Mean-field variational Bayes approximation with closed-form updates and efficient inference algorithm.
result Successfully identifies important variables and is orders of magnitude faster than MCMC.
CAVI converges for log-concave measures via optimal transport.
problem Finding the closest product measure to a log-concave measure via CAVI.
method Adapting coordinate descent techniques from Euclidean space to optimal transport for log-concave densities.
result Proves convergence of CAVI for log-concave densities and provides rates of convergence under additional conditions.
Bayesian model selection via mean-field variational approximation improves efficiency and accuracy.
problem Bayesian model selection under model mis-specification and latent variables.
method Mean-field variational approximation with non-asymptotic properties and geometric convergence.
result ELBO tends to select models closer to the true model than BIC as sample size increases.
New method speeds up inference for non-conjugate Gaussian processes.
problem Inference for non-conjugate Gaussian processes is slow and unreliable.
method Automated augmented conjugate inference method that constructs auxiliary variables to make the model conditionally conjugate.
result Our method is up to two orders of magnitude faster and more robust than existing methods.
Paper introduces f-divergence variational inference for broader application.
problem Variational inference limited to specific divergences.
method Generalizes variational inference to all f-divergences using f-divergence minimization.
result Unified framework for variational inference with arbitrary f-divergences.
CAVI converges globally or locally exponentially for two-block models.
problem Convergence analysis of coordinate ascent variational inference (CAVI).
method Analysis of CAVI convergence using functional analysis and optimization.
result General conditions for certifying global or local exponential convergence of CAVI.
Develops a new framework for analyzing MFVI algorithms.
problem Analyzes mean field variational inference (MFVI) formulations.
method Inspired by variational Bayesian formulations, represents MFVI problem in three ways: gradient flow, Fokker-Planck-like equations, and diffusion process.
result Establishes rigorous guarantees for convergence of time-discretized coordinate ascent variational inference algorithms.
Mean-field variational methods are widely used for approximate posterior inference in many probabilistic models. In a typical application, mean-field methods approximately compute the posterior with a coordinate-ascent optimization algorithm. When the model is conditionally conjugate, the coordinate updates are easily …
CAVI speeds up Bayesian MIDAS regression by 107x-1,772x with similar accuracy.
problem Efficiently estimating Bayesian MIDAS regression models with many predictors.
method Coordinate Ascent Variational Inference (CAVI) for linear MIDAS regression.
result CAVI produces posterior means nearly identical to Gibbs sampling with significant speedup.
Efficient multi-class classification with well-calibrated uncertainty.
problem Trade-off between uncertainty calibration and speed in multi-class Gaussian process classification.
method Proposes a new likelihood function leading to a conditionally conjugate model with efficient variational inference.
result Up to two orders faster than state-of-the-art methods with well-calibrated uncertainty estimates.
Improved Kalman filtering with hierarchical variational approach.
problem Inconsistent process covariance estimation and slow convergence speed in traditional variational Kalman filtering.
method Introducing a surrogate variable for process-noise-free state, reformulating CAVI, and sliding-window hyperparameter estimation.
result Enhanced convergence speed and superior estimation accuracy compared to existing methods.
New method for probabilistic modeling of integer submodular functions.
problem Lack of probabilistic modeling for integer submodular functions.
method Proposed Generalized Multilinear Extension and block-coordinate ascent algorithm.
result Demonstrated effectiveness and viability on real-world datasets.
Bayesian entity resolution merges together multiple, noisy databases and returns the minimal collection of unique individuals represented, together with their true, latent record values. Bayesian methods allow flexible generative models that share power across databases as well as principled quantification of uncertain…
VBphenoR uses variational Bayes for EHR-based patient phenotyping.
problem Phenotyping patients from EHR data for targeted treatments.
method Variational Bayes Gaussian Mixture Model (GMM) and logistic regression.
result Closed-form inference for efficient patient phenotype determination.
New taxonomy and improved solvers for discrete energy minimization.
problem Maximum-a-posteriori inference in discrete graphical models.
method Dual block-coordinate ascent rule, theoretical analysis, new solver variants.
result Improved state-of-the-art solver outperforming existing methods on all test instances.
Coordinate ascent variational inference is an important algorithm for inference in probabilistic models, but it is slow because it updates only a single variable at a time. Block coordinate methods perform inference faster by updating blocks of variables in parallel. However, the speed and stability of these algorithms…
New method improves uncertainty estimation in complex statistical models.
problem Challenges in estimating high-dimensional mixed models due to computational complexity.
method Partially factorized variational inference to relax mean-field assumption.
result Relaxed variational inference provides accurate uncertainty quantification without high computational cost.
This paper presents a non-trivial reconstruction of a previous joint topic-sentiment-preference review model TSPRA with stick-breaking representation under the framework of variational inference (VI) and stochastic variational inference (SVI). TSPRA is a Gibbs Sampling based model that solves topics, word sentiments an…
Gradient-free method improves predictive accuracy for probabilistic models.
problem Balancing computational efficiency and robust predictive performance in deep learning.
method CAVI-CMN, a gradient-free variational method for conditional mixture networks.
result CAVI-CMN achieves competitive and often superior predictive accuracy compared to MLE with backpropagation.
Paper introduces a new text clustering model using Beta-Liouville priors.
problem Clustering short text data.
method Develops a hierarchical mixture model with Beta-Liouville priors for short text clustering.
result The Beta-Liouville distribution offers a more flexible correlation structure for short text clustering.
RSI uses Bayesian inference to monitor compliance in rule-governed domains.
problem Structural obstacles in compliance monitoring, including unlabeled outcomes and selective withholding of evidence.
method Rule-State Inference (RSI) treats formalized rules as Bayesian priors and infers compliance states through mean-field variational inference.
result RSI delivers formal guarantees of adaptability, consistency, and convergence, validated on a synthetic enterprise benchmark.
Nash integrates covariate-specific side info into sparse regression via neural networks.
problem Sparse linear regression struggles with covariates exhibiting structure or coming from heterogeneous sources.
method Neural Adaptive Shrinkage (Nash) framework that integrates side information into sparse regression via neural networks. Uses split variational empirical Bayes algorithm.
result Nash improves accuracy and adaptability over existing methods in real data experiments.
CAVI converges exponentially fast for Bayesian PCA models.
problem Characterizing the convergence speed of CAVI for BPCA.
method Proved exponential convergence using power iteration analogy and novel lower bounds.
result Exponential convergence of CAVI for BPCA models with any number of principal components.
Stochastic Gradient Descent (SGD) has become popular for solving large scale supervised machine learning optimization problems such as SVM, due to their strong theoretical guarantees. While the closely related Dual Coordinate Ascent (DCA) method has been implemented in various software packages, it has so far lacked go…
Stochastic dual coordinate ascent (SDCA) is an effective technique for solving regularized loss minimization problems in machine learning. This paper considers an extension of SDCA under the mini-batch setting that is often used in practice. Our main contribution is to introduce an accelerated mini-batch version of SDC…
Improves posterior approximation speed for Dirichlet process mixture models.
problem Inefficiency of stochastic variational inference in large datasets.
method Uses stochastic gradient ascent with adaptive stepsize optimization.
result Adaptive stepsize improves speed and performance of posterior approximation.
We introduce a proximal version of dual coordinate ascent method. We demonstrate how the derived algorithmic framework can be used for numerous regularized loss minimization problems, including ℓ1 regularization and structured output SVM. The convergence rates we obtain match, and sometimes improve, state-of-the-…
NeuralSurv models survival analysis with Bayesian uncertainty.
problem Capturing time-varying risk relationships in survival analysis.
method Two-stage data-augmentation scheme, mean-field variational algorithm, coordinate-ascent updates, locally linearized Bayesian neural network.
result Delivers superior calibration compared to state-of-the-art models.
In machine learning, Feature Selection (FS) is a major part of efficient algorithm. It fuels the algorithm and is the starting block for our prediction. In this paper, we present a new method, called Optimal Coordinate Ascent (OCA) that allows us selecting features among block and individual features. OCA relies on coo…
The stochastic dual coordinate-ascent (S-DCA) technique is a useful alternative to the traditional stochastic gradient-descent algorithm for solving large-scale optimization problems due to its scalability to large data sets and strong theoretical guarantees. However, the available S-DCA formulation is limited to finit…
We introduce a proximal version of the stochastic dual coordinate ascent method and show how to accelerate the method using an inner-outer iteration procedure. We analyze the runtime of the framework and obtain rates that improve state-of-the-art results for various key machine learning optimization problems including …
Bayesian SPCA method tackles orthogonality constraint with spike and slab prior.
problem Bayesian SPCA method for high-dimensional data with orthogonality constraint.
method Parameter-expanded coordinate ascent variational inference (PX-CAVI) with spike and slab prior.
result PX-CAVI algorithm outperforms existing SPCA approaches in performance.
Mean field inference in probabilistic models is generally a highly nonconvex problem. Existing optimization methods, e.g., coordinate ascent algorithms, can only generate local optima. In this work we propose provable mean filed methods for probabilistic log-submodular models and its posterior agreement (PA) with stron…
This paper introduces AdaSDCA: an adaptive variant of stochastic dual coordinate ascent (SDCA) for solving the regularized empirical risk minimization problems. Our modification consists in allowing the method adaptively change the probability distribution over the dual variables throughout the iterative process. AdaSD…
New solver MPLP++ outperforms existing solvers for dense graph models.
problem Efficiently solving dense, discrete Graphical Models with pairwise potentials.
method Dual Block-Coordinate Ascent with MPLP++ modification.
result MPLP++ significantly outperforms existing solvers, including TRWS.
We propose a new stochastic dual coordinate ascent technique that can be applied to a wide range of regularized learning problems. Our method is based on Alternating Direction Multiplier Method (ADMM) to deal with complex regularization functions such as structured regularizations. Although the original ADMM is a batch…
DPGIIL clusters structural anomalies using transmissibility functions with deep learning and Dirichlet process.
problem Clustering structural anomalies in high-dimensional streaming data with optimal cluster number determination.
method Combines Dirichlet process and deep generative models for incremental learning and anomaly detection.
result DPGIIL outperforms traditional methods in anomaly detection and clustering.
Bayesian hierarchical tensor factorization model for international trade flows
problem Sparse semi-continuous tensor data modeling
method Bayesian hierarchical tensor factorization with Poisson and Gamma models
result Identifies multiway dependence in trade flows