The study provides statistical guarantees for Bayesian variational boosting.
problem Statistical and convergence issues in variational boosting.
method Proposed a novel variational family and a functional Frank-Wolfe optimization algorithm.
result Demonstrated stochastic boundedness and provided convergence rate for boosting iterates.
Bayesian GBMs improve predictive uncertainty calibration for tabular data.
problem Lack of well-calibrated predictive uncertainties in gradient boosting machines.
method Variational inference with soft decision trees.
result Variational soft GBMs provide useful uncertainty estimates and maintain good predictive performance.
We offer a novel view of AdaBoost in a statistical setting. We propose a Bayesian model for binary classification in which label noise is modeled hierarchically. Using variational inference to optimize a dynamic evidence lower bound, we derive a new boosting-like algorithm called VIBoost. We show its close connections …
BBNN improves neural network accuracy and uncertainty quantification.
problem Overfitting and lack of interpretability in probabilistic neural networks.
method Boosted Bayesian Neural Networks (BBNN) using Boosting Variational Inference (BVI).
result BBNN achieves ~5% higher accuracy and superior uncertainty quantification.
MFAI uses gradient boosted trees to leverage auxiliary info for scalable Bayesian matrix factorization.
problem Matrix factorization struggles with poor data quality, especially high sparsity and low SNR.
method Integrates gradient boosted trees into probabilistic matrix factorization framework.
result MFAI effectively leverages auxiliary information, improving model performance.
Approximating a probability density in a tractable manner is a central task in Bayesian statistics. Variational Inference (VI) is a popular technique that achieves tractability by choosing a relatively simple variational family. Borrowing ideas from the classic boosting framework, recent approaches attempt to \emph{boo…
Variational inference is a popular technique to approximate a possibly intractable Bayesian posterior with a more tractable one. Recently, boosting variational inference has been proposed as a new paradigm to approximate the posterior by a mixture of densities by greedily adding components to the mixture. However, as i…
Bayesian approach generalizes ADMM for federated learning.
problem Improving federated learning efficiency and accuracy.
method Integrates Bayesian duality with ADMM for optimization.
result New extensions of ADMM for various distributions.
A new method combines VI and IS to improve Bayesian inference accuracy.
problem Bayesian inference often underestimates posterior tails, leading to miscalibration and degeneracy.
method Proposes a novel combination of optimization and sampling techniques using the forward KL divergence.
result The method guarantees asymptotic consistency and fast convergence to optimal IS and variational approximations.
Variational inference (VI) provides fast approximations of a Bayesian posterior in part because it formulates posterior approximation as an optimization problem: to find the closest distribution to the exact posterior over some family of distributions. For practical reasons, the family of distributions in VI is usually…
Improved IVON boosts LoRA model accuracy and calibration.
problem Improving the accuracy and calibration of large language models.
method Replaced AdamW with IVON for finetuning Llama-2.
result IVON improves Llama-2 accuracy by 2.8% and calibration error by 4.6%.
Natural image statistics exhibit hierarchical dependencies across multiple scales. Representing such prior knowledge in non-factorial latent tree models can boost performance of image denoising, inpainting, deconvolution or reconstruction substantially, beyond standard factorial "sparse" methodology. We derive a large …
Boosting Variational Inference improves posterior approximations with adaptive step-sizes.
problem Limited resources hinder the widespread adoption of Boosting Variational Inference.
method Characterized global curvature impact, introduced local curvature, and developed an approximate backtracking algorithm.
result New theoretical convergence rates and experimental validation demonstrate improved performance.
BayesBoost combines boosting and Bayesian methods for linear mixed models, improving uncertainty estimation and variable selection.
problem Lack of straightforward uncertainty estimation for parameters in high-dimensional linear mixed models.
method BayesBoost: Combines boosting and Bayesian inference for linear mixed models.
result Improves uncertainty estimation and variable selection in linear mixed models.
Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training of deep networks with natural-gradient variational inference. By applying techniques such as batch n…
A framework to boost the efficiency of Bayesian inference in probabilistic programs is introduced by embedding a sampler inside a variational posterior approximation. We call it the refined variational approximation. Its strength lies both in ease of implementation and automatically tuning of the sampler parameters to …
We propose a black-box variational inference method to approximate intractable distributions with an increasingly rich approximating class. Our method, termed variational boosting, iteratively refines an existing variational approximation by solving a sequence of optimization problems, allowing the practitioner to trad…
While much research effort has been dedicated to scaling up sparse Gaussian process (GP) models based on inducing variables for big data, little attention is afforded to the other less explored class of low-rank GP approximations that exploit the sparse spectral representation of a GP kernel. This paper presents such a…
Bayesian model predicts oncology demand trends with high accuracy.
problem Accurate forecasting of oncology demand for resource planning.
method Boosting-based Bayesian conjugate models for Poisson process.
result Model outperforms other methods in trend detection accuracy.
Bayesian additive regression trees (BART) (Chipman et. al., 2010) is a powerful predictive model that often outperforms alternative models at out-of-sample prediction. BART is especially well-suited to settings with unstructured predictor variables and substantial sources of unmeasured variation as is typical in the so…
BOOST automates kernel and acquisition function selection in Bayesian optimization.
problem Inappropriate kernel and acquisition function combinations lead to poor performance in Bayesian optimization.
method BOOST uses offline evaluation to predict and select the best kernel-acquisition function pair.
result BOOST consistently improves over fixed-hyperparameter BO and is competitive with state-of-the-art adaptive methods.
CCI combines Bayesian and gradient boosting to create fair, reliable credit risk scores.
problem Tackles high-stakes lending decisions with changing data distributions and fairness constraints.
method Combines Bayesian neural risk scorer and fairness-constrained gradient boosting with shift-aware fusion.
result CCI achieves best trade-off between discrimination, calibration, stability, and fairness.
Develops robust ML systems for predictive uncertainties and adversarial examples.
problem Uncertainties and vulnerabilities in ML predictions and adversarial examples.
method Bayesian structural time series models, SG-MCMC, SVGD, Markov chain samplers, reinforcement learning.
result Improved robustness and efficiency in Bayesian inference and adversarial machine learning.
Deep generative models are effective methods of modeling data. However, it is not easy for a single generative model to faithfully capture the distributions of complex data such as images. In this paper, we propose an approach for boosting generative models, which cascades meta-models together to produce a stronger mod…
Gradient boosting improved with lassoed trees achieves faster convergence.
problem Improving gradient boosting convergence in large nonparametric spaces.
method Lassoed gradient boosted trees with early stopping.
result Achieves faster than n−1/4 L2 convergence rate. Study on surfaces in Heisenberg group with constant mean curvature.
problem Constant mean curvature surfaces in sub-Lorentzian Heisenberg group.
method First-variation formula derivation and isoperimetric candidates classification.
result Characterization and conjecture of isoperimetric maximizers.
Improved MUSE boosts performance and reduces error in Bayesian inference.
problem Hierarchical Bayesian inference problems
method Implicit differentiation applied to MUSE algorithm
result Significant speedup and improved accuracy compared to Hamiltonian Monte Carlo
Infinitesimal gradient boosting is a new algorithm derived from gradient boosting.
problem Improving the efficiency and smoothness of gradient boosting.
method Introduced a new class of randomized regression trees and used a limit process in vanishing-learning-rate asymptotic.
result Convergence of the stochastic algorithm and characterization of the limiting procedure as a unique solution of a nonlinear ODE.
This paper presents a novel method for structural data recognition using a large number of graph models. In general, prevalent methods for structural data recognition have two shortcomings: 1) Only a single model is used to capture structural variation. 2) Naive recognition methods are used, such as the nearest neighbo…
Bayesian active learning improves stress and affect detection on wearable devices.
problem Handling unlabeled data in real-time for stress and affect detection.
method Bayesian Neural Networks with Monte-Carlo Dropout and suitable acquisition functions.
result Framework achieves significant efficiency boost and low number of acquired pool points.
There has recently been a steady increase in the number iterative approaches to density estimation. However, an accompanying burst of formal convergence guarantees has not followed; all results pay the price of heavy assumptions which are often unrealistic or hard to check. The Generative Adversarial Network (GAN) lite…
Improved uncertainty estimation in neural networks with VBLL.
problem Improving uncertainty estimation in neural networks.
method Deterministic variational formulation for training Bayesian last layer neural networks.
result Improves predictive accuracy, calibration, and out-of-distribution detection.
Boosting variational inference (BVI) approximates an intractable probability density by iteratively building up a mixture of simple component distributions one at a time, using techniques from sparse convex optimization to provide both computational scalability and approximation error guarantees. But the guarantees hav…
A new variational method speeds up Bayesian phylogenetic inference.
problem Slow and inefficient MCMC methods in Bayesian phylogenetic inference.
method Combining subsplit Bayesian networks with variational inference for efficient tree topology and branch length estimation.
result Variational approach provides competitive performance with significantly fewer iterations.
Variational Inference shows promise for Bayesian GARCH model estimation.
problem Bayesian estimation of GARCH-family models using Monte Carlo sampling.
method Variational Inference as an alternative to Monte Carlo sampling.
result Variational Inference is a reliable and competitive method for Bayesian learning in GARCH-like models.
Forest tree species mapped with high accuracy using satellite data.
problem Classifying dominant tree species in Swedish forests.
method Extreme gradient boosting model with Bayesian optimization, combining Sentinel-1/2 satellite data and field observations.
result Overall accuracy of 85%, F1 score of 0.82, Matthews correlation coefficient of 0.81.
SMAC method optimizes tree-boosting hyperparameters best.
problem Optimizing hyperparameters for tree-boosting to improve model accuracy.
method Compared and evaluated various hyperparameter optimization methods.
result SMAC method outperforms other methods for hyperparameter tuning.
BayesPy is an open-source Python software package for performing variational Bayesian inference. It is based on the variational message passing framework and supports conjugate exponential family models. By removing the tedious task of implementing the variational Bayesian update equations, the user can construct model…
This study examines how randomness affects machine learning model performance.
problem The impact of randomness on machine learning model performance.
method Empirical study comparing randomness in model training and dataset partitioning.
result Randomness in model training causes more variation in FFNNs than tree-based methods.
Improved Bayesian inference via variational approximations of generalized rho-posteriors.
problem Robust Bayesian inference under model misspecification and data contamination.
method Introducing a modified ρ-posterior and using PAC-Bayesian analysis with variational approximations. result Theoretical guarantees for tractable inference with competitive robustness and computational efficiency.
A new Randomized-Hyperopt method improves XGBoost hyperparameter tuning.
problem Improving the performance of XGBoost through hyperparameter optimization.
method Proposes Randomized-Hyperopt for XGBoost hyperparameter tuning.
result Randomized-Hyperopt outperforms other methods in terms of accuracy and execution time.
Hyperboost uses gradient boosting for hyperparameter optimization, outperforming state-of-the-art methods.
problem Hyperparameter tuning for machine learning algorithms
method Gradient boosting surrogate model with quantile regression and distance metric
result Hyperboost outperforms state-of-the-art techniques in empirical tests
Bayesian supertrees method uses variational Bayes for overlapping taxon subsets.
problem Inference of phylogenetic tree topologies for overlapping taxon sets.
method Variational Bayesian approach.
result Demonstrates effectiveness of variational Bayes for Bayesian supertrees.
Variational Prediction simplifies Bayesian inference without test time costs.
problem Bayesian inference's computational costs and posterior predictive distribution marginalization.
method Variational Prediction learns a variational approximation to the posterior predictive distribution using a variational bound.
result Directly learns a variational approximation to the posterior predictive distribution without test time marginalization costs.
We review three algorithms for Latent Dirichlet Allocation (LDA). Two of them are variational inference algorithms: Variational Bayesian inference and Online Variational Bayesian inference and one is Markov Chain Monte Carlo (MCMC) algorithm -- Collapsed Gibbs sampling. We compare their time complexity and performance.…
Bayesian method improves SOM training for dynamic data.
problem Training Self Organizing Maps (SOM) for non-stationary data.
method Variational Bayesian approach with adaptive neighborhood radius.
result Method outperforms other adaptive methods in high dimensions.
Estimates density ratio for two-sample comparison using tree models.
problem Comparing two distributions given i.i.d. observations.
method Additive tree models with balancing loss for density ratio estimation.
result Bayesian inference provides uncertainty quantification for density ratio.
Improves Bayesian neural networks inference efficiency and accuracy.
problem Inflexibility of factorized structure in Dropout posterior.
method Introduces Variational Structured Dropout (VSD) with orthogonal transformation.
result VSD induces adaptive regularization and better generalization.