CAVI converges globally or locally exponentially for two-block models.
problem Convergence analysis of coordinate ascent variational inference (CAVI).
method Analysis of CAVI convergence using functional analysis and optimization.
result General conditions for certifying global or local exponential convergence of CAVI.
CAVI converges exponentially fast for Bayesian PCA models.
problem Characterizing the convergence speed of CAVI for BPCA.
method Proved exponential convergence using power iteration analogy and novel lower bounds.
result Exponential convergence of CAVI for BPCA models with any number of principal components.
CAVI converges for log-concave measures via optimal transport.
problem Finding the closest product measure to a log-concave measure via CAVI.
method Adapting coordinate descent techniques from Euclidean space to optimal transport for log-concave densities.
result Proves convergence of CAVI for log-concave densities and provides rates of convergence under additional conditions.
Bayesian SPCA method tackles orthogonality constraint with spike and slab prior.
problem Bayesian SPCA method for high-dimensional data with orthogonality constraint.
method Parameter-expanded coordinate ascent variational inference (PX-CAVI) with spike and slab prior.
result PX-CAVI algorithm outperforms existing SPCA approaches in performance.
CAVI speeds up Bayesian MIDAS regression by 107x-1,772x with similar accuracy.
problem Efficiently estimating Bayesian MIDAS regression models with many predictors.
method Coordinate Ascent Variational Inference (CAVI) for linear MIDAS regression.
result CAVI produces posterior means nearly identical to Gibbs sampling with significant speedup.
Paper introduces a new text clustering model using Beta-Liouville priors.
problem Clustering short text data.
method Develops a hierarchical mixture model with Beta-Liouville priors for short text clustering.
result The Beta-Liouville distribution offers a more flexible correlation structure for short text clustering.
Gradient-based optimization improves variational empirical Bayes regression.
problem Sparse, large-scale multiple regression models.
method Gradient-based optimization (GradVI) for variational empirical Bayes (VEB) regression.
result GradVI produces similar predictive performance to CAVI but converges faster and is faster in certain settings.
Gradient-free method improves predictive accuracy for probabilistic models.
problem Balancing computational efficiency and robust predictive performance in deep learning.
method CAVI-CMN, a gradient-free variational method for conditional mixture networks.
result CAVI-CMN achieves competitive and often superior predictive accuracy compared to MLE with backpropagation.
A new particle algorithm improves mean-field variational inference.
problem Efficiently approximating nonparametric posterior distributions in machine learning.
method Introduces PArticle VI (PAVI), a novel particle-based algorithm for nonparametric mean-field approximation.
result Obtains non-asymptotic error bounds for PArticle VI, providing the first end-to-end guarantee for particle-based MFVI.
Improved Kalman filtering with hierarchical variational approach.
problem Inconsistent process covariance estimation and slow convergence speed in traditional variational Kalman filtering.
method Introducing a surrogate variable for process-noise-free state, reformulating CAVI, and sliding-window hyperparameter estimation.
result Enhanced convergence speed and superior estimation accuracy compared to existing methods.
We study a mean-field spike and slab variational Bayes (VB) approximation to Bayesian model selection priors in sparse high-dimensional linear regression. Under compatibility conditions on the design matrix, oracle inequalities are derived for the mean-field VB approximation, implying that it converges to the sparse tr…
VBphenoR uses variational Bayes for EHR-based patient phenotyping.
problem Phenotyping patients from EHR data for targeted treatments.
method Variational Bayes Gaussian Mixture Model (GMM) and logistic regression.
result Closed-form inference for efficient patient phenotype determination.
Random scan CAVI converges linearly under log-concave assumptions.
problem Analyzing the convergence rate of random scan Coordinate Ascent Variational Inference (CAVI) under log-concave conditions.
method Building on previous work, we analyze the random scan version of CAVI using optimal transport geometry.
result We obtain tight linear convergence rates for the random scan version of CAVI.
Nash integrates covariate-specific side info into sparse regression via neural networks.
problem Sparse linear regression struggles with covariates exhibiting structure or coming from heterogeneous sources.
method Neural Adaptive Shrinkage (Nash) framework that integrates side information into sparse regression via neural networks. Uses split variational empirical Bayes algorithm.
result Nash improves accuracy and adaptability over existing methods in real data experiments.
Improved Bayesian uncertainty quantification using variational bagging.
problem Inefficient and underestimating uncertainty in mean-field variational Bayes.
method Integrates bagging with variational Bayes for improved inference.
result Bagged variational posterior provides proper uncertainty quantification.
Bayesian Tensor Ring factorization improved for scalability and handling of discrete data.
problem Scalability issues and handling of discrete data in Bayesian Tensor Ring factorization.
method Proposes a novel Bayesian Tensor Ring model with a nonparametric Multiplicative Gamma Process prior and Pólya-Gamma augmentation for discrete data. Developed efficient Gibbs sampler and online EM algorithm for scalability.
result Significantly improved scalability and handling of discrete data compared to previous methods.
Paper introduces f-divergence variational inference for broader application.
problem Variational inference limited to specific divergences.
method Generalizes variational inference to all f-divergences using f-divergence minimization.
result Unified framework for variational inference with arbitrary f-divergences.
Bayesian model selection via mean-field variational approximation improves efficiency and accuracy.
problem Bayesian model selection under model mis-specification and latent variables.
method Mean-field variational approximation with non-asymptotic properties and geometric convergence.
result ELBO tends to select models closer to the true model than BIC as sample size increases.
Examines algorithmic modeling across three cultures.
problem Tackles algorithmic modeling in different cultural contexts.
method Uses parametric regressions, interpretable algorithms, and complex algorithms.
result Extension of Leo Breiman's thesis to include cultural differences.
Playing repeated matrix games (RMG) while maximizing the cumulative returns is a basic method to evaluate multi-agent learning (MAL) algorithms. Previous work has shown that U C B UCB U C B , M 3 M3 M 3 , S S S or E x p 3 Exp3 E x p 3 algorithms have good behaviours on average in RMG. Besides, hedging algorithms have been shown to be effective on predi…
Meta-algorithm selection aims to choose the best algorithm selector for a given problem instance.
problem Selecting the best algorithm selector for a specific problem instance.
method Apply algorithm selection to the selection of other algorithms (meta-algorithm selection).
result Meta-algorithm selection can be beneficial in some cases but faces challenges in solving the meta-level problem.
Proposes CLRS benchmark to evaluate algorithmic reasoning.
problem Difficulty in transferring results across publications due to targeted algorithmic data.
method Develops a comprehensive benchmark covering various algorithmic tasks.
result Demonstrates performance of algorithmic reasoning baselines on the CLRS benchmark.
Combines multiple bandit algorithms to create a nearly optimal single algorithm.
problem Designing a single bandit algorithm that performs nearly as well as the best individual algorithm in a stochastic environment.
method Develops two general corralling algorithms that achieve favorable regret guarantees.
result The regret of the corralling algorithms is no worse than the best individual algorithm's performance.
We propose accelerated randomized coordinate descent algorithms for stochastic optimization and online learning. Our algorithms have significantly less per-iteration complexity than the known accelerated gradient algorithms. The proposed algorithms for online learning have better regret performance than the known rando…
The exchange algorithm is studied for its convergence and asymptotic variance.
problem Theoretical limitations of the exchange algorithm in sampling from doubly-intractable distributions.
method Theoretical analysis of the exchange algorithm's convergence speed and asymptotic variance.
result The exchange algorithm converges at a geometric rate and satisfies a Central Limit Theorem.
Bayesian networks (BN) are used in a big range of applications but they have one issue concerning parameter learning. In real application, training data are always incomplete or some nodes are hidden. To deal with this problem many learning parameter algorithms are suggested foreground EM, Gibbs sampling and RBE algori…
No algorithm outperforms uniform sampling in A/B testing.
problem Identifying the best arm in A/B testing with fixed budget.
method Introducing consistent and stable algorithms, deriving lower bounds, and proving optimality of uniform sampling.
result No algorithm performs better than uniform sampling in A/B testing.
Improves algorithm selection for thousands of candidates using dyadic features.
problem Selecting the best algorithm from a large set of candidates for specific problems.
method Proposes extreme algorithm selection (XAS) with dyadic feature representation.
result Improves significantly over current state of the art in various metrics.
New ELM algorithms reduce computation time and complexity.
problem Efficient computation of extreme learning machine (ELM) algorithms.
method Developed inverse-free ELM algorithms using recursive matrix inverse and inverse LDL' factorization.
result Proposed algorithms significantly reduce computational complexity.
This review article surveys data augmentation MCMC algorithms.
problem Sampling from intractable probability distributions.
method Comprehensive study of DA MCMC algorithms, their convergence properties, and acceleration strategies.
result Synthesizes recent developments and provides insights for researchers.
Bayesian learning rule unifies and generalizes various machine learning algorithms.
problem Machine learning algorithms are diverse and not always understood.
method Bayesian principles and natural gradients are used to derive algorithms.
result Derives a wide range of algorithms including classical and modern ones.
Algorithm design is a laborious process and often requires many iterations of ideation and validation. In this paper, we explore automating algorithm design and present a method to learn an optimization algorithm, which we believe to be the first method that can automatically discover a better algorithm. We approach th…
This review summarizes five Lasso optimization algorithms.
problem Optimizing the Lasso objective function.
method Five representative algorithms: ISTA, FISTA, CGDA, SLA, PFA.
result Comparison of convergence rates and strengths/weaknesses.
Neural networks mimic algorithms to solve complex problems.
problem Current machine learning methods struggle with generalisation and efficiency.
method Representing algorithms in a continuous space and adapting them to real-world problems.
result Neural networks can execute classical algorithms more efficiently.
Paper proposes a reinforcement learning framework for efficient hyper-parameter tuning of stochastic optimization algorithms.
problem Efficient tuning of hyper-parameters for stochastic optimization algorithms.
method Modeling hyper-parameter tuning as a Markov decision process and using policy gradient algorithms.
result The proposed framework significantly reduces the time required for hyper-parameter tuning compared to Bayesian optimization.
Approximate probabilistic inference algorithms are central to many fields. Examples include sequential Monte Carlo inference in robotics, variational inference in machine learning, and Markov chain Monte Carlo inference in statistics. A key problem faced by practitioners is measuring the accuracy of an approximate infe…
In this paper, we propose a convergent parallel best-response algorithm with the exact line search for the nondifferentiable nonconvex sparsity-regularized rank minimization problem. On the one hand, it exhibits a faster convergence than subgradient algorithms and block coordinate descent algorithms. On the other hand,…
New algorithms reduce bilevel optimization complexity to ε^(-1.5).
problem Efficiently solving bilevel optimization problems in machine learning.
method Proposed two new algorithms: one using momentum-based recursive iterations, the other using recursive gradient estimations.
result Achieved computational complexity of ε^(-1.5), significantly faster than previous methods.
Researchers analyze how algorithmic and implementation choices affect RL performance.
problem Difficulty in separating algorithmic and implementation differences in RL performance.
method Unified derivations through a single control-as-inference objective, categorizing algorithms as EM or KL minimization.
result Implementation details are co-adapted with algorithmic choices, some transferable across algorithms.
Study on selecting between base algorithms in stochastic bandit problems.
problem Model selection in stochastic environments with contextual information.
method Developed a meta-algorithm-base algorithm abstraction with a smoothing transformation for optimal O ( T ) O(\sqrt{T}) O ( T ) guarantees. result Optimal O ( T ) O(\sqrt{T}) O ( T ) model selection guarantees for stochastic contextual bandit problems. New bounds derived for KG algorithm's performance in finite time.
problem Best arm identification problem in multi-armed bandit.
method Theoretical analysis of finite-time performance, deriving bounds for sample allocation, error probability, and regret.
result Upper and lower bounds for the probability of error and simple regret of the KG algorithm.
Paper proves linear convergence of SCMS algorithm for directional data.
problem Identifying density ridges in directional data.
method Generalized SCMS algorithm to directional data, derived from SCGA with adaptive step size.
result Linear convergence of the proposed directional SCMS algorithm.
MLE and CVE are equivalent under exponential families, leading to faster and more stable EM algorithms.
problem Finding maximum likelihood estimators (MLE) efficiently and stably.
method Proved equivalence between MLE and CVE under exponential families, leading to an EM algorithm.
result EM algorithm achieves the same asymptotic variance as MLE and is faster and more stable.
We resolve the fundamental problem of online decoding with general n t h n^{th} n t h order ergodic Markov chain models. Specifically, we provide deterministic and randomized algorithms whose performance is close to that of the optimal offline algorithm even when latency is small. Our algorithms admit efficient implementation vi…
New algorithm speeds up learning of graphical models.
problem Learning graphical models with sparse structure efficiently.
method Vertex-greedy score-based algorithm for learning DAGs.
result Polynomial runtime for learning DAG models.
The goal of data-driven algorithm design is to obtain high-performing algorithms for specific application domains using machine learning and data. Across many fields in AI, science, and engineering, practitioners will often fix a family of parameterized algorithms and then optimize those parameters to obtain good perfo…
The paper examines how algorithmic classification affects behavior and proposes democratizing stakes to mitigate predatory practices.
problem The impact of algorithmic classification on individual behavior and fairness in decision-making processes.
method Characterization of optimal classification by an algorithm designer and analysis of the effect of democratizing stakes.
result Optimal classification can lead to surprising behavior patterns, and democratizing stakes can mitigate predatory practices.
Run2Survive uses survival analysis for algorithm selection, outperforming traditional methods.
problem Handling censored runtime data in algorithm selection.
method Decision-theoretic approach leveraging survival analysis for censored data.
result Run2Survive outperforms state-of-the-art AS approaches in experiments.