Adapts SDCA for CRFs, improving performance on sequence prediction tasks.
problem Training conditional random fields (CRFs) efficiently.
method Adapts SDCA algorithm with adaptive sampling strategy for CRFs training.
result SDCA outperforms state-of-the-art methods on three out of four sequence prediction tasks.
SDCA converges linearly in statistical estimation without strong convexity.
problem Statistical estimation problems not necessarily strongly convex.
method Stochastic dual coordinate (SDCA) without strong convexity assumption.
result SDCA converges linearly under restricted strong convexity conditions.
Uniform sampling of training data has been commonly used in traditional stochastic optimization algorithms such as Proximal Stochastic Gradient Descent (prox-SGD) and Proximal Stochastic Dual Coordinate Ascent (prox-SDCA). Although uniform sampling can guarantee that the sampled stochastic quantity is an unbiased estim…
Paper proves linear convergence of R-FDM and RC-FDM under weak strong convexity.
problem Optimizing SVM dual problem and LASSO problem.
method Randomized feasible descent method (R-FDM) and coordinate-wise random feasible descent method (RC-FDM).
result Both R-FDM and RC-FDM converge linearly under weak strong convexity assumption.
Stochastic dual coordinate ascent (SDCA) is an effective technique for solving regularized loss minimization problems in machine learning. This paper considers an extension of SDCA under the mini-batch setting that is often used in practice. Our main contribution is to introduce an accelerated mini-batch version of SDC…
In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for compos…
Using recycled samples can improve runtime for variance reduction methods.
problem Improving runtime efficiency with variance reduction methods.
method Recycling previously used samples instead of fresh ones for SDCA, SAG, and SVRG.
result Optimal sample size and integer number of epochs can be wasteful for SDCA.
A new ADMM variant reduces storage needs while maintaining fast convergence.
problem High storage requirements in ADMM variants.
method Integrates SVRG with ADMM, reducing storage needs.
result Converges faster and uses less memory than existing methods.
Unified analysis of SAGA, Finito, SDCA using jump systems and quadratic constraints.
problem Analyzing convergence rates of stochastic optimization methods.
method Incorporating jump system theory and quadratic constraints to derive convergence rate certifications.
result Derives linear matrix inequalities (LMIs) for convergence rates of SAGA, Finito, and SDCA.
This paper introduces AdaSDCA: an adaptive variant of stochastic dual coordinate ascent (SDCA) for solving the regularized empirical risk minimization problems. Our modification consists in allowing the method adaptively change the probability distribution over the dual variables throughout the iterative process. AdaSD…
Stochastic Gradient Descent (SGD) has become popular for solving large scale supervised machine learning optimization problems such as SVM, due to their strong theoretical guarantees. While the closely related Dual Coordinate Ascent (DCA) method has been implemented in various software packages, it has so far lacked go…
Improves SDCA convergence for convex objectives with linear constraints.
problem Minimizing convex objectives with linear constraints under gradient-Lipschitz assumption failure.
method Shifted Stochastic Dual Coordinate Ascent (SDCA) under smoothness assumption.
result Obtains linear convergence rate for Poisson regression and Hawkes process objectives.
Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning. In this paper, we propose a communication-efficient framework, CoCoA, that uses local computation in a primal-dual setting to dramatically reduce the amount of necessary comm…
New algorithm for ERM with flexible mini-batching and non-convex losses.
problem Regularized empirical risk minimization with non-convex losses and arbitrary mini-batching.
method Extends SDCA techniques to arbitrary mini-batching schemes, focusing on non-convex losses.
result Achieves convergence for non-convex losses with dual-free analysis.
Unified view of stochastic optimization methods with improved convergence and robustness.
problem Stochastic convex composite optimization with noise.
method Estimate sequence approach, accelerated algorithms, robust strategies.
result Optimal complexity accelerated SVRG algorithm robust to noise.
Catalyst accelerates convex optimization methods with practical guidelines.
problem Improving convergence of gradient-based optimization methods.
method Catalyst scheme based on inexact accelerated proximal point algorithm.
result Faster convergence rates for various convex optimization methods.
Accelerates ADMM for convex problems with linear convergence.
problem Improving convergence rate of ADMM for general convex problems.
method Introduces momentum acceleration trick into SVRG-ADMM for both strongly convex and general convex cases.
result ASVRG-ADMM achieves O(1/T^2) convergence rate for general convex problems.
New findings on optimizing finite sum problems with variance reduction and acceleration.
problem Conditions for efficient variance reduction and acceleration in finite sum optimization.
method Analysis of first-order and coordinate-descent finite sum algorithms.
result Optimal complexity bounds for minimizing L-smooth and convex finite sums.
New lower bounds for gradient methods in strongly convex finite-sum optimization.
problem Developing tight lower bounds for randomized gradient methods in finite-sum optimization.
method Deriving tight lower complexity bounds for SAG, SAGA, SVRG, SARAH, and related methods.
result Tight matches between lower bounds and upper bounds for various methods under specific conditions.
This paper shows hedging algorithms improve performance in repeated matrix games.
problem Improving multi-agent learning algorithms in repeated matrix games.
method Develops and experiments with hedging algorithms combining a top-level and a set of basic algorithms.
result Well-selected hedging algorithms outperform previous MAL algorithms on repeated matrix games.
Examines algorithmic modeling across three cultures.
problem Tackles algorithmic modeling in different cultural contexts.
method Uses parametric regressions, interpretable algorithms, and complex algorithms.
result Extension of Leo Breiman's thesis to include cultural differences.
Meta-algorithm selection aims to choose the best algorithm selector for a given problem instance.
problem Selecting the best algorithm selector for a specific problem instance.
method Apply algorithm selection to the selection of other algorithms (meta-algorithm selection).
result Meta-algorithm selection can be beneficial in some cases but faces challenges in solving the meta-level problem.
Proposes CLRS benchmark to evaluate algorithmic reasoning.
problem Difficulty in transferring results across publications due to targeted algorithmic data.
method Develops a comprehensive benchmark covering various algorithmic tasks.
result Demonstrates performance of algorithmic reasoning baselines on the CLRS benchmark.
Combines multiple bandit algorithms to create a nearly optimal single algorithm.
problem Designing a single bandit algorithm that performs nearly as well as the best individual algorithm in a stochastic environment.
method Develops two general corralling algorithms that achieve favorable regret guarantees.
result The regret of the corralling algorithms is no worse than the best individual algorithm's performance.
New algorithms improve stochastic optimization and online learning efficiency.
problem Efficient optimization and online learning algorithms for stochastic problems.
method Accelerated randomized coordinate descent algorithms.
result Significantly less per-iteration complexity and better regret performance.
Automates algorithm design using reinforcement learning.
problem Manual algorithm design is time-consuming and error-prone.
method Guided policy search for optimizing policies representing algorithms.
result Learned algorithm outperforms existing hand-engineered algorithms.
The exchange algorithm is studied for its convergence and asymptotic variance.
problem Theoretical limitations of the exchange algorithm in sampling from doubly-intractable distributions.
method Theoretical analysis of the exchange algorithm's convergence speed and asymptotic variance.
result The exchange algorithm converges at a geometric rate and satisfies a Central Limit Theorem.
New algorithms optimize algorithm parameters in online settings with reduced computational costs.
problem Optimizing algorithm parameters in online settings with volatile and discontinuous losses.
method Developed semi-bandit optimization algorithms that leverage extra information to reduce computational costs.
result Achieved regret bounds as good as full-information feedback with significantly less computational effort.
Bayesian networks (BN) are used in a big range of applications but they have one issue concerning parameter learning. In real application, training data are always incomplete or some nodes are hidden. To deal with this problem many learning parameter algorithms are suggested foreground EM, Gibbs sampling and RBE algori…
Parallel algorithm finds sparse solutions for nonconvex problems.
problem Nonconvex sparsity-regularized rank minimization.
method Parallel best-response algorithm with exact line search.
result Guaranteed convergence to a stationary point.
No algorithm outperforms uniform sampling in A/B testing.
problem Identifying the best arm in A/B testing with fixed budget.
method Introducing consistent and stable algorithms, deriving lower bounds, and proving optimality of uniform sampling.
result No algorithm performs better than uniform sampling in A/B testing.
Improves algorithm selection for thousands of candidates using dyadic features.
problem Selecting the best algorithm from a large set of candidates for specific problems.
method Proposes extreme algorithm selection (XAS) with dyadic feature representation.
result Improves significantly over current state of the art in various metrics.
AIDE measures the accuracy of probabilistic inference algorithms.
problem Measuring the accuracy of approximate inference algorithms on specific data sets.
method AIDE is an algorithm based on viewing inference algorithms as probabilistic models and auxiliary variables.
result AIDE captures the qualitative behavior of inference algorithms and detects failure modes.
New algorithm improves worst Value-at-Risk computation for risky portfolios.
problem Computing worst Value-at-Risk in heterogeneous portfolios is numerically challenging.
method Introduced an Adaptive Rearrangement Algorithm to improve the Rearrangement Algorithm.
result The Adaptive Rearrangement Algorithm provides more accurate approximations of worst Value-at-Risk.
New algorithms decode Markov chains with near-optimal performance, even with small latency.
problem Online decoding of n t h n^{th} n t h order ergodic Markov chains with latency constraints. method Deterministic and randomized algorithms using dynamic programs, with lower bounds established.
result Near-optimal performance of algorithms with minimal latency, outperforming existing methods.
Combines online learning algorithms to achieve better performance.
problem Improving online learning algorithms with varying guarantees.
method Adding iterates of two parameter-free algorithms to create a new algorithm with improved regret.
result Generates efficient algorithms that adapt to multiple norms and maintain dimension-free guarantees.
New ELM algorithms reduce computation time and complexity.
problem Efficient computation of extreme learning machine (ELM) algorithms.
method Developed inverse-free ELM algorithms using recursive matrix inverse and inverse LDL' factorization.
result Proposed algorithms significantly reduce computational complexity.
This review article surveys data augmentation MCMC algorithms.
problem Sampling from intractable probability distributions.
method Comprehensive study of DA MCMC algorithms, their convergence properties, and acceleration strategies.
result Synthesizes recent developments and provides insights for researchers.
Bayesian learning rule unifies and generalizes various machine learning algorithms.
problem Machine learning algorithms are diverse and not always understood.
method Bayesian principles and natural gradients are used to derive algorithms.
result Derives a wide range of algorithms including classical and modern ones.
This review summarizes five Lasso optimization algorithms.
problem Optimizing the Lasso objective function.
method Five representative algorithms: ISTA, FISTA, CGDA, SLA, PFA.
result Comparison of convergence rates and strengths/weaknesses.
Neural networks mimic algorithms to solve complex problems.
problem Current machine learning methods struggle with generalisation and efficiency.
method Representing algorithms in a continuous space and adapting them to real-world problems.
result Neural networks can execute classical algorithms more efficiently.
Paper proposes a reinforcement learning framework for efficient hyper-parameter tuning of stochastic optimization algorithms.
problem Efficient tuning of hyper-parameters for stochastic optimization algorithms.
method Modeling hyper-parameter tuning as a Markov decision process and using policy gradient algorithms.
result The proposed framework significantly reduces the time required for hyper-parameter tuning compared to Bayesian optimization.
Describes MLC search spaces in MEKA and WEKA software.
problem Understanding MLC algorithms and their transformations into SLC problems.
method Overviewed 26 MLC algorithms and 28 SLC algorithms, proposed a context-free grammar.
result Formal description of MLC search spaces and their transformations.
New algorithms reduce bilevel optimization complexity to ε^(-1.5).
problem Efficiently solving bilevel optimization problems in machine learning.
method Proposed two new algorithms: one using momentum-based recursive iterations, the other using recursive gradient estimations.
result Achieved computational complexity of ε^(-1.5), significantly faster than previous methods.
Researchers analyze how algorithmic and implementation choices affect RL performance.
problem Difficulty in separating algorithmic and implementation differences in RL performance.
method Unified derivations through a single control-as-inference objective, categorizing algorithms as EM or KL minimization.
result Implementation details are co-adapted with algorithmic choices, some transferable across algorithms.
Study on selecting between base algorithms in stochastic bandit problems.
problem Model selection in stochastic environments with contextual information.
method Developed a meta-algorithm-base algorithm abstraction with a smoothing transformation for optimal O ( T ) O(\sqrt{T}) O ( T ) guarantees. result Optimal O ( T ) O(\sqrt{T}) O ( T ) model selection guarantees for stochastic contextual bandit problems. New bounds derived for KG algorithm's performance in finite time.
problem Best arm identification problem in multi-armed bandit.
method Theoretical analysis of finite-time performance, deriving bounds for sample allocation, error probability, and regret.
result Upper and lower bounds for the probability of error and simple regret of the KG algorithm.
Paper proves linear convergence of SCMS algorithm for directional data.
problem Identifying density ridges in directional data.
method Generalized SCMS algorithm to directional data, derived from SCGA with adaptive step size.
result Linear convergence of the proposed directional SCMS algorithm.