AdaOja improves Oja's algorithm for streaming PCA with adaptive learning rates.
problem Lack of standardized learning rates in Oja's algorithm for streaming PCA.
method Proposes AdaOja, a novel learning rate scheme for Oja's method.
result AdaOja outperforms common learning rate choices and performs comparably to state-of-the-art algorithms.
A new single-pass algorithm improves sparse PCA under limited computational resources.
problem Sparse PCA with limited computational budget.
method Thresholding Oja's algorithm output.
result Achieves minimax error bound in O(d) space and O(nd) time.
We quantify uncertainty in Oja's algorithm's leading eigenvector estimation.
problem Estimating the error of Oja's algorithm's leading eigenvector from streaming data.
method Combining U-statistics, high-dimensional central limit theorems, and multiplier bootstrap.
result Established a weighted χ² approximation for the error between the eigenvector and algorithm output.
Paper improves Oja's algorithm for Markovian data streams.
problem Estimating the top eigenvector of a covariance matrix from Markovian data.
method Improves Oja's algorithm for streaming PCA with Markovian dependence.
result First sharp rate for Oja's algorithm on entire data stream, removing sample size dependence.
Gen-Oja efficiently computes principal vectors and canonical correlations in streaming data.
problem Principal Generalized Eigenvector computation and Canonical Correlation Analysis in stochastic settings.
method Gen-Oja is a simple and efficient algorithm that leverages two-time-scale stochastic approximation and fast-mixing Markov chains.
result Gen-Oja achieves optimal convergence rates for these problems.
New algorithm reduces kernel PCA to O ( n ) O(\sqrt{n}) O ( n ) features for streaming data.
problem Efficiently performing kernel PCA on large datasets.
method Random Fourier features and Oja's algorithm for streaming.
result Achieves O ( 1 / ε 2 ) O(1/ε^2) O ( 1/ ε 2 ) sample complexity with O ( n ) O(\sqrt{n}) O ( n ) features. New derivation shows how a three-factor learning rule is derived from Oja's rule.
problem Deriving a three-factor learning rule from Oja's rule.
method Using frame theory to systematically derive EGHR-PCA from Oja's rule.
result A principled derivation of a biologically plausible learning rule.
Low-precision streaming PCA estimates the leading eigenvector with limited precision.
problem Estimating the leading eigenvector in a streaming setting with limited precision.
method Oja's algorithm with linear and nonlinear stochastic quantization.
result A batched version of the quantized variants achieves the lower bound on quantization error up to logarithmic factors.
Oja's rule improves neural network training without engineered tricks.
problem Training deep neural networks with biological constraints.
method Incorporating Oja's plasticity rule into error-driven training.
result Stable, efficient learning in feedforward and recurrent architectures.
We study streaming principal component analysis (PCA), that is to find, in O ( d k ) O(dk) O ( d k ) space, the top k k k eigenvectors of a d × d d\times d d × d hidden matrix Σ \bf Σ Σ with online vectors drawn from covariance matrix Σ \bf Σ Σ . We provide global \textit{global} global convergence for Oja's algorithm which is popularly used in practice but lacks t…
The paper uses diffusion approximations to analyze and optimize online principal component estimation.
problem Optimizing online principal component estimation from streaming data.
method Diffusion approximation tools applied to Oja's iteration for principal component analysis.
result The Oja's iteration for the top eigenvector generates a continuous-state discrete-time Markov chain over the unit sphere.
Robust PCA method works under uncertain covariance.
problem Principal component analysis under uncertain covariance.
method Robust streaming PCA with temporal uncertainty set.
result Noisy power method is rate-optimal in our setting.
In this paper we are interested in the problem of learning an over-complete basis and a methodology such that the reconstruction or inverse problem does not need optimization. We analyze the optimality of the presented approaches, their link to popular already known techniques s.a. Artificial Neural Networks,k-means or…
We present a high-dimensional analysis of three popular algorithms, namely, Oja's method, GROUSE and PETRELS, for subspace estimation from streaming and highly incomplete observations. We show that, with proper time scaling, the time-varying principal angles between the true subspace and its estimates given by the algo…
Online PCA phase transition depends on n / d log ( d ) n/d\log(d) n / d log ( d ) , not just n / d n/d n / d .
problem Understanding the phase transition in online PCA.
method Study of Oja's algorithm for online PCA.
result Phase transition in correlation between estimated and true eigenvector.
This work provides improved guarantees for streaming principle component analysis (PCA). Given A 1 , … , A n ∈ R d × d A_1, \ldots, A_n\in \mathbb{R}^{d\times d} A 1 , … , A n ∈ R d × d sampled independently from distributions satisfying E [ A i ] = Σ \mathbb{E}[A_i] = Σ E [ A i ] = Σ for Σ ⪰ 0 Σ\succeq \mathbf{0} Σ ⪰ 0 , this work provides an O ( d ) O(d) O ( d ) -space linear-time single-pass streaming algorithm …
We consider a situation in which we see samples in R d \mathbb{R}^d R d drawn i.i.d. from some distribution with mean zero and unknown covariance A. We wish to compute the top eigenvector of A in an incremental fashion - with an algorithm that maintains an estimate of the top eigenvector in O(d) space, and incrementally adju…
New implicit Krasulina's k-PCA update avoids QR-decomposition and improves convergence.
problem Online k-PCA problem with orthonormality constraint.
method Derived an implicit form of Krasulina's update that bypasses orthonormality constraint.
result The new update avoids costly QR-decomposition and yields superior convergence.
EigenGame reinterprets PCA as a game to find eigenvectors.
problem Finding principal components efficiently and accurately.
method EigenGame treats PCA as a Nash equilibrium game, using gradient-based updates.
result EigenGame algorithm combines Oja's rule and Gram-Schmidt orthogonalization.
Stochastic optimization naturally arises in machine learning. Efficient algorithms with provable guarantees, however, are still largely missing, when the objective function is nonconvex and the data points are dependent. This paper studies this fundamental challenge through a streaming PCA problem for stationary time s…
We study the problem of recovering the subspace spanned by the first k k k principal components of d d d -dimensional data under the streaming setting, with a memory bound of O ( k d ) O(kd) O ( k d ) . Two families of algorithms are known for this problem. The first family is based on the framework of stochastic gradient descent. Nevertheles…
Paper explores stochastic algorithms for PCA, focusing on convergence and runtime.
problem Finding top k eigen vectors of data covariance matrix.
method Revisits and analyzes stochastic approaches to PCA optimization.
result Stochastic methods offer comparable or superior empirical performance to direct non-convex methods.
pPCA speeds up PCA by priming initial estimates for faster, more accurate results.
problem Improving the speed and accuracy of principal component analysis (PCA).
method pPCA is a two-step algorithm: first, an approximate-PCA method primes the data, then exact PCA is applied in the span of the initial estimate.
result pPCA improves accuracy significantly with a small computational cost, outperforming other methods across various datasets.
A neuron is a basic physiological and computational unit of the brain. While much is known about the physiological properties of a neuron, its computational role is poorly understood. Here we propose to view a neuron as a signal processing device that represents the incoming streaming data matrix as a sparse vector of …
A new method for streaming PCA provides confidence intervals for eigenvector entries.
problem Uncertainty quantification for individual entries in streaming PCA.
method Oja's algorithm, Bernstein-type concentration bound, Central Limit Theorem, subsampling algorithm.
result Sharp concentration bound and Central Limit Theorem for streaming PCA entries.
Improved online PCA algorithm learns from evolving norm of parameter vector.
problem Discarding evolving norm in online PCA leads to suboptimal learning.
method Implicitly Normalized Online PCA (INO-PCA) removes unit-norm constraint.
result Parameter norm evolution leads to improved learning behavior.
New algorithm learns principal subspace from random samples.
problem Learning principal subspace from small random submatrices.
method Stochastic gradient descent algorithm for neural networks.
result Algorithm can handle infinite rows and columns.
Principal component analysis (PCA) is one of the most powerful tools in machine learning. The simplest method for PCA, the power iteration, requires O ( 1 / Δ ) \mathcal O(1/Δ) O ( 1/Δ ) full-data passes to recover the principal component of a matrix with eigen-gap Δ Δ Δ . Lanczos, a significantly more complex method, achieves an accelerated…
Algorithm estimates principal eigenvector with adaptive sensing, improving over non-adaptive methods.
problem Estimating principal eigenvector with limited scalar measurements.
method Compressed variant of Oja's algorithm using two adaptive measurements per sample.
result Convergence rate of O ( λ 1 λ 2 d 2 / ( Δ 2 t ) ) \mathcal{O}(λ_1λ_2 d^2 / (Δ^2 t)) O ( λ 1 λ 2 d 2 / ( Δ 2 t )) after t t t iterations, matching information-theoretic lower bound. AgFlow speeds up model selection in penalized PCA.
problem Efficient model selection in penalized PCA for HDLSS settings.
method Implicit regularization effect of gradient flow to reduce computation complexity.
result AgFlow achieves the complete solution path of L2-penalized PCA.
This paper shows hedging algorithms improve performance in repeated matrix games.
problem Improving multi-agent learning algorithms in repeated matrix games.
method Develops and experiments with hedging algorithms combining a top-level and a set of basic algorithms.
result Well-selected hedging algorithms outperform previous MAL algorithms on repeated matrix games.
Examines algorithmic modeling across three cultures.
problem Tackles algorithmic modeling in different cultural contexts.
method Uses parametric regressions, interpretable algorithms, and complex algorithms.
result Extension of Leo Breiman's thesis to include cultural differences.
Meta-algorithm selection aims to choose the best algorithm selector for a given problem instance.
problem Selecting the best algorithm selector for a specific problem instance.
method Apply algorithm selection to the selection of other algorithms (meta-algorithm selection).
result Meta-algorithm selection can be beneficial in some cases but faces challenges in solving the meta-level problem.
Proposes CLRS benchmark to evaluate algorithmic reasoning.
problem Difficulty in transferring results across publications due to targeted algorithmic data.
method Develops a comprehensive benchmark covering various algorithmic tasks.
result Demonstrates performance of algorithmic reasoning baselines on the CLRS benchmark.
Combines multiple bandit algorithms to create a nearly optimal single algorithm.
problem Designing a single bandit algorithm that performs nearly as well as the best individual algorithm in a stochastic environment.
method Develops two general corralling algorithms that achieve favorable regret guarantees.
result The regret of the corralling algorithms is no worse than the best individual algorithm's performance.
We propose accelerated randomized coordinate descent algorithms for stochastic optimization and online learning. Our algorithms have significantly less per-iteration complexity than the known accelerated gradient algorithms. The proposed algorithms for online learning have better regret performance than the known rando…
The exchange algorithm is studied for its convergence and asymptotic variance.
problem Theoretical limitations of the exchange algorithm in sampling from doubly-intractable distributions.
method Theoretical analysis of the exchange algorithm's convergence speed and asymptotic variance.
result The exchange algorithm converges at a geometric rate and satisfies a Central Limit Theorem.
New algorithms optimize algorithm parameters in online settings with reduced computational costs.
problem Optimizing algorithm parameters in online settings with volatile and discontinuous losses.
method Developed semi-bandit optimization algorithms that leverage extra information to reduce computational costs.
result Achieved regret bounds as good as full-information feedback with significantly less computational effort.
Bayesian networks (BN) are used in a big range of applications but they have one issue concerning parameter learning. In real application, training data are always incomplete or some nodes are hidden. To deal with this problem many learning parameter algorithms are suggested foreground EM, Gibbs sampling and RBE algori…
No algorithm outperforms uniform sampling in A/B testing.
problem Identifying the best arm in A/B testing with fixed budget.
method Introducing consistent and stable algorithms, deriving lower bounds, and proving optimality of uniform sampling.
result No algorithm performs better than uniform sampling in A/B testing.
Improves algorithm selection for thousands of candidates using dyadic features.
problem Selecting the best algorithm from a large set of candidates for specific problems.
method Proposes extreme algorithm selection (XAS) with dyadic feature representation.
result Improves significantly over current state of the art in various metrics.
New algorithms decode Markov chains with near-optimal performance, even with small latency.
problem Online decoding of n t h n^{th} n t h order ergodic Markov chains with latency constraints. method Deterministic and randomized algorithms using dynamic programs, with lower bounds established.
result Near-optimal performance of algorithms with minimal latency, outperforming existing methods.
Combines online learning algorithms to achieve better performance.
problem Improving online learning algorithms with varying guarantees.
method Adding iterates of two parameter-free algorithms to create a new algorithm with improved regret.
result Generates efficient algorithms that adapt to multiple norms and maintain dimension-free guarantees.
New ELM algorithms reduce computation time and complexity.
problem Efficient computation of extreme learning machine (ELM) algorithms.
method Developed inverse-free ELM algorithms using recursive matrix inverse and inverse LDL' factorization.
result Proposed algorithms significantly reduce computational complexity.
This review article surveys data augmentation MCMC algorithms.
problem Sampling from intractable probability distributions.
method Comprehensive study of DA MCMC algorithms, their convergence properties, and acceleration strategies.
result Synthesizes recent developments and provides insights for researchers.
Bayesian learning rule unifies and generalizes various machine learning algorithms.
problem Machine learning algorithms are diverse and not always understood.
method Bayesian principles and natural gradients are used to derive algorithms.
result Derives a wide range of algorithms including classical and modern ones.
Algorithm design is a laborious process and often requires many iterations of ideation and validation. In this paper, we explore automating algorithm design and present a method to learn an optimization algorithm, which we believe to be the first method that can automatically discover a better algorithm. We approach th…
This review summarizes five Lasso optimization algorithms.
problem Optimizing the Lasso objective function.
method Five representative algorithms: ISTA, FISTA, CGDA, SLA, PFA.
result Comparison of convergence rates and strengths/weaknesses.