arXiv research

The paper uses diffusion approximations to analyze and optimize online principal component estimation.

AdaOja improves Oja's algorithm for streaming PCA with adaptive learning rates.

New derivation shows how a three-factor learning rule is derived from Oja's rule.

Oja's rule improves neural network training without engineered tricks.

A new single-pass algorithm improves sparse PCA under limited computational resources.

We quantify uncertainty in Oja's algorithm's leading eigenvector estimation.

Paper improves Oja's algorithm for Markovian data streams.

Gen-Oja efficiently computes principal vectors and canonical correlations in streaming data.

Low-precision streaming PCA estimates the leading eigenvector with limited precision.

New algorithm reduces kernel PCA to $O(\sqrt{n})$ features for streaming data.

In this paper we are interested in the problem of learning an over-complete basis and a methodology such that the reconstruction or inverse problem does not need optimization. We analyze the optimality of the presented approaches, their link to popular already known techniques s.a. Artificial Neural Networks,k-means or…

We study streaming principal component analysis (PCA), that is to find, in $O(dk)$ space, the top $k$ eigenvectors of a $d\times d$ hidden matrix $\bf Σ$ with online vectors drawn from covariance matrix $\bf Σ$. We provide $\textit{global}$ convergence for Oja's algorithm which is popularly used in practice but lacks t…

Robust PCA method works under uncertain covariance.

Principal component analysis (PCA) is one of the most powerful tools in machine learning. The simplest method for PCA, the power iteration, requires $\mathcal O(1/Δ)$ full-data passes to recover the principal component of a matrix with eigen-gap $Δ$. Lanczos, a significantly more complex method, achieves an accelerated…

We present a high-dimensional analysis of three popular algorithms, namely, Oja's method, GROUSE and PETRELS, for subspace estimation from streaming and highly incomplete observations. We show that, with proper time scaling, the time-varying principal angles between the true subspace and its estimates given by the algo…

Algorithm estimates principal eigenvector with adaptive sensing, improving over non-adaptive methods.

Online PCA phase transition depends on $n/d\log(d)$, not just $n/d$.

New implicit Krasulina's k-PCA update avoids QR-decomposition and improves convergence.

We consider a situation in which we see samples in $\mathbb{R}^d$ drawn i.i.d. from some distribution with mean zero and unknown covariance A. We wish to compute the top eigenvector of A in an incremental fashion - with an algorithm that maintains an estimate of the top eigenvector in O(d) space, and incrementally adju…

This work provides improved guarantees for streaming principle component analysis (PCA). Given $A_1, \ldots, A_n\in \mathbb{R}^{d\times d}$ sampled independently from distributions satisfying $\mathbb{E}[A_i] = Σ$ for $Σ\succeq \mathbf{0}$, this work provides an $O(d)$-space linear-time single-pass streaming algorithm …

EigenGame reinterprets PCA as a game to find eigenvectors.

Stochastic optimization naturally arises in machine learning. Efficient algorithms with provable guarantees, however, are still largely missing, when the objective function is nonconvex and the data points are dependent. This paper studies this fundamental challenge through a streaming PCA problem for stationary time s…

Paper explores stochastic algorithms for PCA, focusing on convergence and runtime.

We study the problem of recovering the subspace spanned by the first $k$ principal components of $d$-dimensional data under the streaming setting, with a memory bound of $O(kd)$. Two families of algorithms are known for this problem. The first family is based on the framework of stochastic gradient descent. Nevertheles…

A neuron is a basic physiological and computational unit of the brain. While much is known about the physiological properties of a neuron, its computational role is poorly understood. Here we propose to view a neuron as a signal processing device that represents the incoming streaming data matrix as a sparse vector of …

pPCA speeds up PCA by priming initial estimates for faster, more accurate results.

AgFlow speeds up model selection in penalized PCA.

A new method for streaming PCA provides confidence intervals for eigenvector entries.

New algorithm learns principal subspace from random samples.

Improved online PCA algorithm learns from evolving norm of parameter vector.

Paper analyzes iterates in high-dimensional linear models and proposes estimators for their generalization error.

Paper analyzes iterative learning for concept classes and learns half-spaces.

WaveFit uses fixed-point iteration to create high-quality neural vocoders.

Last iterate of Extragradient algorithm converges slower than averaged iterates in saddle point problems.

In this paper, we introduce the notions of an iterated planar Lefschetz fibration and an iterated planar open book decomposition and prove the Weinstein conjecture for contact manifolds supporting an open book that has iterated planar pages. For $n\geq 1$, we show that a $(2n+1)$-dimensional contact manifold $M$ suppor…

We prove that an iterated torus knot type fails the uniform thickness property (UTP) if and only if all of its iterations are positive cablings, which is precisely when an iterated torus knot type supports the standard contact structure. We also show that all iterated torus knots that fail the UTP support cabling knot …

Sharp analysis of power iteration for tensor PCA, improving convergence and stopping criteria.

Paper proves exponential lower bounds for policy iteration in MDPs.

We deduce from the work of Chen, that the restriction morphism from closed free iterated integrals to closed iterated integrals on loops is onto. We use this to show that the module of higher order invariants of smooth functions is generated by free closed iterated integrals.

Adaptive optimal control of nonlinear dynamic systems with deterministic and known dynamics under a known undiscounted infinite-horizon cost function is investigated. Policy iteration scheme initiated using a stabilizing initial control is analyzed in solving the problem. The convergence of the iterations and the optim…

An iterative SE(3)-Transformer model is developed for graph data.

Study Ricci iteration on spheres and projective spaces, proving existence and convergence.

Stability of Einstein metrics under Ricci iteration studied.

Proposes a faster second-order method for MDPs.

Iterative method learns unknown constraints for MPC control.

OMWU shows last iterate convergence in convex-concave games.

Paper introduces Ricci iteration for studying Kähler-Einstein metrics.

Iterates towards Kähler metrics with constant scalar curvature.