arXiv research

Paper unifies subspace identification and DMD for dynamical systems.

Study explores GAN dynamics for high-dimensional subspace learning.

SGD updates align with a low-rank subspace but do not lead to further loss reduction.

Multivariate functional data from a complex system are naturally high-dimensional and have complex cross-correlation structure. The complexity of data structure can be observed as that (1) some functions are strongly correlated with similar features, while some others may have almost no cross-correlations with quite di…

Dynamic robust PCA refers to the dynamic (time-varying) extension of robust PCA (RPCA). It assumes that the true (uncorrupted) data lies in a low-dimensional subspace that can change with time, albeit slowly. The goal is to track this changing subspace over time in the presence of sparse outliers. We develop and study …

Efficiently extracts linear dynamics from complex observations.

This work takes the first steps towards solving the "phaseless subspace tracking" (PST) problem. PST involves recovering a time sequence of signals (or images) from phaseless linear projections of each signal under the following structural assumption: the signal sequence is generated from a much lower dimensional subsp…

New algorithm catches moving subspaces in bandit problems.

New algorithms extract Koopman invariant subspaces from large-scale data.

A new DDR framework learns low-dimensional data representations using dynamical systems.

ISOKANN learns collective variables and effective dynamics for metastable transitions.

A new method tackles bilevel optimization using Lanczos process for efficient hyper-gradient computation.

Paper proves IRLS converges to subspace from any start, with practical benefits.

Analyzes word2vec-like models revealing linear subspaces learned during training.

We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of tr…

Study non-asymptotic bounds on correlation in high-dimensional linear systems, revealing invariant subspaces and bottlenecks.

Studying a softmax-attention model, we show that the learned query converges to the latent signal subspace spanned by the informative direction.

Diffusion models' sampling paths lie in a low-dimensional subspace, resembling boomerangs.

Spectral decomposition of the Koopman operator is attracting attention as a tool for the analysis of nonlinear dynamical systems. Dynamic mode decomposition is a popular numerical algorithm for Koopman spectral analysis; however, we often need to prepare nonlinear observables manually according to the underlying dynami…

New algorithm updates eigenvectors of evolving graphs efficiently.

We consider the problem of online subspace tracking of a partially observed high-dimensional data stream corrupted by noise, where we assume that the data lie in a low-dimensional linear subspace. This problem is cast as an online low-rank tensor completion problem. We propose a novel online tensor subspace tracking al…

In this work, we study the robust subspace tracking (RST) problem and obtain one of the first two provable guarantees for it. The goal of RST is to track sequentially arriving data vectors that lie in a slowly changing low-dimensional subspace, while being robust to corruption by additive sparse outliers. It can also b…

Operator-theoretic analysis of nonlinear dynamical systems has attracted much attention in a variety of engineering and scientific fields, endowed with practical estimation methods using data such as dynamic mode decomposition. In this paper, we address a lifted representation of nonlinear dynamical systems with random…

Diffusion models improve creativity by smoothing the score function, leading to interpolated data.

We present a framework for supervised subspace tracking, when there are two time series $x_t$ and $y_t$, one being the high-dimensional predictors and the other being the response variables and the subspace tracking needs to take into consideration of both sequences. It extends the classic online subspace tracking work…

Proposes PredVAR model for reduced-dimensional dynamics from noisy data.

Online detection of abrupt changes in high-dimensional data streams.

LASER compresses recursive model activations by exploiting their low-dimensional structure.

New method prevents forgetting in LLMs by dynamically identifying task-specific subspaces.

In an era of ubiquitous large-scale streaming data, the availability of data far exceeds the capacity of expert human analysts. In many settings, such data is either discarded or stored unprocessed in datacenters. This paper proposes a method of online data thinning, in which large-scale streaming datasets are winnowed…

Paper projects GP basis functions using tensor networks to reduce complexity.

The ECME algorithm has proven to be an effective way of accelerating the EM algorithm for many problems. Recognising the limitation of using prefixed acceleration subspace in ECME, we propose the new Dynamic ECME (DECME) algorithm which allows the acceleration subspace to be chosen dynamically. Our investigation of an …

In this paper we define currents relative to a free factor system. We prove that a fully irreducible outer automorphism relative to a free factor system acts with uniform north-south dynamics on a subspace of the space of projective relative currents.

Principal Components Analysis (PCA) is one of the most widely used dimension reduction techniques. Robust PCA (RPCA) refers to the problem of PCA when the data may be corrupted by outliers. Recent work by Cand{è}s, Wright, Li, and Ma defined RPCA as a problem of decomposing a given data matrix into the sum of a low-ran…

We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime. Recent work of \citep{Gale2019, Liu2018} has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that a…

The paper finds Koopman invariant subspaces using personalized PageRank.

The paper develops methods to reduce deployment risk under dynamic covariate shifts.

Emergent misalignment is influenced by training dynamics, model priors, and data.

New framework tracks communities in dynamic networks.

A method for noise reduction in functional time series using FPCA.

FastForest boosts Random Forest speed by 24%.

The development of a metric for structural data is a long-term problem in pattern recognition and machine learning. In this paper, we develop a general metric for comparing nonlinear dynamical systems that is defined with Perron-Frobenius operators in reproducing kernel Hilbert spaces. Our metric includes the existing …

Efficiently compress overparameterized deep models by focusing on low-dimensional learning dynamics.

Gradient flow solves multi-index regression for high-dimensional Gaussian data.

TFPS improves time series forecasting by learning pattern-specific experts.

We consider Schrödinger operators $H=-Δ_{g_\varepsilon} + V$ on a fibre bundle $M\stackrelπ{\to}B$ with compact fibres and a metric $g_\varepsilon$ that blows up directions perpendicular to the fibres by a factor ${\varepsilon^{-1}\gg 1}$. We show that for an eigenvalue $λ$ of the fibre-wise part of $H$, satisfying a l…

We present a high-dimensional analysis of three popular algorithms, namely, Oja's method, GROUSE and PETRELS, for subspace estimation from streaming and highly incomplete observations. We show that, with proper time scaling, the time-varying principal angles between the true subspace and its estimates given by the algo…

New insights into how deep models generalize, focusing on matrix factorization.