Paper connects contrastive learning to MI maximization and establishes robust methods for nonlinear ICA and subspace estimation.
problem Understanding and improving unsupervised representation learning and density ratio estimation.
method The paper connects contrastive learning to MI maximization, establishes new recovery conditions for nonlinear ICA, and proposes a practical outlier-robust method for nonlinear subspace estimation.
result The proposed methods can be seen as maximizing MI, performing nonlinear ICA, or estimating nonlinear subspaces, and are robust to outliers.
Develops a method for causal inference with noisy confounders.
problem Noisy measurements of confounders in treatment effects models.
method Local principal subspace approximation combining K-nearest neighbors matching and PCA.
result Estimators of treatment effects and counterfactual distributions are constructed.
Operator-theoretic analysis of nonlinear dynamical systems has attracted much attention in a variety of engineering and scientific fields, endowed with practical estimation methods using data such as dynamic mode decomposition. In this paper, we address a lifted representation of nonlinear dynamical systems with random…
A recent theoretical analysis shows the equivalence between non-negative matrix factorization (NMF) and spectral clustering based approach to subspace clustering. As NMF and many of its variants are essentially linear, we introduce a nonlinear NMF with explicit orthogonality and derive general kernel-based orthogonal m…
We develop embeddings for nonlinear subspaces preserving vector norms.
problem Preserving vector norms in nonlinear subspaces.
method Low-distortion embeddings for subspaces under nonlinear transformations.
result First low-distortion embeddings for a wide class of nonlinear functions.
DKLM learns adaptive kernels for robust nonlinear subspace clustering.
problem Nonlinear structures in data and challenges with kernel-based clustering.
method Data-driven kernel learning with adaptive weighting and optimal block-diagonal affinity matrix.
result DKLM enhances robustness and preserves manifold structure in nonlinear space.
Paper extends ICA to ISA with auxiliary variables for better speech representation learning.
problem Learning unsupervised speech representations with independent subspaces.
method Theoretical framework of nonlinear ISA with auxiliary variables.
result Proposes an algorithm to learn speech representations with independent subspaces.
This work improves sample efficiency in meta-learning for nonlinear tasks.
problem Learning complex tasks efficiently with limited data.
method Subspace-based representations for nonlinear tasks.
result Subspace-based representations can be learned efficiently and improve future task performance.
Kernel-based methods enjoy powerful generalization capabilities in handling a variety of learning tasks. When such methods are provided with sufficient training data, broadly-applicable classes of nonlinear functions can be approximated with desired accuracy. Nevertheless, inherent to the nonparametric nature of kernel…
Spectral decomposition of the Koopman operator is attracting attention as a tool for the analysis of nonlinear dynamical systems. Dynamic mode decomposition is a popular numerical algorithm for Koopman spectral analysis; however, we often need to prepare nonlinear observables manually according to the underlying dynami…
We introduce the probabilistic sequential matrix factorization (PSMF) method for factorizing time-varying and non-stationary datasets consisting of high-dimensional time-series. In particular, we consider nonlinear Gaussian state-space models where sequential approximate inference results in the factorization of a data…
Proposes a method to reveal nonlinearities in tensor data.
problem Capturing nonlinear relationships in high-dimensional tensor data.
method Linear tensor projection method to maximize prediction accuracy.
result Effective in revealing nonlinear relationships in tensor data.
A new DDR framework learns low-dimensional data representations using dynamical systems.
problem Learning efficient low-dimensional data representations.
method DDR framework based on nonlinear dynamical systems, using linear combinations of functions and regularization.
result DDR method outperforms other methods on synthetic and real datasets.
Most recent results in matrix completion assume that the matrix under consideration is low-rank or that the columns are in a union of low-rank subspaces. In real-world settings, however, the linear structure underlying these models is distorted by a (typically unknown) nonlinear transformation. This paper addresses the…
Paper proposes S2ConvSCN for robust subspace clustering and classification.
problem Insufficient handling of nonlinear manifolds, data corruptions, and out-of-sample data.
method Self-supervised convolutional subspace clustering network (S2ConvSCN) with FC layer, CIM for robustness, and BD regularization. result Robust S2ConvSCN outperforms baseline on unseen data. Modern information processing relies on the axiom that high-dimensional data lie near low-dimensional geometric structures. This paper revisits the problem of data-driven learning of these geometric structures and puts forth two new nonlinear geometric models for data describing "related" objects/phenomena. The first o…
Proposes a tensor Laplacian-based method for better subspace clustering of non-uniformly distributed data.
problem LRR's inability to handle non-uniform data distribution and local information loss.
method Tensor Laplacian Regularized Low-Rank Representation (TLRR) using hypergraph model and tensor Laplacian algorithm.
result Higher accuracy and precision in subspace clustering compared to state-of-the-art methods.
New algorithms extract Koopman invariant subspaces from large-scale data.
problem Difficulty in discerning the Koopman invariant subspace from many Koopman eigenmodes.
method Multi-task feature learning and pruning procedure to remove spurious modes.
result Effective in approximating Koopman operator for complex flows.
Reservoir subspace injection improves online ICA by preserving injected features.
problem Discarding injected features in top-n whitening can degrade performance. method Formalized reservoir subspace injection (RSI) and developed diagnostics (IER, SSO, ρ_x) to identify and mitigate the failure mode.
result RSI controller preserves passthrough retention, improving performance by up to 2.2 dB.
Paper uses Koopman operator and Nyström method for efficient nonlinear control.
problem Control of nonlinear dynamical systems.
method Combines Koopman operator framework with Nyström approximation for kernel methods.
result Theoretical guarantees on the convergence rates of the approximated Riccati operator and regulator objective.
The development of a metric for structural data is a long-term problem in pattern recognition and machine learning. In this paper, we develop a general metric for comparing nonlinear dynamical systems that is defined with Perron-Frobenius operators in reproducing kernel Hilbert spaces. Our metric includes the existing …
Paper bounds subspace estimator error from noisy projections.
problem Estimating subspaces from noisy data.
method Derives perturbation bound on optimal subspace estimator.
result Fundamental result with implications in matrix completion and clustering.
Proposes σ-PCA to learn identifiable linear transformations without whitening.
problem Cannot identify axes with equal variances in PCA.
method Unified model for linear and nonlinear PCA, introducing a missing piece to eliminate rotational indeterminacy.
result Eliminates subspace rotational indeterminacy in PCA.
Bayesian methods reduce variance in subspace identification for small data sets.
problem High variance in traditional subspace identification methods for large models or small sample sizes.
method Investigation of Bayesian estimation solutions (regularized and shrinkage estimators) for subspace identification.
result Bayesian estimators reduce estimation risk by up to 40% compared to traditional methods.
In this letter, we consider two sets of observations defined as subspace signals embedded in noise and we wish to analyze the distance between these two subspaces. The latter entails evaluating the angles between the subspaces, an issue reminiscent of the well-known Procrustes problem. A Bayesian approach is investigat…
Additive principal components (APCs for short) are a nonlinear generalization of linear principal components. We focus on smallest APCs to describe additive nonlinear constraints that are approximately satisfied by the data. Thus APCs fit data with implicit equations that treat the variables symmetrically, as opposed t…
Paper develops methods for PCA inference with missing data and heteroskedastic noise.
problem Constructing confidence regions for PCA in high dimensions with missing data and heteroskedastic noise.
method Proposes HeteroPCA and develops non-asymptotic distributional guarantees for valid inference.
result Valid inference on principal subspace and spiked covariance matrix with missing data.
Efficiently extracts linear dynamics from complex observations.
problem Learning policies directly from rich, high-dimensional observations.
method Modeling linear dynamics in a hidden subspace and developing an efficient algorithm.
result Successfully extracts linear dynamics from rich observations.
New method recovers matrices with nonlinear structures using optimization on Grassmann manifold.
problem Recovering high-rank matrices with nonlinear structures like subspaces or clusters.
method Formulated as rank minimization of a nonlinear feature map, approximated by constrained non-convex optimization on the Grassmann manifold, using Riemannian and alternating minimization schemes.
result Global convergence and worst-case complexity bounds for alternating minimization scheme, leading to unique limit point.
This paper considers the problem of robust subspace recovery: given a set of N points in RD, if many lie in a d-dimensional subspace, then can we recover the underlying subspace? We show that Tyler's M-estimator can be used to recover the underlying subspace, if the percentage of the inliers is larger t…
Most existing approaches address multi-view subspace clustering problem by constructing the affinity matrix on each view separately and afterwards propose how to extend spectral clustering algorithm to handle multi-view data. This paper presents an approach to multi-view subspace clustering that learns a joint subspace…
Unified framework for structured principal subspace estimation with bounds and rates.
problem Structured principal subspace estimation problems.
method Unified framework, minimax lower and upper bounds, information-geometric complexity.
result Minimax rates of convergence for specific settings, including optimal rates for non-negative PCA/SVD.
Theoretical guarantees for STE, a robust subspace recovery method.
problem Recovering a low-dimensional subspace from corrupted data.
method Subspace-constrained Tyler's estimator (STE) with initialization conditions.
result STE can effectively recover the subspace under certain conditions.
New method identifies latent components in PNL mixtures without strong assumptions.
problem Identifying latent components in PNL mixtures under unknown nonlinear functions.
method Carefully designed UML criterion to identify a null space associated with the mixing system.
result Identification/removal of unknown nonlinearity under minimal conditions.
Despite the fact that nonlinear subspace learning techniques (e.g. manifold learning) have successfully applied to data representation, there is still room for improvement in explainability (explicit mapping), generalization (out-of-samples), and cost-effectiveness (linearization). To this end, a novel linearized subsp…
We propose a novel adaptive learning algorithm based on iterative orthogonal projections in the Cartesian product of multiple reproducing kernel Hilbert spaces (RKHSs). The task is estimating/tracking nonlinear functions which are supposed to contain multiple components such as (i) linear and nonlinear components, (ii)…
New method estimates active subspaces for jump-discontinuous functions.
problem Estimating active subspaces for discontinuous functions like ABMs.
method Extending active subspaces to discontinuous functions, using Gaussian process.
result Identifies important parameters in ABM simulations of refugee movement.
Extends importance sampling to nonlinear models using adjoint operators.
problem Lack of tools for identifying important data points in nonlinear models.
method Introduces adjoint operator for nonlinear maps, generalizes norm and leverage scores.
result Generalized scores provide approximation guarantees for nonlinear mappings.
Kernel methods obtain superb performance in terms of accuracy for various machine learning tasks since they can effectively extract nonlinear relations. However, their time complexity can be rather large especially for clustering tasks. In this paper we define a general class of kernels that can be easily approximated …
We study sparse principal components analysis in high dimensions, where p (the number of variables) can be much larger than n (the number of observations), and analyze the problem of estimating the subspace spanned by the principal eigenvectors of the population covariance matrix. We introduce two complementary not…
A method for identifying joint and individual subspaces from multi-view data.
problem Unclear conditions for reliably identifying joint and individual subspaces from noisy, high-dimensional measurements.
method Rigorously quantifies conditions based on signal rank, principal angles, and noise levels. Characterizes spectrum perturbations of product of projection matrices.
result Estimates joint and individual subspaces more accurately than existing approaches in simulations and real-world applications.
Proposes a new algorithm to estimate invariant subspaces across multilayer networks.
problem Estimating invariant subspaces across heterogeneous multiple networks.
method Bias-corrected joint spectral embedding algorithm that recursively calibrates diagonal bias and iteratively updates the subspace estimator.
result Established entrywise subspace perturbation bound and entrywise eigenvector central limit theorem for the algorithm.
This paper contains a thorough introduction to the basic geometric properties of the manifold of Lagrangian subspaces of a linear symplectic space, known as the Lagrangian Grassmannian. It also reviews the important relationship between hypersurfaces in the Lagrangian Grassmannian and second-order PDEs.
Study optimizes shared singular subspace estimation from noisy matrices.
problem Estimating shared singular subspaces across multiple noisy matrices.
method Low-rank matrix denoising framework with Stack-SVD and novel estimators.
result Stack-SVD achieves minimax rate-optimality for identical shared subspaces, and novel estimators for partial sharing.
Scientists and engineers rely on accurate mathematical models to quantify the objects of their studies, which are often high-dimensional. Unfortunately, high-dimensional models are inherently difficult, i.e. when observations are sparse or expensive to determine. One way to address this problem is to approximate the or…
Paper proves IRLS converges to subspace from any start, with practical benefits.
problem Robust subspace estimation in machine learning.
method Iteratively Reweighted Least Squares (IRLS) with dynamic smoothing regularization.
result IRLS converges linearly to the underlying subspace from any initialization under deterministic conditions.
GDMaps reduces high-dimensional data to lower dimensions for better classification.
problem High-dimensional data classification and representation.
method Grassmannian Diffusion Maps technique for nonlinear dimensionality reduction.
result GDMaps effectively identifies intrinsic subspace structures in high-dimensional data.
This paper analyzes AJIVE for estimating shared subspace across multiple datasets, revealing its strengths and limitations.
problem Estimating shared subspace across multiple datasets with varying degrees of misalignment.
method Angle-based Joint and Individual Variation Explained (AJIVE) method, a two-stage spectral approach.
result AJIVE's performance in high signal-to-noise ratio (SNR) regimes and its non-diminishing error in low-SNR settings.