Deep-RLS uses deep learning to improve PCA for better source separation.
problem Improving PCA for better source separation in nonlinear systems.
method Inspired by RLS, Deep-RLS unfolds RLS iterations into a deep neural network.
result Deep-RLS significantly improves accuracy in recovering source signals.
Linear principal component analysis (PCA) can be extended to a nonlinear PCA by using artificial neural networks. But the benefit of curved components requires a careful control of the model complexity. Moreover, standard techniques for model selection, including cross-validation and more generally the use of an indepe…
Proposes σ-PCA to learn identifiable linear transformations without whitening.
problem Cannot identify axes with equal variances in PCA.
method Unified model for linear and nonlinear PCA, introducing a missing piece to eliminate rotational indeterminacy.
result Eliminates subspace rotational indeterminacy in PCA.
KAN-PCA improves asset return analysis by capturing more variance than classical PCA during market crises.
problem Inefficient classical PCA during market crises when correlations between assets change dramatically.
method KAN-PCA uses KAN (Kolmogorov-Arnold Networks) with B-spline functions to learn nonlinear projections.
result KAN-PCA achieves a higher reconstruction R^2 (66.57%) compared to classical PCA (62.99%) on 20 S&P 500 stocks.
We solve a high-dimensional model where nonlinear autoencoders detect hidden structure missed by PCA.
problem Hidden structure in high-dimensional data not detected by PCA.
method Tractable spiked model with two latent factors, one visible and one uncorrelated.
result Nonlinear autoencoders can extract hidden structure missed by PCA, even if reconstruction loss is higher.
Principal component analysis (PCA) is largely adopted for chemical process monitoring and numerous PCA-based systems have been developed to solve various fault detection and diagnosis problems. Since PCA-based methods assume that the monitored process is linear, nonlinear PCA models, such as autoencoder models and kern…
In this paper, we propose and study a Nyström based approach to efficient large scale kernel principal component analysis (PCA). The latter is a natural nonlinear extension of classical PCA based on considering a nonlinear feature map or the corresponding kernel. Like other kernel approaches, kernel PCA enjoys good mat…
Classical methods such as Principal Component Analysis (PCA) and Canonical Correlation Analysis (CCA) are ubiquitous in statistics. However, these techniques are only able to reveal linear relationships in data. Although nonlinear variants of PCA and CCA have been proposed, these are computationally prohibitive in the …
Low-precision streaming PCA estimates the leading eigenvector with limited precision.
problem Estimating the leading eigenvector in a streaming setting with limited precision.
method Oja's algorithm with linear and nonlinear stochastic quantization.
result A batched version of the quantized variants achieves the lower bound on quantization error up to logarithmic factors.
For many years, a combination of principal component analysis (PCA) and independent component analysis (ICA) has been used for blind source separation (BSS). However, it remains unclear why these linear methods work well with real-world data that involve nonlinear source mixtures. This work theoretically validates that…
This work connects LLE, factor analysis, and probabilistic PCA through a stochastic perspective.
problem Exploring the theoretical connection between LLE, factor analysis, and probabilistic PCA.
method Solving the stochastic linear reconstruction of LLE using expectation maximization.
result LLE, factor analysis, and probabilistic PCA are shown to be connected through a stochastic perspective.
PGPCA improves PCA for nonlinear data in neuroscience.
problem Nonlinear data distribution in neuroscience.
method Developed PGPCA for nonlinear manifolds, incorporating EM algorithm.
result PGPCA outperforms PPCA in modeling data around nonlinear manifolds.
Many problems in machine learning and statistics can be formulated as (generalized) eigenproblems. In terms of the associated optimization problem, computing linear eigenvectors amounts to finding critical points of a quadratic function subject to quadratic constraints. In this paper we show that a certain class of con…
A new low-dimensional parameterization based on principal component analysis (PCA) and convolutional neural networks (CNN) is developed to represent complex geological models. The CNN-PCA method is inspired by recent developments in computer vision using deep learning. CNN-PCA can be viewed as a generalization of an ex…
Paper develops methods for PCA inference with missing data and heteroskedastic noise.
problem Constructing confidence regions for PCA in high dimensions with missing data and heteroskedastic noise.
method Proposes HeteroPCA and develops non-asymptotic distributional guarantees for valid inference.
result Valid inference on principal subspace and spiked covariance matrix with missing data.
In the era of big data, reducing data dimensionality is critical in many areas of science. Widely used Principal Component Analysis (PCA) addresses this problem by computing a low dimensional data embedding that maximally explain variance of the data. However, PCA has two major weaknesses. Firstly, it only considers li…
Unified method learns latent spaces from labeled data.
problem Identifying low-dimensional latent structures in high-dimensional data.
method Nonlinear multiple-response regression within an index model context.
result Unified method offers better interpretability and reduced computational complexity.
Principal component analysis (PCA) is widely used for feature extraction and dimensionality reduction, with documented merits in diverse tasks involving high-dimensional data. Standard PCA copes with one dataset at a time, but it is challenged when it comes to analyzing multiple datasets jointly. In certain data scienc…
Improved online PCA algorithm learns from evolving norm of parameter vector.
problem Discarding evolving norm in online PCA leads to suboptimal learning.
method Implicitly Normalized Online PCA (INO-PCA) removes unit-norm constraint.
result Parameter norm evolution leads to improved learning behavior.
Probabilistic Autoencoder learns latent space weights' distribution.
problem Nonlinear model reconstruction error and sample quality.
method Normalizing flow for latent space weights' probability distribution.
result PAE achieves small reconstruction errors, high sample quality, and good performance.
Unified framework for fast large-scale portfolio optimization.
problem Efficient portfolio optimization for large-scale financial data.
method Incorporates shrinkage and regularization techniques, addressing multiple objectives.
result AP-Trees and PCA-based factor models consistently outperform other approaches in out-of-sample portfolio performance.
When confronted with massive data streams, summarizing data with dimension reduction methods such as PCA raises theoretical and algorithmic pitfalls. Principal curves act as a nonlinear generalization of PCA and the present paper proposes a novel algorithm to automatically and sequentially learn principal curves from d…
This paper introduces a new unsupervised method for dimensionality reduction via regression (DRR). The algorithm belongs to the family of invertible transforms that generalize Principal Component Analysis (PCA) by using curvilinear instead of linear features. DRR identifies the nonlinear features through multivariate r…
R-PCA extends PCA to Riemannian manifolds for structured data.
problem Applying PCA to data on Riemannian manifolds without vector space operations.
method Adapting PCA to Riemannian manifolds by equipping data with local metrics.
result Unified approach for dimensionality reduction and statistical analysis on manifolds.
PCA is a classical statistical technique whose simplicity and maturity has seen it find widespread use as an anomaly detection technique. However, it is limited in this regard by being sensitive to gross perturbations of the input, and by seeking a linear subspace that captures normal behaviour. The first issue has bee…
MPTE uses Transformer attention to estimate mixed-frequency factor models.
problem Estimating factor models in panel datasets with mixed frequencies and nonlinear signals.
method Mixed-Panels-Transformer Encoder (MPTE) with attention mechanisms.
result MPTE achieves competitive performance in nonlinear forecasting environments.
New approach interprets machine learning models through feature space transformations.
problem Interpreting models with strongly dependent features in high-dimensional spaces.
method Feature space transformations, including PCA and partial orthogonalization.
result Enhances model interpretation tools for domain experts.
Paper proposes a new descriptor for early trajectory characterization in matrix iterations.
problem Comparing early behavior of high-dimensional trajectories in nonlinear matrix iterations.
method Develops a two-channel fuzzy coordinate system using F-transform for compact representation.
result The descriptor achieves high R^2 values (mean = 0.6480) in approximating convergence lengths.
Derives a primal-dual MLSVD formulation for multilinear data.
problem Efficiently decompose multilinear data for signal analysis and deep learning.
method Kernelizable primal-dual formulation of MLSVD.
result Derives a new MLSVD formulation with computational advantages.
A new method for efficient nonlinear process monitoring using random Bernoulli features.
problem High computational demands and real-time responsiveness in online monitoring systems.
method Random Bernoulli principal component analysis to capture nonlinear patterns efficiently.
result The proposed methods offer excellent scalability and reduced computational complexity.
Constructing an efficient parameterization of a large, noisy data set of points lying close to a smooth manifold in high dimension remains a fundamental problem. One approach consists in recovering a local parameterization using the local tangent plane. Principal component analysis (PCA) is often the tool of choice, as…
We present a new technique called contrastive principal component analysis (cPCA) that is designed to discover low-dimensional structure that is unique to a dataset, or enriched in one dataset relative to other data. The technique is a generalization of standard PCA, for the setting where multiple datasets are availabl…
Simplifies fair PCA with fast, efficient solution.
problem Learning fair low-rank approximations of data.
method Conceptually simple approach with analytic solution.
result Faster and similar results to existing fair PCA methods.
In this paper, we propose novel strategies for neutral vector variable decorrelation. Two fundamental invertible transformations, namely serial nonlinear transformation and parallel nonlinear transformation, are proposed to carry out the decorrelation. For a neutral vector variable, which is not multivariate Gaussian d…
A new DDR framework learns low-dimensional data representations using dynamical systems.
problem Learning efficient low-dimensional data representations.
method DDR framework based on nonlinear dynamical systems, using linear combinations of functions and regularization.
result DDR method outperforms other methods on synthetic and real datasets.
A minimalist approach generates synthetic tabular data with sparse PCA and XGBoost.
problem Generating robust synthetic tabular data for model testing.
method Minimalistic unsupervised SparsePCA encoder with XGBoost decoder.
result The method provides an alternative to raw and quantile perturbation for model robustness testing.
New algorithm solves fair PCA, robust PCA, and sparse PCA problems efficiently.
problem Fair Principal Component Analysis (FPCA) to ensure fairness in PCA solutions.
method Iterative MM algorithm with SDP reformulation to quadratic program.
result Algorithm monotonically improves fairness objectives at each iteration.
Unified framework improves PCA for outliers and distributed data.
problem Outliers and limitations in PCA for large-scale applications.
method φ-PCA framework that retains PCA efficiency and adds robustness.
result HM-PCA achieves optimal robustness and efficiency.
TL-PCA uses transfer learning to improve PCA performance with limited target data.
problem PCA performance is limited with scarce target data.
method Transfer learning approach to PCA (TL-PCA) that combines source task knowledge with target task data.
result Improved PCA representation for dimensionality reduction with limited target data.
Bucketed PCA-NN outperforms DNNs by 96% on MNIST.
problem Benchmarking deep neural networks for supervised classification.
method Applies PCA to individual buckets constructed in two phases, retains neural network architecture, and uses neurons that mirror input signals.
result Bucketed PCA-NN achieves 96% accuracy on MNIST, similar to DNNs.
Anchor PCA improves robustness in multi-domain PCA.
problem PCA on pooled data can focus on spurious directions.
method Anchor PCA focuses on shared directions of variation.
result Anchor PCA outperforms pooling and worst-case alternatives.
A new method for fair PCA ensures balanced error across groups.
problem Balancing approximation error across different groups in multi-group data.
method Iterative method to compute fair principal components minimizing max group-wise reconstruction error.
result Preserves the containment property of standard PCA and reduces to standard PCA for single-group data.
This is a detailed tutorial paper which explains the Principal Component Analysis (PCA), Supervised PCA (SPCA), kernel PCA, and kernel SPCA. We start with projection, PCA with eigen-decomposition, PCA with one and multiple projection directions, properties of the projection matrix, reconstruction error minimization, an…
Revisits PCA with new formulations and insights.
problem Improving PCA formulations and understanding.
method Difference-of-convex (DC) framework, kernelizability, out-of-sample applicability, simultaneous iteration, DCA perspective.
result PCA-like problems are kernelizable and have new optimization perspectives.
Solution to sparse PCA tuning problem using Empirical Bayes.
problem Sparse PCA multiple tuning problem (MTP).
method Empirical Bayes covariance decomposition for penalized PCA.
result Empirical Bayes approach efficiently solves MTP in sparse PCA.
PCA++ improves robustness to background noise in contrastive learning.
problem Recovering shared signal subspaces from positive pairs in high-dimensional data with structured background noise.
method PCA++ uses hard uniformity-constrained contrastive learning to enforce identity covariance on projected features.
result PCA++ outperforms standard PCA and alignment-only PCA+ in simulations and real-world datasets.
Manifold methods improve amino acid classification in LIBS spectra.
problem Improving classification accuracy of amino acids in LIBS spectra.
method Developed an information theoretic method for measuring LIBS energy spectra, implemented manifold methods for nonlinear dimensionality reduction.
result Nonlinear methods lead to increased classification accuracy in amino acid classification.
EB-PCA reduces noise in high-dimensional PCA by estimating a joint prior distribution.
problem High-dimensional PCA noise in samples comparable to or larger than data.
method Empirical Bayes PCA using Kiefer-Wolfowitz MLE, random matrix theory, and AMP algorithm.
result EB-PCA achieves Bayes-optimal accuracy in spiked models and significantly improves over PCA in simulations and real data.