This work connects LLE, factor analysis, and probabilistic PCA through a stochastic perspective.
problem Exploring the theoretical connection between LLE, factor analysis, and probabilistic PCA.
method Solving the stochastic linear reconstruction of LLE using expectation maximization.
result LLE, factor analysis, and probabilistic PCA are shown to be connected through a stochastic perspective.
Probabilistic Autoencoder learns latent space weights' distribution.
problem Nonlinear model reconstruction error and sample quality.
method Normalizing flow for latent space weights' probability distribution.
result PAE achieves small reconstruction errors, high sample quality, and good performance.
Principal Component Analysis (PCA) is a popular tool for dimensionality reduction and feature extraction in data analysis. There is a probabilistic version of PCA, known as Probabilistic PCA (PPCA). However, standard PCA and PPCA are not robust, as they are sensitive to outliers. To alleviate this problem, this paper i…
HeMPPCAT improves PCA for data with varying noise.
problem PCA's suboptimal performance on data with heterogeneous noise.
method HeMPPCAT uses a GEM algorithm to estimate factors, means, and noise variances.
result Improved factor estimates and clustering accuracy compared to MPPCA.
Paper develops a dual formulation for PCA in Hilbert spaces.
problem Characterizing probabilistic PCA in Hilbert spaces.
method Dual formulation for probabilistic PCA in Hilbert spaces.
result Generative framework for kernel methods developed.
Sparse versions of principal component analysis (PCA) have imposed themselves as simple, yet powerful ways of selecting relevant features of high-dimensional data in an unsupervised manner. However, when several sparse principal components are computed, the interpretation of the selected variables is difficult since ea…
Paper revisits PCA for anomaly detection in network security.
problem Understanding and improving anomaly detection in network security.
method Revisit probabilistic PCA model and its connection to MSNM framework.
result Mathematical model connects PCA to MSNM for anomaly detection.
MN-PCA models structured noise in data and feature spaces.
problem Complex and structured noise in real-world data.
method Matrix normal distribution for structured noise modeling, generalized Mahalanobis distance approximation.
result MN-PCA obtains a low-rank data representation and structured noise simultaneously.
Auto-Associative models cover a large class of methods used in data analysis. In this paper, we describe the generals properties of these models when the projection component is linear and we propose and test an easy to implement Probabilistic Semi-Linear Auto- Associative model in a Gaussian setting. We show it is a g…
We present a method to compute the Shapley values of reconstruction errors of principal component analysis (PCA), which is particularly useful in explaining the results of anomaly detection based on PCA. Because features are usually correlated when PCA-based anomaly detection is applied, care must be taken in computing…
We consider probabilistic PCA and related factor models from a Bayesian perspective. These models are in general not identifiable as the likelihood has a rotational symmetry. This gives rise to complicated posterior distributions with continuous subspaces of equal density and thus hinders efficiency of inference as wel…
Survey of factor analysis, PCA, variational inference, and VAE.
problem Dimensionality reduction and generative modeling of data.
method Variational inference, factor analysis, probabilistic PCA, and VAE.
result Derivation and explanation of ELBO, EM, and closed-form solutions.
CAVI converges exponentially fast for Bayesian PCA models.
problem Characterizing the convergence speed of CAVI for BPCA.
method Proved exponential convergence using power iteration analogy and novel lower bounds.
result Exponential convergence of CAVI for BPCA models with any number of principal components.
Paper proposes GPM for heteroscedastic PCA estimation.
problem Estimating ground truth from heterogeneous data.
method Generalized power method (GPM) for HQPOC.
result GPM achieves geometrically decreasing distances to ground truth.
PGPCA improves PCA for nonlinear data in neuroscience.
problem Nonlinear data distribution in neuroscience.
method Developed PGPCA for nonlinear manifolds, incorporating EM algorithm.
result PGPCA outperforms PPCA in modeling data around nonlinear manifolds.
We shed new insights on the two commonly used updates for the online k-PCA problem, namely, Krasulina's and Oja's updates. We show that Krasulina's update corresponds to a projected gradient descent step on the Stiefel manifold of the orthonormal k-frames, while Oja's update amounts to a gradient descent step using…
Proposes MPCA for robust PCA using mode estimation.
problem Outliers sensitivity in PCA.
method Modal Principal Component Analysis (MPCA) based on mode estimation.
result MPCA shows advantages over conventional methods.
This paper improves PPCA robustness using t-distributions.
problem Improving robustness of probabilistic PCA.
method Using multivariate t-distributions and a hierarchical model. result Clarified the correct correspondence between the multivariate t-PPCA framework and the hierarchical model. We present a unifying framework which reduces the construction of probabilistic component analysis techniques to a mere selection of the latent neighbourhood, thus providing an elegant and principled framework for creating novel component analysis models as well as constructing probabilistic equivalents of deterministi…
Principal component analysis (PCA) is arguably the most popular tool in multivariate exploratory data analysis. In this paper, we consider the question of how to handle heterogeneous variables that include continuous, binary, and ordinal. In the probabilistic interpretation of low-rank PCA, the data has a normal multiv…
Autoencoders are a deep learning model for representation learning. When trained to minimize the distance between the data and its reconstruction, linear autoencoders (LAEs) learn the subspace spanned by the top principal directions but cannot learn the principal directions themselves. In this paper, we prove that $L_2…
Linear VAEs explain posterior collapse in VAEs via local maxima in log marginal likelihood.
problem Posterior collapse in VAEs where variational posterior matches prior for some latent variables.
method Analysis of linear VAEs and their relation to pPCA, proving ELBO does not introduce spurious local maxima.
result Linear VAEs have identifiable global maxima corresponding to principal component directions, explaining posterior collapse.
RFPCA improves robustness of FPCA for matrix data.
problem Outliers in matrix data degrade the performance of FPCA.
method RFPCA uses matrix-variate t-distribution and EM algorithm for robust estimation.
result RFPCA outperforms other methods in detecting matrix-valued outliers.
A novel 3D shape registration method using spectral graph embedding and probabilistic matching.
problem Challenges in 3D shape analysis and registration, especially with large variability.
method Combining spectral graph matching with Laplacian embedding for large graphs, using commute-time embedding and PCA.
result A method to register shapes with different samplings and isometric deformations.
Efficient and high-fidelity prior sampling and inversion for complex geological media is still a largely unsolved challenge. Here, we use a deep neural network of the variational autoencoder type to construct a parametric low-dimensional base model parameterization of complex binary geological media. For inversion purp…
In the era of big data, reducing data dimensionality is critical in many areas of science. Widely used Principal Component Analysis (PCA) addresses this problem by computing a low dimensional data embedding that maximally explain variance of the data. However, PCA has two major weaknesses. Firstly, it only considers li…
Normalizing flows optimize Jacobian determinant for unique likelihood objective.
problem Optimizing normalizing flows for unique likelihood.
method Showed Jacobian determinant is unique for given distributions, leading to a unique global optimum. Used eigenvalues of auto-correlation matrix for explicit likelihood expression.
result Explicit expression of likelihood for flows, independent of neural network parameterization, with theoretical optimal value.
Using the linear Gaussian latent variable model as a starting point we relax some of the constraints it imposes by deriving a nonparametric latent feature Gaussian variable model. This model introduces additional discrete latent variables to the original structure. The Bayesian nonparametric nature of this new model al…
Dimensionality reduction on Riemannian manifolds is challenging due to the complex nonlinear data structures. While probabilistic principal geodesic analysis~(PPGA) has been proposed to generalize conventional principal component analysis (PCA) onto manifolds, its effectiveness is limited to data with a single modality…
PCA whitening weighted by Zipfian word frequencies improves task performance.
problem Skewed word embedding spaces in neural models.
method PCA whitening weighted by empirical word frequencies following Zipf's law.
result Significantly improves task performance, surpassing baselines.
Introduces a probabilistic framework for dimension reduction methods.
problem Lack of clear probabilistic foundations for popular DR methods.
method A unifying statistical framework based on the coupling of hidden graphs using cross entropy.
result Existing DR methods suffer from a statistical deficiency that affects performance.
ProbDR framework interprets DR algorithms as probabilistic inference.
problem Efficiently compressing high-dimensional data into lower dimensions.
method ProbDR variational framework that treats DR as probabilistic inference.
result ProbDR unifies various DR algorithms and enables probabilistic reasoning.
This paper proposes a probabilistic imputation method with uncertainty quantification.
problem Missing value imputation with uncertainty estimation for large datasets.
method Low Rank Gaussian Copula framework that augments PPCA with column-specific transformations.
result The method yields state-of-the-art imputation accuracy and well-calibrated uncertainty estimates.
An important preprocessing step in most data analysis pipelines aims to extract a small set of sources that explain most of the data. Currently used algorithms for blind source separation (BSS), however, often fail to extract the desired sources and need extensive cross-validation. In contrast, their rarely used probab…
Simplifies fair PCA with fast, efficient solution.
problem Learning fair low-rank approximations of data.
method Conceptually simple approach with analytic solution.
result Faster and similar results to existing fair PCA methods.
Generative neural networks model multivariate time series data.
problem Modeling cross-sectional dependence in multivariate time series data.
method ARMA-GARCH for serial dependence, PCA for dimensionality reduction, GMMN for cross-sectional dependence.
result GMMN-GARCH approach produces better predictive distributions and probabilistic forecasts.
KAN-PCA improves asset return analysis by capturing more variance than classical PCA during market crises.
problem Inefficient classical PCA during market crises when correlations between assets change dramatically.
method KAN-PCA uses KAN (Kolmogorov-Arnold Networks) with B-spline functions to learn nonlinear projections.
result KAN-PCA achieves a higher reconstruction R^2 (66.57%) compared to classical PCA (62.99%) on 20 S&P 500 stocks.
New algorithm solves fair PCA, robust PCA, and sparse PCA problems efficiently.
problem Fair Principal Component Analysis (FPCA) to ensure fairness in PCA solutions.
method Iterative MM algorithm with SDP reformulation to quadratic program.
result Algorithm monotonically improves fairness objectives at each iteration.
Unified framework improves PCA for outliers and distributed data.
problem Outliers and limitations in PCA for large-scale applications.
method φ-PCA framework that retains PCA efficiency and adds robustness.
result HM-PCA achieves optimal robustness and efficiency.
HPPCA improves imputation of longitudinal data with missing values.
problem Handling incomplete, high-dimensional longitudinal data with nested sources of variation and temporal dependency.
method Hierarchical probabilistic principal component analysis (HPPCA) with a two-level latent factor model and Gaussian process.
result HPPCA outperforms standard PPCA and multivariate functional PCA in imputation accuracy, even under heavy missingness and model misspecification.
TL-PCA uses transfer learning to improve PCA performance with limited target data.
problem PCA performance is limited with scarce target data.
method Transfer learning approach to PCA (TL-PCA) that combines source task knowledge with target task data.
result Improved PCA representation for dimensionality reduction with limited target data.
Bucketed PCA-NN outperforms DNNs by 96% on MNIST.
problem Benchmarking deep neural networks for supervised classification.
method Applies PCA to individual buckets constructed in two phases, retains neural network architecture, and uses neurons that mirror input signals.
result Bucketed PCA-NN achieves 96% accuracy on MNIST, similar to DNNs.
A new low-dimensional parameterization based on principal component analysis (PCA) and convolutional neural networks (CNN) is developed to represent complex geological models. The CNN-PCA method is inspired by recent developments in computer vision using deep learning. CNN-PCA can be viewed as a generalization of an ex…
New method improves calibration in multi-output probabilistic models.
problem Challenges in achieving multivariate calibration in multi-output regression.
method General regularization framework to enforce multivariate calibration during training for arbitrary pre-rank functions.
result Significant improvement in calibration across all pre-rank functions without sacrificing predictive accuracy.
Anchor PCA improves robustness in multi-domain PCA.
problem PCA on pooled data can focus on spurious directions.
method Anchor PCA focuses on shared directions of variation.
result Anchor PCA outperforms pooling and worst-case alternatives.
A new method for fair PCA ensures balanced error across groups.
problem Balancing approximation error across different groups in multi-group data.
method Iterative method to compute fair principal components minimizing max group-wise reconstruction error.
result Preserves the containment property of standard PCA and reduces to standard PCA for single-group data.
This is a detailed tutorial paper which explains the Principal Component Analysis (PCA), Supervised PCA (SPCA), kernel PCA, and kernel SPCA. We start with projection, PCA with eigen-decomposition, PCA with one and multiple projection directions, properties of the projection matrix, reconstruction error minimization, an…
Revisits PCA with new formulations and insights.
problem Improving PCA formulations and understanding.
method Difference-of-convex (DC) framework, kernelizability, out-of-sample applicability, simultaneous iteration, DCA perspective.
result PCA-like problems are kernelizable and have new optimization perspectives.