Study on dynamics of non-linear autoencoders learning principal components.
problem Technical difficulty in studying non-linear autoencoders due to non-trivial correlations.
method Derive asymptotically exact equations for SGD training of shallow, non-linear autoencoders.
result Autoencoders learn principal components sequentially and tie weights are ineffective.
The paper analyzes L2-regularized linear autoencoders and their loss landscapes.
problem Understanding the loss landscapes of L2-regularized linear autoencoders. method Smoothly parameterizing the critical manifold and relating minima to the MAP estimate of probabilistic PCA.
result Proves that L2-regularized LAEs learn principal directions as left singular vectors of the decoder. The autoencoder is an effective unsupervised learning model which is widely used in deep learning. It is well known that an autoencoder with a single fully-connected hidden layer, a linear activation function and a squared error cost function trains weights that span the same subspace as the one spanned by the principa…
The paper analyzes the generalizability of linear autoencoders and multivariate linear regression.
problem Limited theoretical understanding of linear autoencoders' performance.
method Proposes a PAC-Bayes bound for multivariate linear regression and shows LAEs as constrained models.
result The proposed PAC-Bayes bound is tight and correlates with practical metrics.
Soft-AE interprets autoencoders with adaptable wavelet units.
problem Lack of interpretability in autoencoders.
method Proposes Soft-AE with adaptable wavelet units and GenLU.
result Soft-AE offers interpretability and competitive performance.
Deep autoencoder network solves spectrum sharing problems efficiently.
problem Resource allocation in wireless communications and D2D networks.
method Generative neural network (autoencoder) for solving linear sum assignment problems.
result Hybrid autoencoder architecture outperforms other methods in accuracy and speed.
This work analyzes how bottleneck layers and skip connections affect linear denoising autoencoders' generalization.
problem Understanding the generalization of linear denoising autoencoders in overparameterized regimes.
method Analyzes two-layer linear denoising autoencoders with a bottleneck layer and skip connection, deriving test risk formulas.
result Bottleneck layers introduce an additional complexity measure, while skip connections can mitigate variance.
We solve a high-dimensional model where nonlinear autoencoders detect hidden structure missed by PCA.
problem Hidden structure in high-dimensional data not detected by PCA.
method Tractable spiked model with two latent factors, one visible and one uncorrelated.
result Nonlinear autoencoders can extract hidden structure missed by PCA, even if reconstruction loss is higher.
IRMAE learns compact latent spaces by minimizing rank.
problem Learning compact latent representations in autoencoders.
method Implicitly minimizes the rank of the covariance matrix through gradient descent in multi-layer linear networks.
result Demonstrates validity on image generation and representation learning tasks.
We study the limits and methods of training two-layer autoencoders.
problem Understanding the limits and methods of training two-layer autoencoders.
method Focus on non-linear two-layer autoencoders trained in the proportional regime, using gradient methods.
result Gradient methods achieve the minimizers of the population risk and reveal the structure of the features.
Simpler linear models outperform complex GCN encoders for graph tasks.
problem Complex graph autoencoders and variational autoencoders.
method Replacing GCN encoders with one-hop linear models.
result Simpler linear models achieve competitive performance with fewer operations and parameters.
Simple linear model outperforms GCN in graph AE tasks.
problem Challenging tasks like link prediction and node clustering.
method Replaced GCN with a simple linear model on adjacency matrix.
result Simple model consistently reaches competitive performances.
Improves supervised learning with target-embedding autoencoders.
problem Improving generalization in purely supervised settings with high-dimensional target spaces.
method Target-Embedding Autoencoders (TEA) for jointly optimizing latent representations for prediction and feature predictability.
result Guaranteed generalization for linear TEAs through uniform stability, and empirical validation across multivariate sequence forecasting.
The autoencoder is an artificial neural network model that learns hidden representations of unlabeled data. With a linear transfer function it is similar to the principal component analysis (PCA). While both methods use weight vectors for linear transformations, the autoencoder does not come with any indication similar…
We provide a series of results for unsupervised learning with autoencoders. Specifically, we study shallow two-layer autoencoder architectures with shared weights. We focus on three generative models for data that are common in statistical machine learning: (i) the mixture-of-gaussians model, (ii) the sparse coding mod…
Simple linear model outperforms deep models on sparse data.
problem Sparse data in recommender systems.
method Linear autoencoder with closed-form solution.
result Simple model achieves better ranking accuracy than deep models.
Paper uses autoencoders for efficient reduced-order modeling of eigenvalue problems.
problem Efficiently modeling eigenvalue problems in high dimensions.
method Autoencoder-based reduced-order modeling for eigenvalue problems.
result Autoencoder-based models outperform standard POD-Galerkin methods in neutron diffusion applications.
Inspired by the success of deep learning techniques in the physical and chemical sciences, we apply a modification of an autoencoder type deep neural network to the task of dimension reduction of molecular dynamics data. We can show that our time-lagged autoencoder reliably finds low-dimensional embeddings for high-dim…
Proposes a neural network autoencoder for smoothing and representation learning of functional data.
problem Lack of sufficient nonlinear representations in existing methods for functional data analysis.
method Develops a neural network autoencoder architecture to process functional data directly, learning both smoothing and representation.
result Outperforms traditional methods in prediction, classification, and computational efficiency.
Regularized LAEs learn principal components efficiently.
problem Learning optimal linear representations with LAEs.
method Proper regularization schemes (non-uniform ℓ2 and nested dropout).
result Convergence to optimal representation is slow due to ill-conditioning.
Study on autoencoder denoising in high dimensions.
problem Denoising data from Gaussian mixtures.
method Two-layer non-linear autoencoder with skip connection in high-dimensional limit.
result Closed-form expressions for denoising mean-squared test error.
The paper identifies five extreme learning regimes for large linear autoencoders.
problem Understanding the learning dynamics of large weight-tied linear autoencoders.
method Formal loss-expansion hierarchy and analysis of gradient flow.
result Five extreme regimes associated with faces of a triangular prism.
High-dimensional data in many areas such as computer vision and machine learning tasks brings in computational and analytical difficulty. Feature selection which selects a subset from observed features is a widely used approach for improving performance and effectiveness of machine learning models with high-dimensional…
Proposes LAE-EnKF for improved nonlinear data assimilation.
problem Performance of EnKF deteriorates for strongly nonlinear dynamics.
method Reformulates assimilation in a learned latent space with linear dynamics.
result LAE-EnKF yields more accurate and stable assimilation.
Enhanced autoencoders improve ROMs for PDEs by capturing essential properties.
problem Autoencoders struggle to capture essential properties for accurate ROMs.
method Introduced symmetric Convolutional AutoEncoders (CAEs) that preserve manifold properties.
result Symmetric CAEs yield more accurate latent trajectories and robust models.
Enhanced GARCH model uses autoencoder for volatility forecasting.
problem Selecting optimal realised volatility estimator for forecasting.
method Proposes an autoencoder-enhanced Realised GARCH model combining multiple realised measures.
result The model outperforms traditional linear methods in one-step-ahead rolling volatility forecasting.
Denoising autoencoders (DAEs) have proven useful for unsupervised representation learning, but a thorough theoretical understanding is still lacking of how the input noise influences learning. Here we develop theory for how noise influences learning in DAEs. By focusing on linear DAEs, we are able to derive analytic ex…
This work tackles posterior collapse in conditional and hierarchical VAEs.
problem Posterior collapse in VAEs leads to poor latent variable representations.
method Theoretical analysis of linear conditional and hierarchical VAEs, empirical validation.
result Theoretical and empirical evidence of posterior collapse causes in conditional and hierarchical VAEs.
Improved bipartite link prediction using 2-hop paths.
problem Link prediction in bipartite networks without node attributes.
method Multiply reconstructed adjacency matrix with symmetrically normalized training adjacency matrix to form 2-hop paths.
result 2-hop paths improve link prediction performance.
We propose a new IRT model that directly factors test items without factor analysis.
problem Existing multidimensional IRT methods require factorization, which is posthoc and linear.
method We use a sparsity-promoting horseshoe prior to factorize items directly within the IRT model.
result Our model performs factorization directly and consistently selects the correct number of factors.
Contrastive learning outperforms autoencoders and GANs in feature recovery and downstream tasks.
problem Theoretical understanding of contrastive learning's superiority in feature learning.
method Theoretical analysis of contrastive learning in linear representation settings.
result Contrastive learning outperforms autoencoders and GANs for feature recovery and in-domain downstream tasks.
To date, the instability of prognostic predictors in a sparse high dimensional model, which hinders their clinical adoption, has received little attention. Stable prediction is often overlooked in favour of performance. Yet, stability prevails as key when adopting models in critical areas as healthcare. Our study propo…
A framework connects VAEs to GLMs for better model initialization and performance.
problem Understanding and optimizing loss function critical points in VAEs.
method Introducing a theoretical framework based on GLM and EDFs.
result Maximum likelihood initialization improves VAE performance.
Paired autoencoders solve inverse problems using latent space projections.
problem Solving inverse problems in scientific computing.
method Paired autoencoder framework that projects data and quantity of interest into a latent space.
result Paired autoencoders generate multiple reconstruction metrics and enable latent-space refinement for accurate data fitting.
Switching linear dynamics improves model-based reinforcement learning and system identification.
problem Complex and nonlinear systems can be approximated by linear dynamical systems.
method Bayesian inference, Variational Autoencoders, Concrete relaxations.
result Improved accuracy in learning dynamics from partial and high-dimensional observations.
This paper examines critical points in deep linear autoencoders.
problem Locating critical points in non-convex surfaces for optimization.
method Investigated two sub-problems: sampling and numerical tolerance.
result Numerical methods often fail to sample all critical points and require strict numerical tolerance.
Linear VAEs explain posterior collapse in VAEs via local maxima in log marginal likelihood.
problem Posterior collapse in VAEs where variational posterior matches prior for some latent variables.
method Analysis of linear VAEs and their relation to pPCA, proving ELBO does not introduce spurious local maxima.
result Linear VAEs have identifiable global maxima corresponding to principal component directions, explaining posterior collapse.
Enhances FAVAR models with autoencoder for better economic forecasting and interpretability.
problem Limitations of linear FAVAR models in forecasting and structural analysis.
method Introduces Grouped Sparse autoencoder with time-varying parameters.
result The Grouped Sparse autoencoder produces more interpretable factors and superior forecasting performance.
The paper provides convergence guarantees for VAEs using SGD and Adam.
problem Understanding theoretical convergence guarantees for VAEs.
method Derives non-asymptotic convergence rates for VAEs trained with SGD and Adam.
result Convergence rate of \(\mathcal{O}(\log n / \sqrt{n})\) with explicit hyperparameter dependencies.
Graphon autoencoder generates graphs with arbitrary sizes using Chebyshev filters.
problem Generating graphs with arbitrary sizes and arbitrary structures.
method Induces graphons from observed graphs, uses Chebyshev filters for latent representation, and learns encoder and decoder to minimize Wasserstein distance.
result Graphon autoencoder provides a new paradigm for graph generation with good generalizability and transferability.
Deep autoencoder finds linear PDE coordinates for nonlinear equations.
problem Discovering linear coordinates for nonlinear PDEs.
method Residual network architecture for finding intrinsic coordinates.
result Deep learning autoencoder transforms nonlinear PDEs into linear ones.
RAEUFS selects features from data without labels, improving robustness to outliers.
problem Feature selection in high-dimensional data, especially in the presence of outliers.
method RAEUFS uses a deep autoencoder to learn nonlinear feature representations, improving robustness to outliers.
result RAEUFS outperforms state-of-the-art UFS methods in both clean and outlier-contaminated data settings.
CRsAE autoencoder learns dictionary for image denoising and brain spike detection.
problem Sparse dictionary learning for image and neural signal processing.
method CRsAE architecture, alternating E-step and M-step, FISTA for sparse coding, EM-inspired regularization.
result CRsAE outperforms conventional methods in image denoising and spike detection.
LOCA learns standardized data coordinates from measurements.
problem Learning invariant data coordinates from non-linearly deformed manifolds.
method LOCA, a LOcal Conformal Autoencoder, learns an isometric embedding.
result LOCA preserves geometric information while learning invariant coordinates.
Novel autoencoder method approximates Koopman operator in low dimensions.
problem Challenges in approximating finite Koopman operators using data-driven methods.
method Mori-Zwanzig autoencoder (MZ-AE) for robust Koopman operator approximation.
result Improved predictive capability and robust long-term statistical performance.
Probabilistic Autoencoder learns latent space weights' distribution.
problem Nonlinear model reconstruction error and sample quality.
method Normalizing flow for latent space weights' probability distribution.
result PAE achieves small reconstruction errors, high sample quality, and good performance.
Proposes variational autoencoder for efficient MMSE estimation.
problem Efficient parameterized MMSE estimation for noisy observations.
method Variational autoencoder models data distribution, approximates MMSE.
result Proposed estimator performs well compared to state-of-the-art.
Improved CAEs reduce training time and enhance generalization.
problem Stability issues in Concrete Autoencoders (CAEs) for feature selection.
method Indirectly Parameterized Concrete Autoencoders (IP-CAEs) learn parameters of Gumbel-Softmax distributions.
result IP-CAEs achieve significant improvements in generalization and training time.