Proposes a Bayesian Autoencoder with sparse Gaussian process priors to capture data correlations.
problem Autoencoders' i.i.d. assumption of latent representations fails to capture data correlations.
method Imposes fully Bayesian sparse Gaussian Process priors on the latent space of a Bayesian Autoencoder and uses stochastic gradient Hamiltonian Monte Carlo for posterior estimation.
result Consistently outperforms alternatives relying on Variational Autoencoders on various tasks.
Improved SINDy autoencoder for identifying noisy dynamical systems.
problem Robust identification of noisy dynamical systems from data.
method Incorporates noise-separating neural network structures into SINDy autoencoder architecture.
result Accurately recovers latent dynamics and estimates measurement noise from noisy observations.
Enhances FAVAR models with autoencoder for better economic forecasting and interpretability.
problem Limitations of linear FAVAR models in forecasting and structural analysis.
method Introduces Grouped Sparse autoencoder with time-varying parameters.
result The Grouped Sparse autoencoder produces more interpretable factors and superior forecasting performance.
AEN-SAEs address feature starvation in sparse autoencoders by stabilizing the geometric alignment of sparse coding.
problem Feature starvation in sparse autoencoders, leading to unstable and misaligned representations.
method Adaptive Elastic Net SAEs (AEN-SAEs) combine ℓ2 and ℓ1 terms to stabilize the sparse coding map and control feature interactions. result AEN-SAEs mitigate feature starvation without heuristic resampling, maintaining competitive reconstruction abilities.
SAE-FiRE extracts key financial info from long documents, improving earnings surprise predictions.
problem Predicting earnings surprises from long, redundant financial documents.
method Sparse Autoencoder feature selection to filter out noise and identify key dimensions.
result SAE-FiRE significantly outperforms baseline approaches in financial datasets.
Binary autoencoder with sparse hidden layer preserves information and zero reconstruction error.
problem Preserving information and zero reconstruction error in binary neural networks.
method Binary autoencoder with random binary weights, sparse hidden layer, and varying neuron thresholds.
result Zero reconstruction error for any input with a large hidden layer and varying neuron thresholds.
Autoencoders fail to capture sparse structure in 1-bit data compression.
problem Proving the performance of shallow autoencoders on sparse data compression.
method Gradient descent analysis and approximate message passing.
result Gradient descent minimizer for sparse data is the identity (up to permutation) above critical sparsity.
Develops SGP-VAE for efficient sparse GP inference in multi-dimensional datasets.
problem Sparse GP approximations and missing data in multi-dimensional spatio-temporal datasets.
method Leverages partial inference networks for sparse GP approximations and amortized variational inference.
result Outperforms multi-output GPs and structured VAEs in various experiments.
This paper introduces structure learning for autoencoder recommenders to improve performance and generalization.
problem Efficient training and generalization in sparse collaborative filtering data.
method Learn groups of related items and use this information to determine the connectivity structure of an auto-encoding neural network.
result The proposed structure learning method results in a sparse network that converges to a local optimum with smaller spectral norm and generalization error.
Regularization improves stability and consistency of sparse autoencoders.
problem Varying features across random seeds and training choices in SAEs.
method Added L1 or L2 penalties on encoder and decoder weights.
result L2 regularization increases cross-seed feature consistency.
We present a transductive deep learning-based formulation for the sparse representation-based classification (SRC) method. The proposed network consists of a convolutional autoencoder along with a fully-connected layer. The role of the autoencoder network is to learn robust deep features for classification. On the othe…
A new method selects features efficiently for high-dimensional data.
problem High computational costs and memory requirements in high-dimensional data.
method QuickSelection uses the strength of neurons in sparse autoencoders to select features.
result QuickSelection achieves the best trade-off of accuracy, speed, and memory usage.
We demonstrate a new deep learning autoencoder network, trained by a nonnegativity constraint algorithm (NCAE), that learns features which show part-based representation of data. The learning algorithm is based on constraining negative weights. The performance of the algorithm is assessed based on decomposing data into…
New GP-VAE model improves scalability and performance.
problem Inability of conventional VAEs to model correlations between data points.
method Principled sparse inference approaches to improve scalability of GP-VAEs.
result New model outperforms existing approaches in runtime and memory usage.
Increasing volume of Electronic Health Records (EHR) in recent years provides great opportunities for data scientists to collaborate on different aspects of healthcare research by applying advanced analytics to these EHR clinical data. A key requirement however is obtaining meaningful insights from high dimensional, sp…
Increasing volume of Electronic Health Records (EHR) in recent years provides great opportunities for data scientists to collaborate on different aspects of healthcare research by applying advanced analytics to these EHR clinical data. A key requirement however is obtaining meaningful insights from high dimensional, sp…
Feature extraction becomes increasingly important as data grows high dimensional. Autoencoder as a neural network based feature extraction method achieves great success in generating abstract features of high dimensional data. However, it fails to consider the relationships of data samples which may affect experimental…
BatchTopK SAEs improve GPT-2 and Gemma activations with adjustable sparsity.
problem Interpreting language model activations using sparse autoencoders.
method Adapting the TopK constraint to the batch-level, allowing variable number of active latents per sample.
result BatchTopK SAEs consistently outperform TopK SAEs in reconstructing GPT-2 and Gemma activations.
New neural operators learn structured patterns efficiently.
problem Learning and representing complex, structured patterns in data.
method Sparse autoencoder neural operators (SAE-NOs) parameterize concepts as functions, enabling efficient and structured representation.
result SAE-FNOs learn localized patterns and generalize across different scales and discretizations.
Bayesian autoencoders discover physics from noisy data.
problem Challenges in identifying governing equations and coordinates from noisy, low-data real-world data.
method Bayesian SINDy autoencoders with hierarchical Bayesian sparsifying prior and adaptive empirical Bayesian method.
result Better physics discovery with lower data and fewer training epochs, along with valid uncertainty quantification.
Evidential Softmax preserves multimodality in sparse probability distributions for generative models.
problem Sparse probability distributions in deep generative models make exact marginalization computationally intractable.
method Introduce ev-softmax, a sparse normalization function that preserves multimodality and can be trained with probabilistic loss functions.
result ev-softmax outperforms existing techniques in distributional accuracy and dimensionality reduction.
This work combines deep learning and sparse coding for CT image reconstruction.
problem Improving image quality in low-dose CT scans.
method Sparse signal representation using learned dictionaries, inspired by variational autoencoders and deep learning techniques.
result Regularization with learned dictionaries achieves competitive performance in CT reconstruction.
In "Dictionary Learning" one tries to recover incoherent matrices A∗∈Rn×h (typically overcomplete and whose columns are assumed to be normalized) and sparse vectors x∗∈Rh with a small support of size hp for some 0<p<1 while having access to observations $y \in \mathbb{…
Metalearning optimizes autoencoder dimensions for efficient data representation.
problem Selecting optimal dimension for autoencoder output to balance accuracy and complexity.
method Metalearning approach using actor-critic algorithm to dynamically adjust dimension.
result Automatic selection of minimum number of bases for optimal reconstruction.
FF layers in transformers are nearly as interpretable as sparse autoencoders.
problem Comparing interpretability of feature vectors in FF layers vs. sparse autoencoders.
method Revisited interpretability of FF layers as key-value memories using modern benchmarks.
result FF and SAE feature vectors are similarly interpretable, but FFs can be better in some aspects.
We provide a series of results for unsupervised learning with autoencoders. Specifically, we study shallow two-layer autoencoder architectures with shared weights. We focus on three generative models for data that are common in statistical machine learning: (i) the mixture-of-gaussians model, (ii) the sparse coding mod…
New SAE algorithm proves feature recovery for LLMs with theoretical guarantees.
problem Achieving interpretable features in large language models (LLMs).
method Proposed a statistical framework and bias adaptation technique for sparse autoencoders (SAEs).
result Proved correct recovery of all monosemantic features under specific data sampling.
SAMS-VAE models cellular perturbations using sparse additive mechanisms.
problem Modeling effects of diverse interventions on cells.
method Sparse Additive Mechanism Shift Variational Autoencoder (SAMS-VAE).
result SAMS-VAE identifies disentangled, perturbation-specific latent subspaces.
Autoencoders have been successful in learning meaningful representations from image datasets. However, their performance on text datasets has not been widely studied. Traditional autoencoders tend to learn possibly trivial representations of text documents due to their confounding properties such as high-dimensionality…
Superposition in autoencoders leads to loss in simple models.
problem Mechanistic interpretability of neural networks
method Analyzing mathematical basis for superposition and providing bounds for reconstruction loss
result Upper and lower bounds for L2 reconstruction loss in the sparse regime
Generates high-quality images using sparse DCT representations.
problem Challenges in generating images due to high dimensionality.
method Transformers trained on sparse DCT block sequences.
result Competitive image generation quality with state-of-the-art methods.
New model reconstructs flow from sparse data with uncertainty quantification.
problem Reconstructing nonlinear flow from limited observations.
method Semi-Conditional Variational Autoencoder (SCVAE) for probabilistic flow reconstruction.
result SCVAE improves reconstruction accuracy compared to Gappy Proper Orthogonal Decomposition (GPOD).
Novel deep learning method predicts reaction coordinates and future MD trajectories.
problem Identifying optimal reaction coordinates for chemical reactions.
method Regularized Sparse Autoencoder (RSE) for discovering reaction coordinates and predicting MD trajectory evolution.
result RSE helps in choosing a small but important set of reaction coordinates.
New method combines domain changes and sparse mixing for better latent variable learning.
problem Challenges in identifying latent variables due to insufficient domain changes and violated sparsity constraints.
method Combines sufficient changes and sparse mixing constraints, using domain encoding networks and variational autoencoders.
result Identifiability of latent variables achieved with less restrictive constraints.
Paper hypothesizes MLP layers in LLMs can be approximated by sparse Mixture of Experts.
problem Understanding dense MLP layers in LLMs.
method Theoretical connection between MoE models and SAE structure in activation space.
result MLP layers in LLMs can be well approximated by sparse Mixture of Experts.
SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.
problem Sparse autoencoders' reconstruction error varies across layers, not fitting existing scaling laws.
method Cross-layer study of 844 SAE checkpoints, fitting and regressing on manifold geometry.
result Manifold geometry predicts layer-dependent width exponents in SAEs, with transferable coefficients.
DSAEE FS selects features for imbalanced data.
problem Feature selection for imbalanced data.
method DSAEE ensemble for feature ranking based on reconstruction error.
result DSAEE FS outperforms other methods in selecting relevant features.
In this paper, we apply a mini-batch based negative sampling method to efficiently train a latent factor autoencoder model on large scale and sparse data for implicit feedback collaborative filtering. We compare our work against a state-of-the-art baseline model on different experimental datasets and show that this met…
Machine learning factors outperform traditional portfolio optimization methods.
problem Comparing machine learning and traditional portfolio optimization methods.
method Examined machine learning and factor-based portfolio optimization using autoencoder neural networks and dimensionality reduction techniques.
result Minimum-variance portfolios using latent factors derived from autoencoders and sparse methods outperform simpler benchmarks in risk minimization.
In this paper we propose Structuring AutoEncoders (SAE). SAEs are neural networks which learn a low dimensional representation of data which are additionally enriched with a desired structure in this low dimensional space. While traditional Autoencoders have proven to structure data naturally they fail to discover sema…
Algorithm learns latent variables for thermodynamically-consistent deep neural networks.
problem Predicting time evolution of large-scale physical systems with thermodynamic consistency.
method Sparse autoencoders and structure-preserving neural networks.
result Method conserves total energy and entropy inequality for both conservative and dissipative systems.
Combining simple elements from the literature, we define a linear model that is geared toward sparse data, in particular implicit feedback data for recommender systems. We show that its training objective has a closed-form solution, and discuss the resulting conceptual insights. Surprisingly, this simple model achieves…
A new method for the unsupervised learning of sparse representations using autoencoders is proposed and implemented by ordering the output of the hidden units by their activation value and progressively reconstructing the input in this order. This can be done efficiently in parallel with the use of cumulative sums and …
New method for disentangling latent factors with sparse dependencies.
problem Disentangling latent factors from observed variables and past factors.
method Mechanism sparsity regularization and sparse causal graphical model.
result Identifiability of latent factors up to a sparse causal graph.
SGD batch size affects autoencoder global minima sparsity and sharpness.
problem Investigating how batch size impacts autoencoder learning.
method Non-convex autoencoder training with SGD, varying batch sizes.
result SGD batch size influences global minimum sparsity and sharpness.
Compressed sensing techniques enable efficient acquisition and recovery of sparse, high-dimensional data signals via low-dimensional projections. In this work, we propose Uncertainty Autoencoders, a learning framework for unsupervised representation learning inspired by compressed sensing. We treat the low-dimensional …
Unified framework for unsupervised concept extraction simplifies guarantees.
problem Establishing guarantees for unsupervised concept extraction.
method Unified theoretical framework, meta-theorem for identifiability.
result Simplifies the task of proving guarantees for concept extraction.
Many applications, such as text modelling, high-throughput sequencing, and recommender systems, require analysing sparse, high-dimensional, and overdispersed discrete (count-valued or binary) data. Although probabilistic matrix factorisation and linear/nonlinear latent factor models have enjoyed great success in modell…