Proposes a proportional masking strategy for better tabular data imputation.
problem Heterogeneity of tabular data disrupts uniform random masking in MAEs.
method Computes missingness statistics, generates proportional masks, uses MLP token mixing.
result Proportional masking preserves missingness distribution, improves imputation performance.
MoCA uses a novel autoencoder to analyze multi-modal health data.
problem Challenges in analyzing continuous multi-modal health data from wearable devices.
method Proposes MoCA, a self-supervised learning framework combining transformer and masked autoencoder methods.
result Demonstrates strong performance boosts across reconstruction and classification tasks.
CACTI improves tabular data imputation by leveraging missingness patterns and contextual information.
problem Tabular data imputation with improved accuracy and robustness.
method Masked autoencoding approach with median truncated copy masking and contextual information.
result Average R2 gain of 7.8% over the next best method across various datasets and missingness conditions. Unified model trained on images and videos using masked autoencoding.
problem Training a single model for multiple visual modalities.
method Masked autoencoding on a Vision Transformer.
result Unified model achieves comparable or better performance than single-modality models.
There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder's parameters to respect autoregressive constraints: ea…
CRATE-MAE learns structured representations from unlabeled data.
problem Learning structured representations from unlabeled data.
method Structured Diffusion with White-Box Transformers.
result CRATE-MAE achieves highly promising performance on large-scale imagery datasets.
VHGM-MAE generates synthetic humans from healthcare data.
problem Handling high-dimensional, sparse healthcare data with missing values.
method Masked autoencoder (MAE) tailored for healthcare data, addressing heterogeneity, missingness, and high-dimensionality.
result VHGM-MAE outperforms existing methods in missing value imputation and synthetic data generation.
We consider the effect of structure-agnostic and structure-dependent masking schemes when training a universal marginaliser (arXiv:1711.00695) in order to learn conditional distributions of the form P(xi∣xb), where xi is a given random variable and xb is some arbitrary sub…
VADD enhances discrete diffusion models by capturing inter-dimensional correlations, improving sample quality.
problem Limited modeling of inter-dimensional dependencies in MDMs degrades performance with few denoising steps.
method Introduces an auxiliary recognition model for latent variable modeling, enabling stable training via variational lower bounds maximization and amortized inference.
result VADD consistently outperforms MDM baselines in sample quality with few denoising steps.
SiamJEPA uses Siamese student encoders to improve JEPA-based representation learning.
problem Improving self-supervised representation learning in JEPA models.
method Proposes SiamJEPA with masked Siamese student encoders and EMA teacher network.
result Siamese student encoders improve representation separability and learning speed.
NeuroPaint infers missing brain area dynamics from multi-animal datasets.
problem Leveraging multi-animal datasets to understand interactions between brain areas.
method Masked autoencoding approach trained across animals with partial observations.
result Models can successfully reconstruct dynamics of unrecorded brain areas.
Reconstruction-based learning produces uninformative features for perception tasks.
problem Misalignment between reconstruction-based learning and perception tasks.
method Investigated the impact of input space reconstruction on feature learning for perception tasks.
result Reconstruction-based learning allocates model capacity to a subspace with uninformative features for perception tasks.
Proposes a new approach to time series representation learning by embedding patches independently.
problem Capturing dependencies between time series patches is not optimal for representation learning.
method 1) Patch reconstruction task, 2) Patch-wise MLP, 3) Complementary contrastive learning.
result Improves time series forecasting and classification performance compared to state-of-the-art models.
VSAE learns from missing heterogeneous data by modeling latent dependencies.
problem Learning from partially-observed heterogeneous data with missingness.
method Variational selective autoencoder (VSAE) models joint distribution of observed, unobserved, and missing data.
result VSAE improves over state-of-the-art models in data generation and imputation tasks.
VSCOUT detects anomalies in high-dimensional data using a hybrid VAE approach.
problem Challenges in classical SPC for high-dimensional, non-Gaussian data.
method Hybrid VAE architecture with ARD prior, ensemble filtering, and changepoint detection.
result VSCOUT achieves superior sensitivity to special-cause structure and controlled false alarms.
PRAE identifies outliers and reconstructs inliers in autoencoders.
problem Accurately identifying anomalies in data.
method Probabilistic Robust AutoEncoder (PRAE) approach.
result PRAE effectively removes outliers and reconstructs inliers.
daep learns from irregular, multimodal astronomical data.
problem Learning from irregular, multimodal astronomical sequences.
method Diffusion Autoencoder with Perceivers (daep) tokenizes, compresses, and reconstructs data.
result daep outperforms VAE and maep baselines in reconstruction and fine-scale structure preservation.
Theoretical analysis of vision transformers' performance with MAE and CL objectives.
problem Understanding the distinct representations learned by vision transformers with MAE and CL objectives.
method Modeling visual data distribution and analyzing ViTs training dynamics with gradient descent.
result ViTs trained with MAE objectives learn both global and local features, while CL-trained ViTs favor global features.
Extends VAEs to handle complex Bayesian network structures.
problem Handling complex dependency structures in Bayesian networks.
method Extends VAEs with graphical residual flows to model arbitrary dependency structures.
result Demonstrates improved performance on synthetic datasets.
Proposes a new model for handling missing data.
problem Nonignorable missingness in data.
method Variational autoencoder architecture with pattern-set mixtures.
result Achieves state-of-the-art imputation performance.
AEN-SAEs address feature starvation in sparse autoencoders by stabilizing the geometric alignment of sparse coding.
problem Feature starvation in sparse autoencoders, leading to unstable and misaligned representations.
method Adaptive Elastic Net SAEs (AEN-SAEs) combine ℓ2 and ℓ1 terms to stabilize the sparse coding map and control feature interactions. result AEN-SAEs mitigate feature starvation without heuristic resampling, maintaining competitive reconstruction abilities.
PMI-Masking improves MLM pretraining by masking correlated spans efficiently.
problem Uniform token masking leads to inefficient and suboptimal performance in MLMs.
method PMI-Masking uses Pointwise Mutual Information to mask n-grams with high collocation.
result PMI-Masking reaches half the training time and improves performance.
A new method selects features efficiently for high-dimensional data.
problem High computational costs and memory requirements in high-dimensional data.
method QuickSelection uses the strength of neurons in sparse autoencoders to select features.
result QuickSelection achieves the best trade-off of accuracy, speed, and memory usage.
ST-MTM models complex time series by decomposing and masking seasonal and trend components.
problem Forecasting complex time series with intricate temporal variations.
method Seasonal-Trend Decomposition with Masking and Contrastive Learning.
result ST-MTM achieves superior forecasting performance compared to existing methods.
Flexible model tackles high-dimensional, missing data, and stochastic processes.
problem High-dimensional longitudinal data with structured missingness and unknown measurement time points.
method Latent variable model using Gaussian processes and variational autoencoder.
result Competitive performance on simulated and real datasets.
Paper uses VAEAC to estimate Shapley values for complex models with mixed features.
problem Estimating Shapley values for models with dependent mixed features.
method Uses variational autoencoder with arbitrary conditioning (VAEAC) to model feature dependencies.
result VAEAC approach outperforms state-of-the-art methods for various settings.
Expands MLM by masking token positions, improving performance and convergence.
problem Improving language model performance and convergence.
method Masking token positions along with [MASK] tokens, using a fully connected classifier stage.
result Shows .3% improvement and 50% faster convergence for BERT Base with position masking.
This paper adapts PATE for semantic segmentation while maintaining privacy.
problem Preserving privacy in medical machine learning, especially for sensitive information.
method Adapting PATE for semantic segmentation using low-dimensional representations and low-sensitivity queries.
result An Autoencoder-based PATE variant achieves a higher Dice coefficient for the same privacy guarantee.
New method reduces diffusion model function evaluations for discrete data.
problem High computational burden in generating samples from masked diffusion models.
method Modified causal attention mask and speculative sampling mechanism for non-factorized predictions.
result Achieved ~2x reduction in required network forward passes.
Normalizing flows and autoregressive models have been successfully combined to produce state-of-the-art results in density estimation, via Masked Autoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-based speech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows (IAF). We unify a…
Anomaly detection refers to the task of finding unusual instances that stand out from the normal data. In several applications, these outliers or anomalous instances are of greater interest compared to the normal ones. Specifically in the case of industrial optical inspection and infrastructure asset management, findin…
New analysis reveals masked self-supervised learning's effectiveness in extracting data structure.
problem Analyzing masked self-supervised learning in high-dimensional data.
method Developed precise high-dimensional analysis of masked modeling objectives.
result Identified phase transitions and structured regimes for masked self-supervised learning.
The recently proposed mask-predict decoding algorithm has narrowed the performance gap between semi-autoregressive machine translation models and the traditional left-to-right approach. We introduce a new training method for conditional masked language models, SMART, which mimics the semi-autoregressive behavior of mas…
This paper connects masked pre-training to Bayesian model selection.
problem Understanding the success of masked pre-training and its generalization.
method The paper shows masked pre-training corresponds to maximizing the marginal likelihood.
result Masked pre-training with a suitable scoring function maximizes the marginal likelihood.
A hybrid Convolutional VAE predicts crypto volatility surfaces, outperforming single-symbol approaches.
problem Predicting crypto volatility surfaces
method Convolutional VAE with hybrid predictor
result Model achieves 0.94-1.56 vol-point RMSE across BTC and ETH markets
Optimizes pruning masks for neural networks using probabilistic fine-tuning and PAC-Bayes bounds.
problem Improving neural network performance through adaptive pruning of weights.
method Optimizes stochastic pruning masks by minimizing expected loss, considering data-adaptive regularization and feature alignment.
result Probabilistic fine-tuning leads to improved test error over baseline methods in neural networks.
We consider the problem of selecting an optimal mask for an image manifold, i.e., choosing a subset of the pixels of the image that preserves the manifold's geometric structure present in the original data. Such masking implements a form of compressive sensing through emerging imaging sensor platforms for which the pow…
Recent object detectors use four-coordinate bounding box (bbox) regression to predict object locations. Providing additional information indicating the object positions and coordinates will improve detection performance. Thus, we propose two types of masks: a bbox mask and a bounding shape (bshape) mask, to represent t…
The paper investigates the effects of invalid action masking in policy gradient algorithms.
problem Invalid actions in policy gradient algorithms can lead to suboptimal performance.
method The paper provides theoretical justification and empirical demonstrations of the importance of invalid action masking.
result Invalid action masking is crucial as the number of invalid actions increases.
Improved YOLOv5 model detects mask-wearing with enhanced accuracy.
problem Detecting mask-wearing in high traffic public places.
method Improved YOLOv5l with Multi-Head Attentional Self-Convolution, Swin Transformer Block, I-CBAM module, and enhanced feature fusion.
result 1.1% improvement in mAP(0.5) and 1.3% improvement in mAP(0.5:0.95) compared to YOLOv5l.
Simplified masked diffusion models improve discrete data generation.
problem Complex model formulations and unclear relationships hinder discrete data generative modeling.
method Developed a simple and general framework for masked diffusion models.
result Models trained on OpenWebText surpass prior diffusion language models and outperform autoregressive models.
Language models trained on chess board states outperform those on moves, even with causal masking.
problem Applying causal masking to spatial data for training unimodal language models.
method Trained bidirectional and causal self-attention models on both spatial (board-based) and sequential (move-based) chess data.
result Models trained on spatial board states achieve stronger playing strength than those trained on sequential data, even with causal masking.
Masking diffusion outperforms other discrete diffusion models by incorporating jump times into the model.
problem Improving the performance of discrete diffusion models.
method Conditioning on the jump schedule of discrete Markov processes.
result Schedule-conditioned discrete diffusion (SCUD) models outperform classical and masking diffusion models.
We develop mask iterative hard thresholding algorithms (mask IHT and mask DORE) for sparse image reconstruction of objects with known contour. The measurements follow a noisy underdetermined linear model common in the compressive sampling literature. Assuming that the contour of the object that we wish to reconstruct i…
The ConditionaL Neural Networks (CLNN) and the Masked ConditionaL Neural Networks (MCLNN) exploit the nature of multi-dimensional temporal signals. The CLNN captures the conditional temporal influence between the frames in a window and the mask in the MCLNN enforces a systematic sparseness that follows a filterbank-lik…
Unified framework for ICL in causal and masked models.
problem Understanding ICL in masked language models and comparing it to causal models.
method Developed a statistical learning framework representing context by empirical measure and predicting using context and query.
result Upper bounds for masked and autoregressive objectives under Wasserstein-type regularity conditions.
VLSI mask optimization is one of the most critical stages in manufacturability aware design, which is costly due to the complicated mask optimization and lithography simulation. Recent researches have shown prominent advantages of machine learning techniques dealing with complicated and big data problems, which bring p…
Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative…