ST-MTM models complex time series by decomposing and masking seasonal and trend components.
problem Forecasting complex time series with intricate temporal variations.
method Seasonal-Trend Decomposition with Masking and Contrastive Learning.
result ST-MTM achieves superior forecasting performance compared to existing methods.
Simplified masked diffusion models improve discrete data generation.
problem Complex model formulations and unclear relationships hinder discrete data generative modeling.
method Developed a simple and general framework for masked diffusion models.
result Models trained on OpenWebText surpass prior diffusion language models and outperform autoregressive models.
Study finds differences in LTs across tasks and architectures, proposing a consensus-based method for generating refined lottery tickets.
problem Understanding the variability and uniqueness of Lottery Tickets across different image classification tasks and architectures.
method 28 combinations of image classification tasks and architectures, iterative pruning techniques, consensus-based method for generating refined lottery tickets.
result Disproves the uniqueness of Lottery Tickets and connects emergent mask structure to the choice of pruning.
We study the problem of data release with privacy, where data is made available with privacy guarantees while keeping the usability of the data as high as possible --- this is important in health-care and other domains with sensitive data. In particular, we propose a method of masking the private data with privacy guar…
Enhances Transformer models for multivariate time series with dataset-specific channel masks.
problem Capturing channel dependency in multivariate time series models.
method Introduces partial channel dependence and channel masks to refine channel dependency in Transformers.
result Improves model performance across diverse tasks and datasets.
Modulating masks improve lifelong reinforcement learning.
problem Catastrophic forgetting and task interference in lifelong reinforcement learning.
method Adapted modulating masks for deep lifelong reinforcement learning (LRL) with PPO and IMPALA agents.
result Superior performance in both discrete and continuous RL tasks compared to LRL baselines.
Deep learning based speech enhancement and source separation systems have recently reached unprecedented levels of quality, to the point that performance is reaching a new ceiling. Most systems rely on estimating the magnitude of a target source by estimating a real-valued mask to be applied to a time-frequency represe…
A framework for using auxiliary data to improve few-shot learning.
problem Few-shot learning with scarce labeled examples and abundant auxiliary data.
method Automatic pseudo-shot selection and masking module to adjust auxiliary features.
result Masking module improves accuracy by 4.68 and 6.03 percentage points.
Recent variants improve knowledge distillation performance.
problem Improving the performance of knowledge distillation.
method Introducing additional components or changing the learning process.
result These variants have shown promising results.
PMI-Masking improves MLM pretraining by masking correlated spans efficiently.
problem Uniform token masking leads to inefficient and suboptimal performance in MLMs.
method PMI-Masking uses Pointwise Mutual Information to mask n-grams with high collocation.
result PMI-Masking reaches half the training time and improves performance.
Adaptive dropout and regularization are shown to be dual in linear networks.
problem Sparsifying deep neural networks.
method Examining dropout in the linear case, revealing a duality with regularization.
result Adaptive dropout methods lead to sparse solutions with effective penalties similar to classical sparse optimization penalties.
This work improves explanation quality for time series predictions by learning perturbations.
problem Explaining predictions on multivariate time series data with time dependencies.
method Learning both masks and associated perturbations to explain predictions.
result Learning perturbations significantly improves explanation quality on time series data.
A new method uses conformal prediction to create reliable confidence masks for image super-resolution.
problem Uncertainty quantification in image super-resolution using generative models.
method Conformal prediction techniques applied to a confidence mask for reliable uncertainty communication.
result Strong theoretical guarantees and empirical solid performance in image super-resolution.
Paper proposes a new sparse Bayesian neural network for simpler, more efficient DNNs.
problem Complex and large DNN architectures require simplification for better performance and efficiency.
method Masked Bayesian Neural Networks (BNN) with nodewise sparsity and optimal posterior distributions.
result The proposed BNN yields well-condensed DNN architectures with similar accuracy and uncertainty quantification to large DNNs.
AaSP improves audio self-supervised learning by addressing aliasing issues.
problem Alias issues in audio spectrogram transformers.
method AaSP combines aliasing-aware patch representation, teacher-student masked modeling, cross-attention predictor, and contrastive regularization.
result AaSP learns more stable representations that integrate high-frequency cues.
Method generates anatomically-controllable medical images with segmentation guidance.
problem Challenging to enforce anatomical constraints in generated medical images.
method Segmentation-guided diffusion models with random mask ablation training.
result New state-of-the-art in faithfulness to input anatomical masks.
Echocardiography (echo) is a common means of evaluating cardiac conditions. Due to the label scarcity, semi-supervised paradigms in automated echo analysis are getting traction. One of the most sought-after problems in echo is the segmentation of cardiac structures (e.g. chambers). Accordingly, we propose an echocardio…
MASnet enhances speech on mobile devices with low latency.
problem Efficiently enhancing speech on mobile devices with low latency.
method MASnet processes linear-scale spectrograms, using ratio masks to enhance noisy frames, and operates in low-latency incremental inference mode.
result MASnet achieves efficient speech enhancement with low latency, reducing FMA/s operations.
Expands MLM by masking token positions, improving performance and convergence.
problem Improving language model performance and convergence.
method Masking token positions along with [MASK] tokens, using a fully connected classifier stage.
result Shows .3% improvement and 50% faster convergence for BERT Base with position masking.
New method reduces diffusion model function evaluations for discrete data.
problem High computational burden in generating samples from masked diffusion models.
method Modified causal attention mask and speculative sampling mechanism for non-factorized predictions.
result Achieved ~2x reduction in required network forward passes.
Proposes a proportional masking strategy for better tabular data imputation.
problem Heterogeneity of tabular data disrupts uniform random masking in MAEs.
method Computes missingness statistics, generates proportional masks, uses MLP token mixing.
result Proportional masking preserves missingness distribution, improves imputation performance.
Our work connects parameter magnitudes and Hessian eigenspaces in deep neural nets.
problem Understanding the relationship between parameter magnitudes and Hessian curvature in deep learning models.
method Developed a matrix-free algorithm based on sketched SVDs to measure similarity between parameter masks and Hessian eigenspaces.
result Top Hessian eigenvectors tend to be concentrated around larger parameters, indicating a connection between parameter magnitudes and loss curvature.
New analysis reveals masked self-supervised learning's effectiveness in extracting data structure.
problem Analyzing masked self-supervised learning in high-dimensional data.
method Developed precise high-dimensional analysis of masked modeling objectives.
result Identified phase transitions and structured regimes for masked self-supervised learning.
Creating large scale high-quality annotations is a known challenge in medical imaging. In this work, based on the CycleGAN algorithm, we propose leveraging annotations from one modality to be useful in other modalities. More specifically, the proposed algorithm creates highly realistic synthetic CT images (SynCT) from …
Paper proposes a method to encrypt faces while maintaining visual similarity.
problem Protecting personal data from unauthorized face recognition.
method Targeted identity-protection iterative method (TIP-IM) to generate adversarial identity masks.
result TIP-IM provides 95%+ protection success rate against face recognition models.
The recently proposed mask-predict decoding algorithm has narrowed the performance gap between semi-autoregressive machine translation models and the traditional left-to-right approach. We introduce a new training method for conditional masked language models, SMART, which mimics the semi-autoregressive behavior of mas…
This paper connects masked pre-training to Bayesian model selection.
problem Understanding the success of masked pre-training and its generalization.
method The paper shows masked pre-training corresponds to maximizing the marginal likelihood.
result Masked pre-training with a suitable scoring function maximizes the marginal likelihood.
Optimizes pruning masks for neural networks using probabilistic fine-tuning and PAC-Bayes bounds.
problem Improving neural network performance through adaptive pruning of weights.
method Optimizes stochastic pruning masks by minimizing expected loss, considering data-adaptive regularization and feature alignment.
result Probabilistic fine-tuning leads to improved test error over baseline methods in neural networks.
We consider the problem of selecting an optimal mask for an image manifold, i.e., choosing a subset of the pixels of the image that preserves the manifold's geometric structure present in the original data. Such masking implements a form of compressive sensing through emerging imaging sensor platforms for which the pow…
Recent object detectors use four-coordinate bounding box (bbox) regression to predict object locations. Providing additional information indicating the object positions and coordinates will improve detection performance. Thus, we propose two types of masks: a bbox mask and a bounding shape (bshape) mask, to represent t…
The paper investigates the effects of invalid action masking in policy gradient algorithms.
problem Invalid actions in policy gradient algorithms can lead to suboptimal performance.
method The paper provides theoretical justification and empirical demonstrations of the importance of invalid action masking.
result Invalid action masking is crucial as the number of invalid actions increases.
Neural networks with low-precision weights and activations offer compelling efficiency advantages over their full-precision equivalents. The two most frequently discussed benefits of quantization are reduced memory consumption, and a faster forward pass when implemented with efficient bitwise operations. We propose a t…
Improved YOLOv5 model detects mask-wearing with enhanced accuracy.
problem Detecting mask-wearing in high traffic public places.
method Improved YOLOv5l with Multi-Head Attentional Self-Convolution, Swin Transformer Block, I-CBAM module, and enhanced feature fusion.
result 1.1% improvement in mAP(0.5) and 1.3% improvement in mAP(0.5:0.95) compared to YOLOv5l.
Language models trained on chess board states outperform those on moves, even with causal masking.
problem Applying causal masking to spatial data for training unimodal language models.
method Trained bidirectional and causal self-attention models on both spatial (board-based) and sequential (move-based) chess data.
result Models trained on spatial board states achieve stronger playing strength than those trained on sequential data, even with causal masking.
Masking diffusion outperforms other discrete diffusion models by incorporating jump times into the model.
problem Improving the performance of discrete diffusion models.
method Conditioning on the jump schedule of discrete Markov processes.
result Schedule-conditioned discrete diffusion (SCUD) models outperform classical and masking diffusion models.
Mixed Sample Data Augmentation (MSDA) has received increasing attention in recent years, with many successful variants such as MixUp and CutMix. By studying the mutual information between the function learned by a VAE on the original data and on the augmented data we show that MixUp distorts learned functions in a way …
We develop mask iterative hard thresholding algorithms (mask IHT and mask DORE) for sparse image reconstruction of objects with known contour. The measurements follow a noisy underdetermined linear model common in the compressive sampling literature. Assuming that the contour of the object that we wish to reconstruct i…
The ConditionaL Neural Networks (CLNN) and the Masked ConditionaL Neural Networks (MCLNN) exploit the nature of multi-dimensional temporal signals. The CLNN captures the conditional temporal influence between the frames in a window and the mask in the MCLNN enforces a systematic sparseness that follows a filterbank-lik…
Unified framework for ICL in causal and masked models.
problem Understanding ICL in masked language models and comparing it to causal models.
method Developed a statistical learning framework representing context by empirical measure and predicting using context and query.
result Upper bounds for masked and autoregressive objectives under Wasserstein-type regularity conditions.
VLSI mask optimization is one of the most critical stages in manufacturability aware design, which is costly due to the complicated mask optimization and lithography simulation. Recent researches have shown prominent advantages of machine learning techniques dealing with complicated and big data problems, which bring p…
Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative…
Novel bio-inspired masking for robust speech emotion recognition.
problem Noise degradation in speech emotion recognition.
method Cochlear cepstrogram-based contrastive learning with temporal and frequency masking.
result Improved speech emotion recognition performance on K-EmoCon benchmark.
This paper analyzes MaskGIT sampler and introduces a moment sampler for faster masked diffusion sampling.
problem Efficiently sampling from masked diffusion models.
method Theoretical analysis of MaskGIT sampler, introduction of moment sampler, and two innovations for improving choose-then-sample efficiency.
result The moment sampler is an asymptotically equivalent, more interpretable alternative to MaskGIT.
Enhanced Sampling Scheme improves masked generative modeling.
problem Limitations of existing sampling schemes in masked non-autoregressive generative modeling.
method ESS consists of three stages: Naive Iterative Decoding, Critical Reverse Sampling, and Critical Resampling.
result ESS achieves significant performance gains in unconditional and class-conditional sampling.
Structured State-Space Duality connects SSMs to masked attention.
problem Connecting SSMs and attention mechanisms for efficient modeling.
method Formalizing and generalizing SSD from scalar-identity to diagonal state matrices.
result Diagonal SSMs match training complexity lower bounds and support richer dynamics.
Study improves healthcare time series imputation by considering structured missingness.
problem Structured missingness in clinical data impacts time series imputation models.
method Analysis of different masking strategies on imputation methods using PhysioNet Challenge 2012 dataset.
result Masking choices significantly affect imputation accuracy and clinical prediction.
Training-free method improves text-to-image generation quality.
problem Error accumulation in simultaneous token updates of masked diffusion models.
method Training-free self-correction framework exploiting inductive biases.
result Significantly improved generation quality on text-to-image tasks.
MoCA uses a novel autoencoder to analyze multi-modal health data.
problem Challenges in analyzing continuous multi-modal health data from wearable devices.
method Proposes MoCA, a self-supervised learning framework combining transformer and masked autoencoder methods.
result Demonstrates strong performance boosts across reconstruction and classification tasks.