Masking diffusion outperforms other discrete diffusion models by incorporating jump times into the model.
problem Improving the performance of discrete diffusion models.
method Conditioning on the jump schedule of discrete Markov processes.
result Schedule-conditioned discrete diffusion (SCUD) models outperform classical and masking diffusion models.
New method reduces diffusion model function evaluations for discrete data.
problem High computational burden in generating samples from masked diffusion models.
method Modified causal attention mask and speculative sampling mechanism for non-factorized predictions.
result Achieved ~2x reduction in required network forward passes.
Simplified masked diffusion models improve discrete data generation.
problem Complex model formulations and unclear relationships hinder discrete data generative modeling.
method Developed a simple and general framework for masked diffusion models.
result Models trained on OpenWebText surpass prior diffusion language models and outperform autoregressive models.
Remasking improves the quality of discrete diffusion models for natural language and image generation.
problem Limited iterative refinement in masked discrete diffusion models.
method Introducing ReMDM sampler that allows remasking during inference.
result Remasking enables better quality outputs with increased sampling steps.
This paper analyzes MaskGIT sampler and introduces a moment sampler for faster masked diffusion sampling.
problem Efficiently sampling from masked diffusion models.
method Theoretical analysis of MaskGIT sampler, introduction of moment sampler, and two innovations for improving choose-then-sample efficiency.
result The moment sampler is an asymptotically equivalent, more interpretable alternative to MaskGIT.
SJDs unify masked, continuous, and hybrid diffusion models.
problem Unified modeling of diffusion processes.
method Continuous-time Markov processes with token embeddings and hazard rates.
result Unified model recovers masked, continuous, and hybrid diffusion as limits.
Training-free method improves text-to-image generation quality.
problem Error accumulation in simultaneous token updates of masked diffusion models.
method Training-free self-correction framework exploiting inductive biases.
result Significantly improved generation quality on text-to-image tasks.
New reweighted losses improve diffusion model training and image quality.
problem Training and improving diffusion models for image generation.
method Constructing a cascade of time-dependent variational lower bounds.
result Significant improvements in pixel-space image modeling quality.
Cosine schedule is optimal for discrete diffusion models.
problem Choosing the best discretization schedule for diffusion models.
method Optimized using Fisher-Rao geometry.
result Cosine schedule is Fisher-Rao optimal.
The paper analyzes CFG in masked diffusion models and proposes a new method to improve sample quality.
problem Improving sample quality in conditional generation with CFG in masked diffusion models.
method Analyzing CFG's effect in a low-dimensional masked diffusion model, revealing early-stage guidance harms quality and late-stage improves it. Proposed a novel method to smooth transitions between distributions.
result The proposed method improves sample quality via a simple one-line code change.
Study error bounds and optimal schedules for Masked Diffusions with factorized approximations.
problem Analyzing trade-offs between computation and accuracy in Masked Diffusion Models.
method Provided general error bounds and identified optimal schedules based on data distribution information profiles.
result Identified optimal schedule sizes for Masked Diffusion Models.
DOS improves language model generation by considering inter-token dependencies.
problem Lack of sequence-level information and inter-token dependencies in existing decoding strategies.
method Dependency-Oriented Sampler (DOS) that uses attention matrices to approximate inter-token dependencies.
result DOS consistently achieves superior performance on code generation and mathematical reasoning tasks.
CADD improves generative quality by augmenting discrete diffusion with continuous latent space.
problem Loss of semantic information between denoising steps in discrete diffusion models.
method Introduces a framework that augments discrete state space with a continuous latent space, allowing for graded, informative masked tokens.
result CADD improves generative quality across text generation, image synthesis, and code modeling.
UDM reparameterization improves language model generation.
problem Mismatch between UDM training objective and denoising posterior.
method Leave-one-out denoising and absorbing state reformulation.
result Improved UDM generation through leave-one-out parameterization.
This work improves the Euler method for masked diffusion models, providing tighter convergence guarantees.
problem Improving the convergence rates of masked diffusion models.
method Developed a direct total-variation (TV) based analysis for the Euler method, relaxing assumptions and improving parameter dependencies.
result Established convergence guarantees for the Euler sampler without requiring surrogate initialization, and provided a tight lower bound.
Method generates anatomically-controllable medical images with segmentation guidance.
problem Challenging to enforce anatomical constraints in generated medical images.
method Segmentation-guided diffusion models with random mask ablation training.
result New state-of-the-art in faithfulness to input anatomical masks.
Improves generative models by optimizing rewards and sample editing.
problem Efficiently generating high-reward samples with structural constraints.
method Introduces MDM-VGB, a discrete diffusion sampler that augments unmasking generation with reward-guided remasking.
result MDM-VGB achieves quadratic complexity and robustness to noise, outperforming heuristics like best-of-N. New study on guidance in masked diffusion models, showing how it shapes sampling dynamics.
problem Understanding how guidance influences the sampling behavior of masked diffusion models.
method Derived explicit solution to guided reverse dynamics, analyzing effects in 1D and 2D.
result Guidance amplifies class-specific regions and suppresses shared regions, affecting covariance structures.
TabSODA improves imputation of surveys with skips and ordinal data.
problem Handling structural skips and ordinal responses in survey data.
method TabSODA uses an Elucidated Diffusion Model with skip pattern detection and ordinal awareness.
result TabSODA reduces ordinal missing-at-random (MACE) by up to 23.7% and improves categorical accuracy by up to 9%.
Theoretical justification for image inpainting using diffusion models.
problem Improving sample recovery in image inpainting without retraining.
method Analysis of RePaint algorithm and proposing RePaint+ to correct misalignment. result RePaint+ algorithm provably recovers the true sample with linear convergence. DTM improves dLLM fine-tuning stability and performance.
problem Intractable sequence-level marginal likelihoods for masked diffusion models.
method Discrete Tilt Matching (DTM) recasts dLLM fine-tuning as state-level matching of local unmasking posteriors under reward tilting.
result DTM yields strong gains on Sudoku and Countdown while remaining competitive on MATH500 and GSM8K.
This research compares masked diffusion models to autoregressive language models, focusing on architectural differences.
problem Comparing masked diffusion models to autoregressive language models due to architectural differences.
method Equitably compare MDMs within a decoder-only framework, investigating architectural influences.
result Decoder-only MDMs can achieve significant speedups and comparable perplexity with temperature annealing.
CRATE-MAE learns structured representations from unlabeled data.
problem Learning structured representations from unlabeled data.
method Structured Diffusion with White-Box Transformers.
result CRATE-MAE achieves highly promising performance on large-scale imagery datasets.
VADD enhances discrete diffusion models by capturing inter-dimensional correlations, improving sample quality.
problem Limited modeling of inter-dimensional dependencies in MDMs degrades performance with few denoising steps.
method Introduces an auxiliary recognition model for latent variable modeling, enabling stable training via variational lower bounds maximization and amortized inference.
result VADD consistently outperforms MDM baselines in sample quality with few denoising steps.
RaMViD uses diffusion models for video prediction and infilling.
problem Predicting and infilling missing information in videos.
method Extends image diffusion models to videos using 3D convolutions and a new conditioning technique.
result Achieves state-of-the-art results on video prediction benchmarks.
MDMs train to decode tokens in a random order, which affects performance; we show they can be optimized for a favorable order.
problem Performance of MDMs is affected by the random order in which tokens are decoded.
method We show that MDMs can be optimized for a favorable decoding order by equipping their continuous-time variational objective with multivariate noise schedules.
result MDMs can be decomposed into a weighted auto-regressive losses over orders, making them auto-regressive models with learnable orders.
Theoretical analysis shows MDMs can be efficient but not for all metrics.
problem Understanding the efficiency-accuracy trade-off of diffusion language models.
method Theoretical analysis of Masked Diffusion Model (MDM) using perplexity and sequence error rate as metrics.
result MDM achieves near-optimal perplexity but requires linear scaling for sequence error rate, highlighting efficiency-accuracy trade-offs.
Unified framework for convergence of discrete diffusion models without state space size dependence.
problem Fundamental limitations in existing convergence theory for discrete diffusion models, especially under singular priors and large vocabularies.
method Unified adjoint-equation-based framework that establishes dimension-free convergence guarantees in any integral probability metric (IPM).
result First dimension-free convergence bounds applicable to both masked and uniform priors, free of state space size S. The paper analyzes sampling efficiency of discrete diffusion models, providing sharp and adaptive guarantees.
problem Theoretical foundations of discrete diffusion models, especially sampling efficiency.
method Continuous-time Markov chain (CTMC) formulation, τ-leaping-based samplers, effective total correlation. result The τ-leaping algorithm achieves an iteration complexity of order ildeO(d/ε) for uniform discrete diffusion, improving existing bounds by a factor of d. Study provides convergence guarantees for discrete diffusion models on finite and infinite state spaces.
problem Challenges in understanding discrete diffusion models on combinatorial state spaces.
method Established convergence bounds for three discrete diffusion models using Euler approximations.
result Optimal non-asymptotic convergence guarantees for discrete diffusion models without boundedness assumptions.
LADD models improve discrete diffusion for faster language generation.
problem Practical discrete diffusion models ignore cross-token dependencies, degrading performance.
method Introduces a learnable auxiliary latent channel, diffusing over the joint (token, latent) space.
result LADD models yield improvements on unconditional generation metrics.
A new approach improves numerical tabular data imputation by addressing diffusion models' limitations.
problem Inaccurate and difficult training in numerical tabular data imputation.
method Kernelized Negative Entropy-regularized Wasserstein gradient flow Imputation (KnewImp) based on Wasserstein gradient flow (WGF) framework.
result KnewImp significantly outperforms existing methods in numerical tabular data imputation.
Paper uses diffusion models to design electric aircraft quickly.
problem Designing electric aircraft efficiently and accurately.
method Simulation-based inference with hierarchical diffusion models.
result Rediscovers known aircraft design trends and laws.
DDPD separates generation into planning and denoising for improved efficiency.
problem Efficiently denoise corrupted data during generation.
method Separates generation into a planner and denoiser, selecting denoising positions based on corruption severity.
result DDPD outperforms traditional methods on language and image generation benchmarks.
Global watermark for diffusion language models decouples detection from local contexts.
problem Watermarking in diffusion language models is challenging due to joint sampling of distributions over many unresolved positions.
method Proposes a global vector-valued sketch representation to control watermarking in masked diffusion language models.
result The method decouples detection from local contexts, resulting in an order-agnostic statistic and robustness.
Researchers improve imputation of missing data using diffusion transformers, quantifying uncertainty.
problem Enhancing the quality of time-series data with missing values.
method Conditional diffusion transformers for imputation, with statistical sample complexity bounds and uncertainty quantification.
result Theoretical insights into the efficiency and accuracy of imputation, influenced by missing patterns.
Unified diffusion model tackles missing data in tabular datasets.
problem Training diffusion models on tabular data with missing values.
method Proposes a masked regression loss for Denoising Score Matching to learn from incomplete data.
result Demonstrates superior performance on tabular datasets with missing values compared to state-of-the-art methods.
ARDMs are a new model class for autoregressive diffusion that generalize existing models and can compress data efficiently.
problem Efficient data compression and generation.
method Autoregressive Diffusion Models (ARDMs) that generalize existing autoregressive models and discrete diffusion models.
result ARDMs require significantly fewer steps for compression compared to discrete diffusion models.
AugMask trains diffusion models on incomplete tabular data by augmenting missing values and applying denoising supervision.
problem Training diffusion models on incomplete tabular data with missing values.
method AugMask uses stochastic augmentation and denoising supervision to adapt diffusion models to incomplete data.
result AugMask enables diffusion-based tabular generators to outperform specialized missing-aware baselines across various datasets and missingness regimes.
Improved Markov models learn from their mistakes and adapt to problem complexity.
problem Limitations of standard masked discrete diffusion models in reasoning tasks.
method Learning a Markov transition kernel trained on its own outputs, allowing remasking and adaptation.
result Significant improvement in solving reasoning problems, especially Sudoku-Extreme and Countdown-4.
This primer explains diffusion models in general state spaces.
problem Diffusion models in general state spaces are not well-introduced.
method Develops discrete-time and continuous-time views of diffusion models, deriving Fokker-Planck and master equations.
result Unified understanding of diffusion models across continuous and discrete domains.
Variational approach improves diffusion models for solving inverse problems.
problem Challenges in solving inverse problems with diffusion models due to the nonlinear and iterative nature of the diffusion process.
method Proposes a variational approach to approximate the posterior distribution, leading to regularization by denoising diffusion process (RED-Diff).
result Demonstrates improved performance in image restoration tasks compared to state-of-the-art sampling-based diffusion models.
Proposes using diffusion models for probabilistic stock market predictions.
problem Uncertainties in financial data make deterministic models ineffective for stock market predictions.
method Utilizes Denoising Diffusion Probabilistic Models (DDPM) and Masked Relational Transformer (MRT).
result Achieves state-of-the-art performance in stock movement prediction and portfolio management.
With the increasing use of online social networks as a source of news and information, the propensity for a rumor to disseminate widely and quickly poses a great concern, especially in disaster situations where users do not have enough time to fact-check posts before making the informed decision to react to a post that…
PMI-Masking improves MLM pretraining by masking correlated spans efficiently.
problem Uniform token masking leads to inefficient and suboptimal performance in MLMs.
method PMI-Masking uses Pointwise Mutual Information to mask n-grams with high collocation.
result PMI-Masking reaches half the training time and improves performance.
daep learns from irregular, multimodal astronomical data.
problem Learning from irregular, multimodal astronomical sequences.
method Diffusion Autoencoder with Perceivers (daep) tokenizes, compresses, and reconstructs data.
result daep outperforms VAE and maep baselines in reconstruction and fine-scale structure preservation.
CANDI solves the gap between continuous and discrete diffusion models for text generation.
problem Underperformance of continuous diffusion models in discrete data domains.
method Introduces token identifiability and a hybrid framework (CANDI) to decouple discrete and continuous corruption.
result CANDI successfully avoids temporal dissonance, enabling continuous diffusion benefits for discrete spaces.
New method uses diffusion models to generate proteins with specific motifs.
problem Generating proteins with specific functional substructures (motifs) using diffusion models.
method Adapting SMC-aided diffusion posterior samplers to zero-shot scaffolding tasks.
result Proposed potentials and samplers improve performance in generating proteins with desired motifs.