A method to automatically learn proposal distributions for energy-based regression models.
problem Manual design and initial estimate of proposal distributions for energy-based regression models.
method Introduces a method to learn an effective proposal distribution automatically, parameterized by a separate network head, and derives a unified training objective to minimize KL divergence and negative log-likelihood.
result Consistently outperforms conventional MDN training on four real-world regression tasks within computer vision.
Bidirectional bounds stabilize training of energy-based models.
problem Training energy-based models is difficult and prone to instability.
method Propose bidirectional bounds linking to gradient penalty and Jacobi-determinant estimator.
result Significant stabilization and high-quality density estimation achieved.
Improves sample quality of generative models using energy-based methods.
problem Low sample quality in generative models.
method Constructs an energy function on latent space, trains an energy-based model, and generates improved samples.
result Significant improvement in sample quality with minimal computational overhead.
New method trains energy-based models faster and more stably.
problem Training efficiency and stability of energy-based models.
method EBFlow with score-matching objectives.
result EBFlow achieves significant speedup and better performance.
A new loss function ED simplifies training energy-based models without scores.
problem Training energy-based models is computationally expensive.
method Energy Discrepancy (ED) loss function that does not rely on scores or MCMC.
result ED effectively interpolates between score matching and negative log-likelihood.
EBR improves NMT by re-ranking samples drawn from MLE-trained models.
problem Discrepancy between MLE and BLEU score in neural machine translation.
method Train an energy-based model to mimic BLEU score, then use it for re-ranking.
result EBR consistently improves NMT performance by +4 BLEU points on IWSLT'14 German-English.
We propose to reinterpret a standard discriminative classifier of p(y|x) as an energy based model for the joint distribution p(x,y). In this setting, the standard class probabilities can be easily computed as well as unnormalized values of p(x) and p(x|y). Within this framework, standard discriminative architectures ma…
RNE provides a flexible framework for diffusion models, enabling inference-time control and energy-based training.
problem Insufficient knowledge of marginal densities in diffusion models.
method Introduces Radon-Nikodym Estimator (RNE) to reveal the connection between marginal densities and transition kernels.
result RNE delivers strong results in inference-time control and energy-based diffusion training.
Persistently trained EBMs generate images and estimate complex densities.
problem Challenges in ML learning for energy-based models, especially non-convergence of MCMC.
method Introduce diffusion data, learn a joint EBM through persistent training with enhanced sampling.
result First simultaneous achievement of stability, post-training image generation, and superior out-of-distribution detection for image data.
Paper proposes CoopFlow, a two-flow generator for energy-based models.
problem Training energy-based models with Langevin flow and normalizing flow.
method CoopFlow trains an energy-based model using a normalizing flow initialization and a short-run Langevin flow revision.
result CoopFlow converges to a moment matching estimator and synthesizes realistic images.
A new method trains discrete EBMs without sampling.
problem Training EBMs on discrete spaces is hard.
method Energy Discrepancy (ED), a contrastive loss.
result ED offers theoretical guarantees for various perturbation types.
This paper studies a training method to jointly estimate an energy-based model and a flow-based model, in which the two models are iteratively updated based on a shared adversarial value function. This joint training method has the following traits. (1) The update of the energy-based model is based on noise contrastive…
iEFM trains CNF models from unnormalized densities efficiently.
problem Training generators from energy functions or unnormalized densities.
method Iterated energy-based flow matching (iEFM) with simulation-free objective.
result iEFM outperforms existing methods in probabilistic modeling.
Paper proposes f-EBM for training deep EBMs using various f-divergences.
problem Training deep EBMs with intractable partition functions.
method Introduces f-EBM framework and optimization algorithm for any f-divergence.
result f-EBM outperforms contrastive divergence and other f-divergences.
This paper proposes a more efficient training method for energy-based models.
problem The computational burden and validity trade-off in Contrastive Divergence training.
method Introducing Diffusion Contrastive Divergence (DCD) to replace Langevin dynamics with diffusion processes.
result The proposed DCDs are more computationally efficient and handle gradient terms better than Contrastive Divergence.
Efficiently samples and learns densities with symmetries using equivariant methods.
problem Efficiently sampling and learning densities with symmetries.
method Equivariant Stein Variational Gradient Descent (SVGD) and equivariant energy based models.
result Improves and scales up training of energy based models.
This paper proposes the divergence triangle as a framework for joint training of generator model, energy-based model and inference model. The divergence triangle is a compact and symmetric (anti-symmetric) objective function that seamlessly integrates variational learning, adversarial learning, wake-sleep algorithm, an…
A new method improves training of energy-based models.
problem Training energy-based models is challenging.
method Diffusive Classification (DiffCLF) objective.
result DiffCLF enables EBMs with higher fidelity and broader applicability.
This paper proposes a multi-grid method for learning energy-based generative ConvNet models of images. For each grid, we learn an energy-based probabilistic model where the energy function is defined by a bottom-up convolutional neural network (ConvNet or CNN). Learning such a model requires generating synthesized exam…
Unified model explains AT's generative ability.
problem Understanding the generative ability of AT.
method Contrastive Energy-based Models (CEM).
result Improved sample quality in supervised and unsupervised learning.
This paper proposes a method to train energy-based models using variational auto-encoders for efficient sampling.
problem Training energy-based models by maximum likelihood is challenging due to intractable partition functions and difficult sampling from the model distribution.
method The authors propose using a variational auto-encoder to initialize finite-step MCMC sampling, specifically Langevin dynamics, to train the energy-based model.
result The proposed method enables training energy-based models using maximum likelihood, generating samples comparable to GANs and EBMs.
STANLEY improves sampling for complex data models.
problem Training Energy-Based models with intractable normalizing constants.
method Anisotropic Langevin Dynamics with gradient-informed covariance.
result Geometrically uniformly ergodic Markov Chain for sampling.
A new method trains and samples from energy-based models using diffusion recovery likelihood.
problem Training and sampling high-dimensional datasets with energy-based models is challenging.
method Trains EBMs with a diffusion recovery likelihood method, maximizing conditional probabilities of data at different noise levels.
result Generates high-fidelity images with low FID and inception scores, and accurately estimates normalized data density.
EBPs model exchangeable data with flexible distributions.
problem Current energy-based models restrict set cardinality and limited distribution forms.
method Introduced Energy-Based Processes (EBPs) that extend energy models to exchangeable data with neural network parameterizations.
result EBPs can express more flexible distributions over sets without cardinality restrictions.
Unified view of contrastive and supervised learning improves model performance.
problem Improving model performance in classification, robustness, and detection.
method Hybrid discriminative-generative training of energy-based models.
result Improved performance on classification, robustness, out-of-distribution detection, and calibration.
EBMs are flexible but hard to train; this paper explains methods.
problem Training Energy-Based Models is difficult due to the unknown normalizing constant.
method Explains MCMC, SM, and NCE for training EBMs, highlighting connections.
result Provides a friendly introduction to modern EBM training methods.
Energy-based diffusion models improve molecular sampling and simulation.
problem Inconsistency between diffusion model scores and equilibrium distributions.
method Fokker-Planck regularization to enforce consistency.
result Improved consistency and efficient sampling of biomolecular systems.
GEBM combines energy function and base distribution for better generative modeling.
problem Improving generative models with better quality samples and performance.
method Alternating training between energy function and base distribution, using MCMC for sampling.
result GEBMs produce higher quality samples and better performance than GANs.
BiDVL improves EBLVMs for visual tasks by optimizing two variational distributions.
problem Training EBLVMs is challenging due to intractable distributions.
method Bi-level doubly variational learning with two tractable distributions.
result BiDVL achieves impressive image generation and reconstruction performance.
New method trains EBMs using NFs for more accurate likelihood estimation.
problem Lack of statistical accuracy in EBMs likelihood estimation.
method Uses normalizing flows (NF) to fit an NF to an EBM during training.
result Accurate gradient for EBMs at all times, leading to a fast sampler.
Dual training method for EBMs with overparametrized neural networks.
problem Training EBMs with non-convex energies is challenging.
method Derive variational principles and dual GDA algorithm for feature-learning regime.
result Dual GDA algorithm performs best with similar time scales for features and particles.
EBMs improve continual learning without external memory or regularization.
problem Improving continual learning without external memory or regularization.
method Energy-Based Models with contrastive divergence training objective.
result EBMs outperform baseline methods on various benchmarks.
MPT improves CNN and energy-based models' OOD detection and generalization.
problem Challenging out-of-distribution detection in computer vision.
method Applying Maximum Probability Theorem as a regularization scheme in CNN and energy-based models.
result MPT-based regularization strategy stabilizes and improves generalization and robustness of base models.
New method improves sampling from noisy energy models.
problem Training and sampling challenges in Energy-Based Models.
method Pseudo-Gibbs sampling with moment matching.
result Effective sampling from clean model using noisy DSM-trained model.
EBM model for functional data using path measure tilting.
problem Modeling distributions of functions from irregularly sampled data.
method Spectral decomposition of an energy-based model with a Gaussian Process path measure to reweight the distribution.
result Effective approach to up-scaling functional data.
Autoregressive models struggle with hard-to-compute distributions, alternatives like energy-based and latent-variable models solve this.
problem Autoregressive models struggle with distributions whose next-symbol probability is hard to compute.
method Alternatives include energy-based models and latent-variable autoregressive models.
result Alternatives to autoregressive models can escape limitations of hard-to-compute distributions.
This paper introduces a new neural ODE model for continuous-time sequence generation.
problem Representing and predicting continuous-time sequences with high accuracy.
method A neural emission model and neural ODE define the latent state evolution, with an Energy-based model for prior distribution.
result The model outperforms existing methods in various tasks, including long-horizon predictions.
EBMs trained on discrete data using heat equations on graph structures.
problem Training EBMs on discrete or mixed data.
method Heat equations on graph structures for data perturbation.
result Efficacy demonstrated in various applications.
Improved generative models using overparametrized shallow neural networks.
problem Improving generative models for data with hidden low-dimensional structure.
method Using energy-based models with overparametrized shallow neural networks as approximators.
result Models trained in the 'active' regime outperform those in the 'lazy' or kernel regime, leading to better adaptivity to hidden structure.
Improves GANs by sampling from an energy-based model induced by discriminator scores.
problem Improving the quality of images generated by GANs.
method DDLS (Discriminator Driven Latent Sampling) using the sum of latent prior log-density and discriminator output score.
result Significantly improves Inception Score on CIFAR-10 dataset.
Improved language models by incorporating statistical discriminators.
problem Distinguishing model-generated text from real text reliably.
method Energy-Based Model framework to incorporate discriminators.
result Improves language model performance in perplexity and human evaluation.
VAV method optimizes learning rate for faster, stable SGD convergence.
problem Optimizing learning rate for efficient and stable machine learning models.
method Energy-based self-adaptive learning rate with auxiliary variable r. result VAV method achieves faster convergence and superior stability with larger learning rates.
This review compares various deep generative models.
problem Training deep neural networks to model data distributions.
method Comprehensive comparison of VAEs, GANs, flows, energy models, and autoregressives.
result Trade-offs and interrelationships among different models.
This paper tackles sampling issues in latent space EBMs by introducing diffusion-based amortization.
problem Degenerate MCMC sampling quality hinders latent space EBM learning and generation quality.
method Introduces diffusion-based amortization for long-run MCMC sampling.
result The learned amortization of MCMC is a valid long-run MCMC sampler.
New method improves EBMs for regression tasks.
problem Training EBMs for regression is challenging.
method Proposed a simple yet effective extension of noise contrastive estimation.
result Our method achieves state-of-the-art performance on 1D regression and object detection.
EMIX minimizes surprise in multi-agent reinforcement learning.
problem Surprise and approximation bias in multi-agent reinforcement learning.
method Energy-based MIXer (EMIX) for minimizing surprise across multiple agents.
result EMIX demonstrates consistent stable performance in challenging StarCraft II scenarios.
Proposes CDRL to improve EBM training and generation quality.
problem Challenges in training energy-based models on high-dimensional data.
method Cooperative diffusion recovery likelihood (CDRL) approach.
result Significantly boosts EBM generation performance on CIFAR-10 and ImageNet.
A deep learning method solves nonlinear filtering problems efficiently.
problem Nonlinear filtering problem
method Deep splitting method combined with energy-based neural network approximation
result Computational efficiency and performance comparable to Kalman and bootstrap filters