A new method learns hierarchical EBM models with diffusion schemes.
problem Challenges in learning EBM models with multi-modal distributions.
method Proposes a diffusion probabilistic scheme to learn EBM models in hierarchical latent spaces.
result Demonstrates superior performance on various tasks with diffusion-learned EBM.
Proposes CDRL to improve EBM training and generation quality.
problem Challenges in training energy-based models on high-dimensional data.
method Cooperative diffusion recovery likelihood (CDRL) approach.
result Significantly boosts EBM generation performance on CIFAR-10 and ImageNet.
Proposes a method to use generators as EBM foundations without latent inference.
problem Training EBMs from generator outputs without latent variables.
method Formulates a Hat EBM using generator outputs and residual variables.
result Strong performance on various generator tasks.
Paper proposes f-EBM for training deep EBMs using various f-divergences.
problem Training deep EBMs with intractable partition functions.
method Introduces f-EBM framework and optimization algorithm for any f-divergence.
result f-EBM outperforms contrastive divergence and other f-divergences.
EBMs improve continual learning without external memory or regularization.
problem Improving continual learning without external memory or regularization.
method Energy-Based Models with contrastive divergence training objective.
result EBMs outperform baseline methods on various benchmarks.
Paper proposes a method to improve MCMC sampling for energy-based models.
problem MCMC sampling of energy-based models is often not mixing in high-dimensional data.
method Proposes using a flow-based model as a backbone to correct the energy-based model, enabling mixing in latent space.
result MCMC sampling of the corrected EBM in the latent space mixes well and traverses modes in the data space.
Model-based planning holds great promise for improving both sample efficiency and generalization in reinforcement learning (RL). We show that energy-based models (EBMs) are a promising class of models to use for model-based planning. EBMs naturally support inference of intermediate states given start and goal state dis…
New method improves EBMs for regression tasks.
problem Training EBMs for regression is challenging.
method Proposed a simple yet effective extension of noise contrastive estimation.
result Our method achieves state-of-the-art performance on 1D regression and object detection.
New method trains EBMs using NFs for more accurate likelihood estimation.
problem Lack of statistical accuracy in EBMs likelihood estimation.
method Uses normalizing flows (NF) to fit an NF to an EBM during training.
result Accurate gradient for EBMs at all times, leading to a fast sampler.
Persistently trained EBMs generate images and estimate complex densities.
problem Challenges in ML learning for energy-based models, especially non-convergence of MCMC.
method Introduce diffusion data, learn a joint EBM through persistent training with enhanced sampling.
result First simultaneous achievement of stability, post-training image generation, and superior out-of-distribution detection for image data.
EBMs are flexible but hard to train; this paper explains methods.
problem Training Energy-Based Models is difficult due to the unknown normalizing constant.
method Explains MCMC, SM, and NCE for training EBMs, highlighting connections.
result Provides a friendly introduction to modern EBM training methods.
A new method to learn EBM in latent space for better data modeling.
problem Efficiently modeling data with complex structures.
method Joint learning of latent space EBM and top-down network using maximum likelihood and MCMC sampling.
result Simple EBM in latent space captures data regularities effectively and performs well in generation and anomaly detection.
VAEBM combines VAEs and EBMs for efficient image generation.
problem Sampling from EBMs is slow and VAEs generate blurry images.
method VAEBM uses a VAE for quick sampling and an EBM to refine and exclude non-data regions.
result VAEBM generates high-quality images with short MCMC chains.
This paper proposes a method to train energy-based models using variational auto-encoders for efficient sampling.
problem Training energy-based models by maximum likelihood is challenging due to intractable partition functions and difficult sampling from the model distribution.
method The authors propose using a variational auto-encoder to initialize finite-step MCMC sampling, specifically Langevin dynamics, to train the energy-based model.
result The proposed method enables training energy-based models using maximum likelihood, generating samples comparable to GANs and EBMs.
3D object detection improved using energy-based models.
problem Accurate 3D object detection in cluttered environments from sparse LiDAR data.
method Designing a differentiable pooling operator for 3D bounding boxes integrated into a state-of-the-art 3D object detector.
result Our approach consistently outperforms the SA-SSD baseline across all 3DOD metrics on the KITTI dataset.
This paper shows ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
problem Understanding the lookahead capabilities of next-token prediction models.
method Unified view of ARMs and EBMs, establishing a bijection and deriving equivalence.
result ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.
EBM improves car insurance claim severity and frequency prediction while maintaining interpretability.
problem Balancing predictive accuracy and interpretability in insurance claim modeling.
method Combines GAM and cyclic gradient boosting, providing interpretable predictions.
result EBM outperforms benchmark models in claim severity and frequency prediction.
Energy-based models (EBMs), a.k.a. un-normalized models, have had recent successes in continuous spaces. However, they have not been successfully applied to model text sequences. While decreasing the energy at training samples is straightforward, mining (negative) samples where the energy should be increased is difficu…
Alzheimer's Disease (AD) is characterized by a cascade of biomarkers becoming abnormal, the pathophysiology of which is very complex and largely unknown. Event-based modeling (EBM) is a data-driven technique to estimate the sequence in which biomarkers for a disease become abnormal based on cross-sectional data. It can…
Bidirectional bounds stabilize training of energy-based models.
problem Training energy-based models is difficult and prone to instability.
method Propose bidirectional bounds linking to gradient penalty and Jacobi-determinant estimator.
result Significant stabilization and high-quality density estimation achieved.
Event-based models (EBM) are a class of disease progression models that can be used to estimate temporal ordering of neuropathological changes from cross-sectional data. Current EBMs only handle scalar biomarkers, such as regional volumes, as inputs. However, regional aggregates are a crude summary of the underlying hi…
A method to automatically learn proposal distributions for energy-based regression models.
problem Manual design and initial estimate of proposal distributions for energy-based regression models.
method Introduces a method to learn an effective proposal distribution automatically, parameterized by a separate network head, and derives a unified training objective to minimize KL divergence and negative log-likelihood.
result Consistently outperforms conventional MDN training on four real-world regression tasks within computer vision.
EBM life cycle project improves MCMC for image generation, defense, and density modeling.
problem Improving MCMC for diverse EBM applications.
method Three novel MCMC initialization methods for negative samples.
result State-of-the-art performance gains across three applications.
EBMs trained on discrete data using heat equations on graph structures.
problem Training EBMs on discrete or mixed data.
method Heat equations on graph structures for data perturbation.
result Efficacy demonstrated in various applications.
Adaptive multi-stage density ratio estimation improves learning of latent space EBM.
problem Learning energy-based models in latent space is computationally expensive and challenging.
method Adaptive multi-stage density ratio estimation using NCE to bridge the gap between prior and posterior densities.
result The method enables more expressive prior models and sharpens the latent space EBM.
Paper proposes a new method to learn EBMs and their partition function.
problem Intractability of exact MLE for EBMs due to partition function computation.
method Jointly learns an energy model and its log-partition function using neural networks.
result First tractable method for optimizing sparsemax loss in large spaces.
A new method improves training of energy-based models.
problem Training energy-based models is challenging.
method Diffusive Classification (DiffCLF) objective.
result DiffCLF enables EBMs with higher fidelity and broader applicability.
Paper tackles multi-task learning for molecular property prediction with limited data.
problem Limited labeled data for each molecular property task in drug discovery.
method Proposes SGNN-EBM method to utilize relation graph between tasks and improve multi-task learning performance.
result Empirical results show the effectiveness of SGNN-EBM.
Energy-Based Models (EBMs) assign unnormalized log-probability to data samples. This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning, and many more. But training of EBMs using standard maximum likelihood is extremely slow b…
The paper addresses interpretability issues in EBM models by improving feature selection and reducing spurious interactions.
problem Interpretability issues in EBM models, especially spurious interactions and single feature dominance.
method Alternate Cross-feature selection, ensemble features, and model configuration alteration techniques.
result Our approach improves interpretability and predictive performance of EBM models, reducing spurious interactions and single feature dominance.
Global Autoregressive Models (GAMs) are a recent proposal [Parshakova et al., CoNLL 2019] for exploiting global properties of sequences for data-efficient learning of seq2seq models. In the first phase of training, an Energy-Based model (EBM) over sequences is derived. This EBM has high representational power, but is u…
A new method combines energy-based models and entropy-regularized optimal transport.
problem Combining energy-based models and optimal transport for generative modeling.
method Energy-guided Entropic Neural Optimal Transport (E-ENT)
result Proves generalization bounds and validates scalability in image translation.
This paper proposes a more efficient training method for energy-based models.
problem The computational burden and validity trade-off in Contrastive Divergence training.
method Introducing Diffusion Contrastive Divergence (DCD) to replace Langevin dynamics with diffusion processes.
result The proposed DCDs are more computationally efficient and handle gradient terms better than Contrastive Divergence.
EBMs become opaque in high dimensions; LASSO sparsifies them.
problem Reducing complexity and improving interpretability of EBMs in high-dimensional settings.
method Applying LASSO to reweight and remove less relevant terms from EBMs.
result EBMs maintain transparency and fast scoring times with reduced complexity.
EBM reduces dimensionality for estimating heterogeneous CATEs.
problem Estimating CATEs requires many confounding variables, increasing sample complexity.
method Proposes an EBM that learns a low-dimensional representation of variables.
result EBM representations keep CATE estimates consistent and perform better than other methods.
New method improves sampling from noisy energy models.
problem Training and sampling challenges in Energy-Based Models.
method Pseudo-Gibbs sampling with moment matching.
result Effective sampling from clean model using noisy DSM-trained model.
This paper tackles sampling issues in latent space EBMs by introducing diffusion-based amortization.
problem Degenerate MCMC sampling quality hinders latent space EBM learning and generation quality.
method Introduces diffusion-based amortization for long-run MCMC sampling.
result The learned amortization of MCMC is a valid long-run MCMC sampler.
A new method trains discrete EBMs without sampling.
problem Training EBMs on discrete spaces is hard.
method Energy Discrepancy (ED), a contrastive loss.
result ED offers theoretical guarantees for various perturbation types.
Energy based models (EBMs) are appealing due to their generality and simplicity in likelihood modeling, but have been traditionally difficult to train. We present techniques to scale MCMC based EBM training on continuous neural networks, and we show its success on the high-dimensional data domains of ImageNet32x32, Ima…
Unified framework connects NCE, MIS, RLR, and bridge sampling for EBMs.
problem Challenges in parameter estimation for intractable likelihood EBMs.
method Unified framework connecting NCE, RLR, MIS, and bridge sampling.
result Unified perspective clarifies relationships among existing methods.
XLabel tool reduces medical experts' workload by 40% and explains its decisions.
problem Efficiently labeling large electronic health records.
method Visual-interactive tool using Explainable Boosting Machine (EBM) for classification and explanation.
result EBM achieves high accuracy and explainability, even with mislabeled data.
Energy Matching unifies flow matching and energy-based models for generative modeling.
problem Inability of flow-based models to integrate partial observations and priors.
method Energy Matching framework that integrates flow matching and energy-based models.
result Substantially outperforms existing EBMs on CIFAR-10 and ImageNet generation.
EBMs trained with ML are shown to behave like GANs with a self-adversarial loss.
problem Training EBMs with ML is intractable due to intractable unnormalized distributions.
method Replaced MCMC with deterministic gradient descent ODE solutions to study density induced by dynamics.
result EBM training is effectively a self-adversarial procedure rather than ML estimation.
A new method trains and samples from energy-based models using diffusion recovery likelihood.
problem Training and sampling high-dimensional datasets with energy-based models is challenging.
method Trains EBMs with a diffusion recovery likelihood method, maximizing conditional probabilities of data at different noise levels.
result Generates high-fidelity images with low FID and inception scores, and accurately estimates normalized data density.
STANLEY improves sampling for complex data models.
problem Training Energy-Based models with intractable normalizing constants.
method Anisotropic Langevin Dynamics with gradient-informed covariance.
result Geometrically uniformly ergodic Markov Chain for sampling.
Paper explores balancing market dynamics and interpretable forecasting models for energy prices.
problem Tackles the challenge of accurately predicting mFRR price and understanding market dynamics.
method Compares XGBoost and EBM for forecasting mFRR activation price in the balancing market.
result EBM provides comparable forecasting accuracy to XGBoost but with higher interpretability.
Stochastic defense improves natural classifiers against adversarial attacks.
problem Vulnerability of deep networks to adversarial attacks.
method Long-run MCMC sampling with Energy-Based Model for adversarial purification.
result Balancing memoryless and metastable behavior leads to effective purification and robust classification.
EBM model for functional data using path measure tilting.
problem Modeling distributions of functions from irregularly sampled data.
method Spectral decomposition of an energy-based model with a Gaussian Process path measure to reweight the distribution.
result Effective approach to up-scaling functional data.