ELBO converges to a sum of entropies for many generative models.
problem Understanding the convergence of variational lower bounds in unsupervised learning.
method Analyzing the ELBO for a broad class of generative models, showing it equals a sum of entropies.
result The ELBO is equal to a sum of entropies at stationary points for many generative models.
We compute the second variation of the Ricci expander entropy and briefly discuss the linear stability of compact negative Einstein manifolds.
ED-VAE improves VAEs by explicitly including entropy components in ELBO.
problem Limitations of traditional VAEs with ELBO in generating high-quality samples and interpreting latent spaces.
method Introduces ED-VAE, a re-formulation of ELBO that includes entropy and cross-entropy components.
result Significantly enhances model flexibility and improves interpretability and generative performance.
VES-Gamma adapts EI using information-theoretic principles.
problem Optimizing black-box functions using Bayesian optimization.
method Variational Entropy Search (VES) and VES-Gamma algorithm.
result VES-Gamma improves EI by incorporating information-theoretic concepts.
ELBO of VAEs converges to a sum of three entropies.
problem Understanding the convergence of ELBO in VAEs.
method Analytical derivation of ELBO convergence for standard Gaussian VAEs.
result ELBO converges to a sum of three entropies at stationary points.
Introduces REVE, a regularization scheme that compresses class conditioned entropy.
problem Improving generalization performance of deep learning models.
method Identifies a variable responsible for final prediction, compresses class conditioned entropy, introduces a variational upper bound, and integrates a tractable loss into training.
result Demonstrates the efficiency of REVE on various neural networks and datasets.
GAN+VER improves GANs by regularizing entropy to reduce mode collapse.
problem Mode collapse in GANs where the generator fails to capture all modes.
method Maximizing a variational lower bound on the entropy of generated samples.
result Significant improvement in evaluation metrics for real and generated samples.
Sharp bounds on neural network approximation rates and widths.
problem Estimating approximation rates, metric entropy, and n-widths of shallow neural networks.
method Introducing smoothly parameterized dictionaries and providing upper and lower bounds.
result Sharp bounds on approximation rates, metric entropy, and n-widths for neural networks with various activation functions.
In this note, we establish the first variation formula of the adjusted log entropy functional Ya introduced by Ye in \cite{Y2}. As a direct consequence, we also obtain the monotonicity of Ya along the Ricci flow.
Entropy analysis via kernel methods for probabilistic inference.
problem Entropy analysis of probability distributions.
method Kernel methods and reproducing kernel Hilbert spaces for entropy estimation.
result New upper-bounds on log partition functions for probabilistic inference.
In this expository note, we study the second variation of Perelman's entropy on the space of Kahler metrics at a Kähler-Ricci soliton. We prove that the entropy is stable in the sense of variations. In particular, Perelman's entropy is stable along the Kähler-Ricci flow. The Chinese version of this note has appeared in…
Minimal volume entropy vanishes for mapping tori over 3-manifolds.
problem Volume entropy of mapping tori over 3-manifolds.
method A variation of amenable category and minimal volume entropy of a homology class.
result Minimal volume entropy vanishes.
A new measure helps compute suboptimality in entropy-regularized methods.
problem Computing suboptimality in entropy-regularized variational objectives when unnormalised densities are unavailable.
method Introduced 'kernel gradient discrepancy' (KGD) to compute suboptimality explicitly.
result KGD characterizes kernel Stein discrepancy (KSD) in the standard Bayesian context and measures variational gradient size.
In this paper we provide a detailed proof of the second variation formula, essentially due to Richard Hamilton, Tom Ilmanen and the first author, for Perelman's ν-entropy. In particular, we correct an error in the stability operator stated in Theorem 6.3 of [2]. Moreover, we obtain a necessary condition for linearly …
AR-DAE approximates entropy gradient for machine learning models.
problem Intractable computation of entropy gradient for continuous distributions.
method Amortized residual denoising autoencoder (AR-DAE) to approximate entropy gradient.
result AR-DAE provides an unbiased gradient approximation for entropy.
Extends specific relative entropy to multidimensional continuous martingales.
problem Mutual singularity of martingale laws in continuous time.
method Extension of specific relative entropy from one to multiple dimensions, including closed-form expressions for simple examples.
result Establishes that the lower bound on specific relative entropy from Gantert carries over to higher dimensions and is tight.
The well known maximum-entropy principle due to Jaynes, which states that given mean parameters, the maximum entropy distribution matching them is in an exponential family, has been very popular in machine learning due to its "Occam's razor" interpretation. Unfortunately, calculating the potentials in the maximum-entro…
Unified framework connects EI and information-theoretic acquisition functions.
problem Distinguish between Expected Improvement and information-theoretic acquisition functions.
method Introduces Variational Entropy Search (VES) to unify EI and information-theoretic approaches.
result EI can be seen as a variational inference approximation of Max-value Entropy Search (MES).
Rate GENERIC extends thermodynamics principles to non-equilibrium systems.
problem Understanding non-equilibrium thermodynamics and its relation to equilibrium thermodynamics.
method Developed a geometrical framework for rate GENERIC, extending Onsager's variational principle.
result Rate GENERIC structure provides a new perspective on thermodynamics in non-equilibrium systems.
Nonnegative sectional curvature linked to matrix displacement convexity.
problem Nonnegative sectional curvature in Riemannian manifolds.
method Matrix displacement convexity as a criterion for nonnegative sectional curvature.
result Entropy functional matrix displacement convexity implies nonnegative sectional curvature.
Study stability of compact Ricci solitons using entropy variations.
problem Linear stability condition for compact shrinking Ricci solitons.
method Second variation of Perelman's ν-entropy.
result Necessary and sufficient condition for linear stability.
This work improves VAEs using MCMC methods for better variational bounds.
problem Improving the expressiveness of variational distributions in VAEs.
method Entropy-based adaptation for MALA/HMC chains to optimize tighter variational bounds.
result Higher held-out log-likelihoods and improved generative metrics.
Paper tests for time-varying entropy in stock prices, finding periods of inefficiency.
problem Testing for time-varying entropy in stock price dynamics.
method Unbiased approximation of Shannon entropy variance, optimal rolling window selection, hypothesis testing.
result Existence of periods of market inefficiency for meme stocks.
Researchers calculate entropy of heat kernel on manifolds for very small times.
problem Estimating entropy of heat kernel on compact Riemannian manifolds for small times.
method Asymptotic expansion, polynomial expressions in curvature tensor components.
result First three coefficients of entropy expansion computed and expressed as polynomials.
The aim of this paper is to provide new theoretical and computational understanding on two loss regularizations employed in deep learning, known as local entropy and heat regularization. For both regularized losses we introduce variational characterizations that naturally suggest a two-step scheme for their optimizatio…
Generative models' ELBOs converge to entropy sums, proving for various models.
problem Proving convergence of ELBOs to entropy sums for various generative models.
method Proofs for individual models under realistic conditions.
result ELBOs of various generative models converge to entropy sums at all stationary points.
Differentiable PF via entropy-regularized OT for better inference.
problem Non-differentiability of traditional PF resampling methods.
method Entropy-regularized optimal transport for differentiable resampling.
result Convergent differentiable PF method with improved gradient estimates.
The paper analyzes how factorized Gaussian approximations underestimate uncertainty in variational inference.
problem Underestimation of uncertainty in variational inference using factorized Gaussian approximations.
method Examined the trade-off between shrinkage and delinking in approximating a Gaussian with a diagonal covariance matrix.
result Entropy of the factorized Gaussian approximation underestimates both componentwise variance and entropy of the original Gaussian.
New entropy-based objective for sparse coding improves learning.
problem Sparse coding with probabilistic priors and non-Gaussian observables.
method Derive a solely entropy-based learning objective for sparse coding parameters.
result Fully analytical ELBO objective for sparse coding with non-trivial posterior approximations.
BBVI with STL converges geometrically under perfect specification, with quadratic variance bound.
problem Convergence rate of BBVI with STL estimator.
method Proved geometric convergence rate with quadratic variance bound for BBVI with STL estimator.
result BBVI with STL converges geometrically under perfect variational family specification.
A new method for multi-objective Bayesian optimization using entropy search and variational lower bound maximization.
problem Efficiently optimizing multiple objectives in continuous domains.
method Approximates the Pareto-frontier using a mixture distribution and optimizes the balance through variational lower bound maximization.
result Demonstrated effectiveness especially with many objective functions.
Improved machine learning method estimates entropy production robustly.
problem Estimating entropy production from trajectory data.
method Variational representation of α-divergence loss functions. result Optimal α=−0.5 yields best performance. Study of entropy-regularized LQG MFGs with exploratory actions.
problem Optimizing multi-population mean field games with entropy regularization.
method Introduced exploratory actions and derived optimal action distributions.
result Optimal action distributions lead to ε-Nash equilibria in finite-population MFGs.
In this paper, we suggest a framework to make use of mutual information as a regularization criterion to train Auto-Encoders (AEs). In the proposed framework, AEs are regularized by minimization of the mutual information between input and encoding variables of AEs during the training phase. In order to estimate the ent…
We study a Boltzmann's type entropy functional (which appeared in existing literature) defined on Kähler metrics of a fixed Kähler class. The critical points of this functional are gradient Kähler-Ricci solitons, and the functional was known to be monotonically increasing along the Kähler-Ricci flow in the canonical cl…
State entropy regularization improves robustness in reinforcement learning, especially under structured perturbations.
problem Structured and spatially correlated perturbations in reinforcement learning.
method State entropy regularization, compared to policy entropy.
result State entropy regularization provides better robustness to structured and spatially correlated perturbations.
In this paper, the author discusses the eigenvalues and entropies under the harmonic-Ricci flow, which is the Ricci flow coupled with the harmonic map flow. We give an alternative proof of results for compact steady and expanding harmonic-Ricci breathers. In the second part, we derive some monotonicity formulas for eig…
Alexandrov spaces with non-negative curvature are characterized by the matrix displacement convexity of an entropy tensor.
problem Characterizing non-negative curvature in Alexandrov spaces
method Constructing a parallel trivialization of the entropy tensor
result The entropy tensor is matrix displacement convex on Alexandrov spaces
Paper presents estimators for entropy and information in probabilistic models.
problem Estimating entropy and mutual information in high dimensions is challenging.
method EEVI uses importance sampling with proposal distributions like amortized variational inference and sequential Monte Carlo.
result EEVI delivers accurate upper and lower bounds on information quantities.
We study a notion of relative entropy motivated by self-expanders of mean curvature flow. In particular, we obtain the existence of this quantity for arbitrary hypersurfaces trapped between two disjoint self-expanders asymptotic to the same cone. This allows us to begin to develop the variational theory for the relativ…
Proposes a new method to measure epistemic uncertainty in Bayesian neural networks.
problem Measuring epistemic uncertainty in Bayesian neural networks for out-of-distribution detection.
method Proposes measuring disagreement between logits and their pre-softmax counterparts as an epistemic uncertainty measure.
result Proposed epistemic uncertainty scores outperform mutual information and equal predictive entropy performance.
A new reinforcement learning method reduces action complexity for robust control.
problem Deep reinforcement learning's susceptibility to spurious correlations.
method Minimizing trajectory entropy to encourage simple, predictable actions.
result Trajectory Entropy Reinforcement Learning achieves superior performance and robustness.
Sharp lower bounds on shallow neural networks' approximation rates are derived.
problem The efficiency of shallow neural networks in approximating functions.
method Lower bounding the L2-metric entropy and Kolmogorov n-widths of the convex hull of neural network basis functions. result Sharp lower bounds on the approximation rates for shallow neural networks are provided.
Adaptive approximations improve variational inference for complex models.
problem Efficiently approximate marginal distributions and partition functions in complex probabilistic models.
method Two classes of adaptive approximations that include Bethe, tree-reweighted, and convex free energies.
result Proposed approximations automatically adapt to a given model and outperform existing methods.
A new method for VAEs improves latent space disentanglement without violating probability laws.
problem Improving latent space disentanglement in VAEs without violating probability laws.
method Developed a Renyi VAE with a conditional distribution not learned, using Singular Value Decomposition for evaluation.
result Improved latent space disentanglement without violating probability laws.
PAC-Bayesian bounds for MLPs with cross entropy loss validated.
problem Generalization bounds for MLPs with cross entropy loss.
method Introduced probabilistic explanations and proved PAC-Bayesian bounds using ELBO.
result MLPs with cross entropy loss inherently guarantee PAC-Bayesian generalization bounds.
In this paper we introduce entropy-stability and F-stability for homothetically shrinking Yang-Mills solitons, employing entropy and second variation of F-functional respectively. For a homothetically shrinking soliton which does not descend, we prove that entropy-stability implies F-stability. These stabil…
Upper bound on index of rotationally symmetric self-shrinking tori.
problem Stability of singularities in mean curvature flow.
method Entropy functional and eigenvalue analysis.
result Upper bound on the index of rotationally symmetric self-shrinking tori.