Technical report on f-divergences and f-GAN training properties.
problem Understanding and optimizing f-divergences for GAN training.
method Elementary derivation and detailed expressions of f-divergences and their variational lower bounds.
result Informative properties of f-divergences and f-GAN training, including gradient matching and stability improvements.
Paper bridges f-GANs and WGANs for better image generation.
problem Learning high-dimensional distributions using GANs.
method List constraints, minimize Lagrangian relaxation, propose KL-Wasserstein GAN.
result Empirical success on synthetic and real-world image generation benchmarks.
Improves Bridge estimators using f-GAN to minimize RMSE.
problem Estimating ratios of normalizing constants efficiently.
method Proposes f-GAN-Bridge estimator using bijective transformations and f-divergence minimization.
result Optimal in minimizing asymptotic RMSE among candidate transformations.
Analyzes how restricted f-GANs differ from classical inference methods.
problem Understanding the inductive bias of generative adversarial networks.
method Theoretical characterization of restricted f-GANs, focusing on linear KL-GANs.
result The optimal generator distribution is a combination of maximum likelihood and method of moments solutions.
The paper analyzes the statistical properties of GANs using f-divergence.
problem Understanding the statistical behavior of GANs and comparing different f-divergences. method Asymptotic analysis of f-divergence GANs, including Kullback-Leibler divergence. result Asymptotically equivalent GANs with the same discriminator classes for correctly specified models.
There has recently been a steady increase in the number iterative approaches to density estimation. However, an accompanying burst of formal convergence guarantees has not followed; all results pay the price of heavy assumptions which are often unrealistic or hard to check. The Generative Adversarial Network (GAN) lite…
Develops a convex duality framework for analyzing GANs.
problem Analyzing how GANs behave under different discriminator constraints.
method Introduces a convex duality framework to interpret GANs under constrained discriminators.
result Shows that the GAN formulation can be interpreted as minimizing a divergence to penalized moments of the data distribution.
In this note, we point out a basic link between generative adversarial (GA) training and binary classification -- any powerful discriminator essentially computes an (f-)divergence between real and generated samples. The result, repeatedly re-derived in decision theory, has implications for GA Networks (GANs), providing…
Paper connects GANs to robust estimation, leading to efficient computation of optimal estimators.
problem Statistical robust estimation under contaminated data models.
method Establishes connection between f-GANs and depth functions through f-Learning. result Appropriate discriminator network structures in GANs lead to optimal robust estimators.
Unified framework for generative models incorporating VAE and GAN.
problem Flexible incorporation of diverse measures of probability distance in generative models.
method Unified f-divergence generative model (f-GM) that incorporates both VAE and f-GAN.
result Unified f-GM enables flexible design of f-divergence functions without changing network structure.
Nowozin \textit{et al} showed last year how to extend the GAN \textit{principle} to all f-divergences. The approach is elegant but falls short of a full description of the supervised game, and says little about the key player, the generator: for example, what does the generator actually converge to if solving the GAN…
This work reveals a primal-dual relationship between GANs and Autoencoders, improving their theoretical understanding.
problem Improving the theoretical understanding of GANs and Autoencoders.
method Study of f-GAN and WAE models, finding a primal-dual relationship and proving generalization bounds. result The f-GAN and WAE objectives are equivalent under certain assumptions, leading to improved theoretical understanding. Study extends DRO with IPMs, linking robustness to regularization and GANs.
problem Addressing robustness of deep neural networks to adversarial attacks.
method Distributionally Robust Optimization (DRO) with Integral Probability Metrics (IPMs).
result DRO under any IPM corresponds to a family of regularization penalties.
New theory explains GAN's high quality but low diversity.
problem Lack of theoretical justification for non-saturating GAN training.
method Showed non-saturating GAN training approximately minimizes a specific f-divergence.
result Non-saturating GAN training minimizes a particular f-divergence.
Generative neural samplers are probabilistic models that implement sampling using feedforward neural networks: they take a random input vector and produce a sample from a probability distribution defined by the network weights. These models are expressive and allow efficient computation of samples and derivatives, but …
A new GAN model α-GAN with tunable loss function addresses gradient vanishing and mode collapse issues.
problem Addressing vanishing gradients and mode collapse in GANs.
method Introduced a tunable GAN α-GAN using a supervised α-loss function. result Holistic understanding of α-GAN related to Arimoto divergence and convergence properties. Survey of GANs and autoencoders, addressing mode collapse and likelihood issues.
problem Addressing mode collapse and likelihood issues in GANs and autoencoders.
method Explains various GAN and autoencoder variants, their applications, and methods to resolve issues.
result Various methods to resolve mode collapse and improve likelihood in GANs and autoencoders.
We introduce a new approximation of f-divergences for machine learning.
problem Variational representations of f-divergences for machine learning. method Definition and analysis of Moreau-Yosida approximation of f-divergences with the Wasserstein-1 metric. result Generalization and relaxation of hard Lipschitz constraints in f-divergences. Paper introduces a new method to improve GANs by leveraging the discriminator's energy.
problem Improving the quality of generated samples in GANs.
method Discriminator Contrastive Divergence (DCD) method.
result Significant improvement in generation quality on synthetic and real-world data.
This paper tackles sample elicitation for learning systems, introducing a method to incentivize truthful samples.
problem Eliciting credible training samples for complex distributions from humans is challenging.
method Introduces a deep learning aided method to incentivize truthful samples from self-interested and rational agents.
result Achieves approximate incentive compatibility in eliciting truthful samples via accurate estimation of f-divergence function. Improved GAN training stability through tunable classification losses.
problem Training instabilities in GANs.
method Reformulated GAN value function using class probability estimation (CPE) losses, defined (αD,αG)-GANs. result Tuning (αD,αG) can alleviate training instabilities. GANs can generate realistic data without minimizing a divergence, contrary to current theory.
problem Current theory suggests GANs minimize a divergence to generate realistic data.
method Discussed various loss functions for G, showing they are not divergences and do not have the same equilibrium.
result GANs can use a wide range of loss functions, not just divergences, to generate realistic data.