New divergences extend Bregman and skew Jensen, including f-divergences.
problem Developing new divergences to include f-divergences.
method Introducing g-Bregman and skew g-Jensen divergences, showing they include f-divergences.
result g-divergences generalize existing divergences and inequalities.
The paper introduces a new divergence measure for variational autoencoders to improve reconstruction and generation.
problem Balancing reconstruction and generalizability in latent space of variational autoencoders.
method Presented a regularisation mechanism based on skew-geometric Jensen-Shannon divergence.
result The skew-geometric Jensen-Shannon divergence leads to better reconstruction and generation in variational autoencoders.
SMT trains generative models by estimating mixture scores, outperforming existing methods.
problem Training one-step generative models efficiently and effectively.
method Score-of-Mixture Training (SMT) estimates the score of mixture distributions between real and fake samples.
result SMT/SMD outperform existing methods on CIFAR-10 and ImageNet 64x64 datasets.
Study on geometric Jensen-Shannon divergence for Gaussian measures in Hilbert space.
problem Computing divergence between Gaussian measures in infinite-dimensional Hilbert space.
method Closed form expression and regularization for divergence calculation.
result Closed form expression and regularization for Geometric Jensen-Shannon divergence.
Paper shows how to break down a specific type of divergence into simpler parts.
problem Understanding and simplifying divergence functions.
method Decomposes the symmetric Bregman divergence into two types of Jensen divergences and a Bregman divergence, and extends this to include f-divergences.
result Sum decomposition of divergence into simpler parts is possible.
New framework using Jensen-Shannon divergence improves domain adaptation theory.
problem Incoherence between empirical domain adversarial training and theoretical H-divergence. method Established new theoretical framework based on Jensen-Shannon divergence, derived bi-directional upper bounds.
result Framework exhibits flexibilities for various transfer learning problems.
Proposes a new divergence measure for probability distributions.
problem Challenges in estimating divergences from empirical samples.
method Embeds data into RKHS, computes Jensen-Shannon divergence between covariance operators.
result Establishes RJSD as a lower bound on Jensen-Shannon divergence, enabling variational estimation.
Study compares statistical properties and power of divergence measures for credit risk monitoring.
problem Detecting distributional shifts in credit risk models.
method Derives statistical properties and chi-square benchmark values for Jensen-Shannon Divergence and Kullback-Leibler Divergence, demonstrating their applicability in credit risk monitoring.
result Jensen-Shannon Divergence and Kullback-Leibler Divergence follow chi-square distributions and reveal practical trade-offs in minimizing false positives vs. detecting changes.
A new objective function using Jensen-Shannon divergence improves generative learning from multiple data types.
problem Learning from multiple data types efficiently and accurately.
method Proposes a novel objective function using Jensen-Shannon divergence to approximate multimodal posteriors directly.
result The mmJSD objective optimizes an ELBO and improves generative learning tasks.
Proposes a new loss function for learning with noisy labels.
problem Improving model learnability with noisy labels.
method Uses generalized Jensen-Shannon divergence as a noise-robust loss function.
result Shows state-of-the-art results on noisy data.
In this report, we derive a non-negative series expansion for the Jensen-Shannon divergence (JSD) between two probability distributions. This series expansion is shown to be useful for numerical calculations of the JSD, when the probability distributions are nearly equal, and for which, consequently, small numerical er…
New divergence measures improve KL approximation.
problem Improving KL divergence approximation without AC condition.
method Introduced α-geodesical skew divergence. result Properties of α-geodesical skew divergence studied. In this paper, we present a new wrapper feature selection approach based on Jensen-Shannon (JS) divergence, termed feature selection with maximum JS-divergence (FSMJ), for text categorization. Unlike most existing feature selection approaches, the proposed FSMJ approach is based on real-valued features which provide mo…
A new metric uses nonparametric comparison for fitting parametric distributions.
problem Measuring goodness-of-fit for nonlinear models using maximum likelihood estimation.
method Survival Jensen-Shannon divergence (SJS) and its empirical counterpart (ESJS) for nonparametric comparison. result The ESJS can be used as a measure of goodness-of-fit in maximum likelihood estimation. New method improves understanding of machine learning model performance.
problem Understanding how well machine learning models generalize from training data to unseen data.
method Auxiliary Distribution Method to derive new generalization error bounds.
result Upper bounds on generalization errors are tighter and more applicable.
New divergences introduced in dually flat spaces with properties.
problem Measuring discrepancy between probability distributions in dually flat spaces.
method Introducing two types of divergences based on affine coordinates and potentials, and deriving relational equations.
result Generalization of the law of cosines and new inequalities between divergences.
JECL clusters images and captions by jointly learning representations and assignments.
problem Clustering image-caption pairs with limited structured training data.
method Parallel encoders trained with clustering and alignment objectives, minimizing KL divergence and maximizing Jensen-Shannon divergence, with regularizers.
result JECL outperforms single-view and multi-view methods on large image-caption datasets.
Mathematical study of excess growth rate connects info theory with finance.
problem Understanding the excess growth rate in portfolio theory.
method Axiomatic characterization theorems of excess growth rate in terms of relative entropy, Jensen's inequality gap, and logarithmic divergence.
result Established rich connections between information theory and finance.
Paper proposes f-DPG for aligning language models with preferences.
problem Aligning language models with user preferences.
method Uses f-divergence to approximate target distributions and minimizes a forward KL from it using DPG.
result Jensen-Shannon divergence often outperforms forward KL divergence, leading to significant improvements.
This paper raises an implicit manifold learning perspective in Generative Adversarial Networks (GANs), by studying how the support of the learned distribution, modelled as a submanifold Mθ, perfectly match with Mr, the support of the real data distribution. We show that optimizing Jensen-Sha…
Develops LSH schemes for f-divergences and mutual information loss.
problem Approximating nearest neighbors in high-dimensional probability distributions.
method General framework and specific LSH schemes for f-divergences and mutual information loss.
result Generalized Jensen-Shannon divergence can be approximated by Hellinger distance.
The study reveals the hierarchical structure of the international FOREX market using currency fluctuation distribution similarities.
problem Understanding the hierarchical structure of the international FOREX market.
method Using Jensen-Shannon divergence to quantify the similarity between normalized logarithmic return distributions of currencies.
result Clusters of currencies are consistent with the nature of underlying economies but diverge during crises.
Paper monitors system state sequences to detect and assess deviations.
problem Detecting and evaluating deviations in dynamic systems.
method Data reduction, symbolic representation, anomaly detection, Markov Chains, generalized Jensen-Shannon Divergence.
result The approach detects and assesses system deviations probabilistically.
Paper resolves bias in ALFT training using generalized alignment games.
problem Systematic bias in estimating logarithmic rewards from small batches.
method Generalized Distributional Alignment Games, U-statistics, minimax polynomial estimators, Variance-Optimal Augmented Polynomial Optimization Program (AQP) Estimator.
result Proves optimal bias and accelerated convergence in ALFT training.
Generative Adversarial Networks (GANs) have become a widely popular framework for generative modelling of high-dimensional datasets. However their training is well-known to be difficult. This work presents a rigorous statistical analysis of GANs providing straight-forward explanations for common training pathologies su…
RényiCL uses Rényi divergence for robust contrastive learning with stronger data augmentations.
problem Learning useful representations from multiple data views with hard augmentations.
method RényiCL employs Rényi divergence for contrastive learning, using a novel variational objective to manage hard negative sampling.
result RényiCL achieves better performance with stronger augmentations compared to other methods.
Paper introduces metrics to evaluate missing data imputation without ground truth.
problem Handling missing data in time series without ground truth.
method Introduces Wasserstein distance (WD) and Jensen-Shannon divergence (JSD) as metrics to evaluate imputation quality.
result WD and JSD are effective metrics for assessing missing data imputation quality.
MIM learns useful representations with high mutual information.
problem Learning useful representations for downstream tasks.
method Symmetric Jensen-Shannon divergence and mutual information regularizer in an encoder/decoder framework.
result MIM learns high mutual information representations without posterior collapse.
Paper introduces SDM for detecting LLM hallucinations, improving on entropy tests.
problem Challenges of Large Language Models (LLMs) with non-factual, nonsensical responses.
method Joint clustering on sentence embeddings to measure semantic divergence between prompts and responses.
result SDM framework detects deeper form of arbitrariness in LLM responses.
Two methods factor out prior knowledge from low-dimensional embeddings.
problem Visualizing data without considering background knowledge.
method JEDI for tSNE and CONFETTI for any embedding.
result Embeddings reveal meaningful structure hidden by prior knowledge.
Proposes a scalable method for counterfactual prediction using machine learning.
problem De-bias causal estimators with high-dimensional data in observational studies.
method Uses entropy balancing to learn weights minimizing Jensen-Shannon divergence, leading to robust counterfactual predictions.
result Consistent causal estimation if either propensity score or outcome model is correctly specified.
Formula derived for sample complexity in binary hypothesis testing.
problem Determine the minimum number of samples to distinguish between two distributions.
method Developed a formula for sample complexity in both prior-free and Bayesian settings, using Jensen-Shannon and Hellinger divergences.
result Formula characterizes sample complexity for a wide range of error parameters, up to multiplicative constants.
GPdoemd optimizes experiments for model discrimination using Gaussian processes.
problem Discriminating between competing models when data is limited.
method Developed a new design criterion and Gaussian process surrogate method for black-box models.
result Demonstrated improved model discrimination using Gaussian process surrogates.
Paper proposes a method to discover topic evolutions from text data.
problem Difficulty in identifying new research topics from large text data.
method Uses sparseness-constrained Non-negative Matrix Factorization with generalized Jensen-Shannon divergence.
result Extracts more prominent topics and visualizes term-topic relationships.
A new framework for offline RL improves policy flexibility and regularity.
problem Lack of environmental interactions in offline RL leads to poor policy performance.
method Proposes a behavior-regularized implicit policy framework with modified policy-matching methods.
result The framework improves policy effectiveness and robustness beyond static datasets.
New method minimizes f-divergence for better imitation learning.
problem Learning from multi-modal demonstrations, especially interpolating between modes.
method Minimizing reverse KL divergence or I-projection for any f-divergence.
result Our method reliably imitates multi-modal behaviors better than existing methods.
Parametric adversarial divergences, which are a generalization of the losses used to train generative adversarial networks (GANs), have often been described as being approximations of their nonparametric counterparts, such as the Jensen-Shannon divergence, which can be derived under the so-called optimal discriminator …
Paper trains two models to improve synthesis quality and unsupervised learning.
problem Improving synthesis quality and unsupervised learning.
method Joint training of an energy-based model and a flow-based model using adversarial value function.
result Significant improvement in synthesis quality of the flow model.
The success of popular algorithms for deep reinforcement learning, such as policy-gradients and Q-learning, relies heavily on the availability of an informative reward signal at each timestep of the sequential decision-making process. When rewards are only sparsely available during an episode, or a rewarding feedback i…
Improved GANs estimate convergence rate for density estimation.
problem Improving the accuracy of density estimation with GANs.
method Proved an oracle inequality for JS divergence between GAN estimate and true density.
result JS-divergence rate of convergence is (logn/n)2β/(2β+d). Study risk-sensitive imitation learning using GAIL and Wasserstein distance.
problem Improve imitation learning performance by considering risk profiles.
method Formulate risk-sensitive imitation learning, derive optimization problems for JS divergence and Wasserstein distance, develop algorithms.
result RS-GAIL algorithms outperform GAIL and RAIL in MuJoCo and OpenAI tasks.
We introduce a new approximation of f-divergences for machine learning.
problem Variational representations of f-divergences for machine learning. method Definition and analysis of Moreau-Yosida approximation of f-divergences with the Wasserstein-1 metric. result Generalization and relaxation of hard Lipschitz constraints in f-divergences. Counterexample found to estimate for skew-symmetric tensors.
problem Estimate for skew-symmetric tensors was claimed and used for classification results.
method Analysis of the estimate in arXiv:2103.15482.
result Counterexample disproves the estimate for skew-symmetric tensors.
MIM learns joint distributions with mutual information and low divergence.
problem Learning joint distributions over observations and latent variables.
method Probabilistic auto-encoder with three design principles: low divergence, high mutual information, and low marginal entropy.
result MIM learns representations with high mutual information, consistent encoding and decoding distributions, effective latent clustering, and comparable data log likelihood to VAE.
This study considers the multivariate segmentation procedure under the assumption of the multivariate Gaussian mixture. Jensen-Shannon divergence between two multivariate Gaussian distributions is employed as a discriminator and a recursive segmentation procedure is proposed. The daily log-return time series for 30 cur…
This paper introduces f-DPO, a generalized approach to Direct Preference Optimization using diverse divergence constraints.
problem Aligning large language models with human preferences while mitigating safety risks.
method Incorporates diverse divergence constraints to simplify the relationship between reward and optimal policy, eliminating the need for estimating the normalizing constant.
result Optimizes LLMs to align with human preferences more efficiently and under a broader set of divergence constraints.
New GAN loss functions improve image generation quality and stability.
problem Improving the performance of GANs in generating high-quality images.
method Introducing least kth-order GAN (LkGAN) and Rényi-centric GAN loss functions. result The proposed loss functions lead to better image quality and stability.
In this report, we present an unsupervised machine learning method for determining groups of molecular systems according to similarity in their dynamics or structures using Ward's minimum variance objective function. We first apply the minimum variance clustering to a set of simulated tripeptides using the information …