ETM identifies field-specific keywords in text classification.
problem Unsupervised text classification with field-specific keywords.
method Weighted Lasso penalty and pairwise Kullback-Leibler divergence penalty for topic separation.
result ETM improves topic coherence by 22% and 10% compared to LDA.
Paper calculates KL divergence for isotropic Gaussian-Markov fields.
problem Measuring divergence between isotropic Gaussian-Markov fields.
method Derives closed-form KL divergence expressions.
result Develops new similarity measures in image processing.
Paper improves variational inference on Boolean hypercube using quantum methods.
problem Improving variational inference for pairwise Markov random fields on the Boolean hypercube.
method Quantum relaxations of the Kullback-Leibler divergence for upper-bounds, primal-dual optimization, and greedy selection of hierarchies.
result Efficient algorithm and improved bounds for variational inference.
This paper presents a distance-based discriminative framework for learning with probability distributions. Instead of using kernel mean embeddings or generalized radial basis kernels, we introduce embeddings based on dissimilarity of distributions to some reference distributions denoted as templates. Our framework exte…
Bayesian optimization agent learns user preferences from pairwise comparisons.
problem Learning user preferences from unknown and infinite choices.
method Sequential Bayesian optimization with pairwise comparisons.
result Optimal agent strategy minimizes remaining system uncertainty.
MAP inference for general energy functions remains a challenging problem. While most efforts are channeled towards improving the linear programming (LP) based relaxation, this work is motivated by the quadratic programming (QP) relaxation. We propose a novel MAP relaxation that penalizes the Kullback-Leibler divergence…
Study shows the corrected Akaike criterion is inadmissible for estimating Kullback-Leibler discrepancy.
problem Inadmissibility of the corrected Akaike information criterion for estimating Kullback-Leibler discrepancy.
method Loss estimation framework to demonstrate inadmissibility and provide improved estimators.
result Improved estimators of Kullback-Leibler discrepancy are provided and perform well in reduced-rank situations.
Mixture distributions arise in many parametric and non-parametric settings -- for example, in Gaussian mixture models and in non-parametric estimation. It is often necessary to compute the entropy of a mixture, but, in most cases, this quantity has no closed-form expression, making some form of approximation necessary.…
New tensor framework connects Fisher information, hypergraphs, and multi-observable correlations.
problem Missing structure in pairwise Fisher graphs for multi-observable radiation patterns.
method Higher-order Fisher tensors and natural exponential-family coordinates.
result Exact triality of Fisher tensors, cumulants, and hypergraphs.
Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using …
New dispersion indices based on inaccuracy and divergence introduced for information measures.
problem Measuring variability in uncertainty measures.
method Introducing new dispersion indices based on Kerridge inaccuracy and Kullback-Leibler divergence.
result Properties, bounds, and examples of new dispersion indices presented.
Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its or…
The problem of filtering information from large correlation matrices is of great importance in many applications. We have recently proposed the use of the Kullback-Leibler distance to measure the performance of filtering algorithms in recovering the underlying correlation matrix when the variables are described by a mu…
A new method optimizes a generalized Kullback-Leibler divergence for better simulation-based inference.
problem Optimizing likelihood functions when they are only known implicitly.
method Optimizes a generalized Kullback-Leibler divergence that accounts for normalization constants in unnormalized distributions.
result Unified approach that combines Neural Posterior Estimation and Neural Ratio Estimation.
We present a derivation of the Kullback Leibler (KL)-Divergence (also known as Relative Entropy) for the von Mises Fisher (VMF) Distribution in d-dimensions.
Paper studies regularized KKL divergence for distributions with disjoint supports.
problem Inability of original KKL divergence to handle distributions with disjoint supports.
method Proposes a regularized variant of KKL divergence, derives bounds, and provides closed-form expression.
result Regularized KKL divergence is well-defined for all distributions and has finite-sample bounds.
Paper analyzes sparse aggregation in GLMs with Kullback-Leibler risk bounds.
problem Sparse aggregation in GLMs for parameter approximation.
method Exponential weighted aggregation scheme with Kullback-Leibler risk bounds.
result Sharp oracle inequality for Kullback-Leibler risk with leading constant 1 and minimax-optimal rate of aggregation.
Proposes a guaranteed regularization method for maximum likelihood estimation using gauge symmetry in Kullback-Leibler divergence.
problem Overfitting in maximum likelihood estimation.
method Introduces a regularization approach based on gauge symmetry in Kullback-Leibler divergence.
result The method provides a theoretically guaranteed optimal model without frequent hyperparameter tuning.
A new method for efficient Gaussian process inference using sparse approximations.
problem Scalable and accurate inference for latent Gaussian processes.
method Variational approximation with sparse inverse Cholesky factors and double Kullback-Leibler minimization.
result The proposed method can achieve highly accurate approximations with polylogarithmic time complexity.
We study a new ensemble of random correlation matrices related to multivariate Student (or more generally elliptic) random variables. We establish the exact density of states of empirical correlation matrices that generalizes the Marcenko-Pastur result. The comparison between the theoretical density of states in the St…
The paper proposes a new method to approximate Wasserstein-Fisher-Rao flows using Monte Carlo techniques.
problem Sampling from probability distributions and minimizing Kullback-Leibler divergence.
method Sequential Monte Carlo approximations of Wasserstein-Fisher-Rao gradient flows.
result The proposed method outperforms other Monte Carlo algorithms in certain conditions.
We study the problem of ranking from crowdsourced pairwise comparisons. Answers to pairwise tasks are known to be affected by the position of items on the screen, however, previous models for aggregation of pairwise comparisons do not focus on modeling such kind of biases. We introduce a new aggregation model factorBT …
Study compares statistical properties and power of divergence measures for credit risk monitoring.
problem Detecting distributional shifts in credit risk models.
method Derives statistical properties and chi-square benchmark values for Jensen-Shannon Divergence and Kullback-Leibler Divergence, demonstrating their applicability in credit risk monitoring.
result Jensen-Shannon Divergence and Kullback-Leibler Divergence follow chi-square distributions and reveal practical trade-offs in minimizing false positives vs. detecting changes.
This paper improves active learning by using robust divergences for committee disagreement.
problem Active learning with high measurement costs.
method Query by committee with Bregman divergence (including Kullback-Leibler divergence as a special case).
result The proposed method is more robust and performs as well as or better than conventional methods.
Machine learning classification limits estimated using Kullback-Leibler divergence and Cohen's Kappa.
problem Estimating the best possible performance of machine learning classification algorithms.
method Relating Kullback-Leibler divergence to Cohen's Kappa and using the Chernoff-Stein Lemma to estimate error rates.
result Classification algorithms could not have performed any better due to underlying probability density functions for the two classes.
The paper optimizes distribution estimation with high probability in Kullback-Leibler divergence.
problem Estimating discrete distributions with high probability in Kullback-Leibler divergence.
method Uses online learning techniques for novel estimator construction via online-to-batch conversion.
result Optimal rate of estimation is pinned down up to a doubly logarithmic factor of K.
Combines expert models using Kullback-Leibler divergence to create a combined model.
problem Combining expert views on stochastic processes.
method Minimizes weighted Kullback-Leibler divergence to create a barycentre model.
result Existence and uniqueness of the barycentre model with explicit representation.
In this study, a pairwise comparison matrix is generalized to the case when coefficients create Lie group G, non necessarily abelian. A necessary and sufficient criterion for pairwise comparisons matrices to be consistent is provided. Basic criteria for finding a nearest consistent pairwise comparisons matrix (extend…
Paper introduces differential pairwise privacy for secure metric learning.
problem Securely measuring similarities of individuals given sensitive pairwise data.
method Develops differential pairwise privacy (DPP) to protect sensitive pairwise data.
result Achieves pairwise data privacy without significant performance loss.
Entropy measure quantifies volatility correlation and risk diversity in asset portfolios.
problem Quantifying volatility correlation and risk diversity in asset portfolios.
method Kullback-Leibler cluster entropy DC[P∥Q] for empirical and model probability distributions of realized volatility. result Portfolio built on diversity indexes derived from Kullback-Leibler entropy measure of realized volatility exhibits better performance.
Cross-entropy loss linked to metric learning, outperforming complex pairwise losses.
problem Improving metric learning performance without complex optimization schemes.
method Theoretical analysis linking cross-entropy to pairwise losses, showing cross-entropy as an upper bound and equivalent to mutual information maximization.
result Minimizing cross-entropy is equivalent to maximizing mutual information, leading to state-of-the-art performance.
Develops a statistical framework to measure uncertainty in model rankings based on human preferences.
problem Uncertainty in model rankings based on human preferences due to mismatch between human and model preferences.
method Statistical framework using pairwise comparisons by humans and models to provide rank-sets for each model.
result Rank-sets constructed using only pairwise comparisons by strong models often do not cover the true ranking of human preferences.
High dimensional data often contain multiple facets, and several clustering patterns can co-exist under different variable subspaces, also known as the views. While multi-view clustering algorithms were proposed, the uncertainty quantification remains difficult --- a particular challenge is in the high complexity of es…
New ONMF model minimizes KL divergence for better sparse data modeling.
problem Clustering and data modeling with sparse vectors.
method Developed KL-ONMF algorithm based on alternating optimization.
result KL-ONMF outperforms Frobenius-norm ONMF for document classification and hyperspectral image unmixing.
As one of the most important types of (weaker) supervised information in machine learning and pattern recognition, pairwise constraint, which specifies whether a pair of data points occur together, has recently received significant attention, especially the problem of pairwise constraint propagation. At least two reaso…
This paper provides efficient algorithms for computing entropy and KL divergence in Bayesian networks.
problem Computing entropy and KL divergence for Bayesian networks efficiently.
method Leveraging the graphical structure of Bayesian networks, the paper provides computationally efficient algorithms.
result Reduces computational complexity of KL divergence from cubic to quadratic for Gaussian BNs.
A brain computer interface (BCI) is a system which provides direct communication between the mind of a person and the outside world by using only brain activity (EEG). The event-related potential (ERP)-based BCI problem consists of a binary pattern recognition. Linear discriminant analysis (LDA) is widely used to solve…
Paper proposes Pcomp classification for binary classification with pairwise confidence comparisons.
problem Lack of pointwise labels due to privacy, confidentiality, or security reasons.
method Developed Pcomp classification, derived an unbiased risk estimator (URE), and improved it using correction functions and consistency regularization.
result Demonstrated the effectiveness of Pcomp classification methods.
Study on pairwise counter-monotonicity, a type of negative dependence.
problem Understanding and quantifying extremal negative dependence structures.
method Established stochastic representation and invariance property; showed implications and connections.
result Pairwise counter-monotonicity implies negative association and joint mix dependence.
Study measures irreversibility in crypto trends using Kullback-Leibler divergence.
problem Assessing irreversibility in cryptocurrency trends.
method Defined irreversibility index using Kullback-Leibler divergence between uptrend and downtrend distributions.
result Strong irreversibility in all analyzed cryptocurrencies, with trends evolving over time.
We show that the Kullback-Leibler distance is a good measure of the statistical uncertainty of correlation matrices estimated by using a finite set of data. For correlation matrices of multivariate Gaussian variables we analytically determine the expected values of the Kullback-Leibler distance of a sample correlation …
Paper proves Jeffrey's update rule minimizes relative entropy.
problem Improving Bayesian learning algorithms.
method More concise proof of Jeffrey's update rule.
result Jeffrey's update rule reduces relative entropy.
This paper examines the problem of ranking a collection of objects using pairwise comparisons (rankings of two objects). In general, the ranking of n objects can be identified by standard sorting methods using nlog2n pairwise comparisons. We are interested in natural situations in which relationships among the o…
Conditions for curves on a torus with specific pairwise intersections.
problem Finding curves on a torus with prescribed pairwise intersections.
method Necessary and sufficient conditions for curves on a torus with given pairwise intersections.
result Necessary and sufficient conditions for the existence of curves on a torus with specific pairwise intersections.
In this paper we study the stability and its trade-off with optimization error for stochastic gradient descent (SGD) algorithms in the pairwise learning setting. Pairwise learning refers to a learning task which involves a loss function depending on pairs of instances among which notable examples are bipartite ranking,…
We propose a robust estimator to improve maximum likelihood in probabilistic models.
problem Overfitting and sensitivity to noise in maximum likelihood estimation.
method Distributionally robust maximum likelihood estimator that minimizes worst-case expected log-loss.
result The robust estimator is statistically consistent and performs well in regression and classification tasks.
Paper characterizes and represents pairwise causal background knowledge for improved causal inference.
problem Improving causal inference by handling pairwise causal constraints.
method Graphical characterization, direct causal clause (DCC), unified representation, MPDAG, polynomial-time algorithms.
result Pairwise causal background knowledge uniquely decomposes into MPDAG and DCCs, improving causal effect identification.
Paper establishes statistical inference for pairwise comparison models.
problem Statistical inference for pairwise comparison models when the number of subjects diverges.
method Identifies Fisher information matrix as a weighted graph Laplacian for asymptotic normality.
result Near-optimal asymptotic normality result for maximum likelihood estimator.