The paper proves properties of Renyi entropy power on Riemannian manifolds.
problem Properties of Renyi entropy power on Riemannian manifolds.
method Proof of concavity, rigidity models, Aronson-Benilan estimates, NIW formula, entropy isoperimetric inequality.
result Rigidity models and intrinsic relationships for Renyi entropy power.
Paper compares Rényi min-entropy vs Shannon entropy for feature selection in machine learning.
problem Feature selection in machine learning to improve model performance.
method Proposes an algorithm based on conditional Rényi min-entropy for feature selection, comparing it to Shannon-based mutual information.
result Rényi-based algorithm tends to outperform Shannon-based in real datasets.
Paper extends matrix-based Renyi's α-order entropy to multivariate data.
problem Estimating multivariate information quantities like joint entropy and interactive information.
method Define matrix-based Renyi's α-order joint entropy for multiple variables.
result Eases estimation of multivariate information quantities.
Accounting for the non-normality of asset returns remains challenging in robust portfolio optimization. In this article, we tackle this problem by assessing the risk of the portfolio through the "amount of randomness" conveyed by its returns. We achieve this by using an objective function that relies on the exponential…
The paper studies practical estimation and interpretation of Rényi transfer entropy.
problem Challenges in accurately estimating and interpreting Rényi transfer entropy.
method Systematic study of k-nearest neighbor estimator for Rényi entropy and transfer entropy.
result Effective estimates of effective Rényi transfer entropy can accurately capture directional information flow.
This is full length article (draft version) where problem number of topics in Topic Modeling is discussed. We proposed idea that Renyi and Tsallis entropy can be used for identification of optimal number in large textual collections. We also report results of numerical experiments of Semantic stability for 4 topic mode…
Paper proposes a Renyi entropy-based method for tuning hierarchical topic models.
problem Tuning hierarchical topic models, especially determining the number of topics at each level, is challenging.
method The paper introduces a Renyi entropy-based metric for quality assessment and a practical tuning concept.
result The proposed method can estimate the number of topics for two hierarchical levels in hARTM model.
The null energy condition is characterized via convexity of entropy in Lorentzian manifolds.
problem Characterizing the null energy condition in Lorentzian manifolds.
method Characterization via convexity of the relative entropy along displacement interpolations on null hypersurfaces.
result The null energy condition is characterized in terms of convexity of the relative entropy.
Novel kernelized Renyi's entropy improves deep learning generalization bounds.
problem Improving generalization bounds for deep learning algorithms.
method Kernelized Renyi's entropy, a new information theoretical measure.
result Theoretical bounds are tighter than current SOTA results.
Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its or…
Efficient approximations reduce computation of matrix-based Renyi's entropy.
problem High computational complexity of matrix-based Renyi's entropy.
method Taylor, Chebyshev, and Lanczos approximations to reduce complexity.
result Reduced complexity to significantly less than O(n2) with negligible accuracy loss. The paper solves portfolio selection using Rényi divergence and optimization.
problem Single-period portfolio selection under CRRA utility.
method Information-theoretic lens, Rényi divergence, Rényi entropy, Blahut-Arimoto-style alternating optimization.
result CRRA portfolio selection is equivalent to a Rényi information-projection problem.
This paper optimizes matrix-based Renyi's entropy computation for large datasets.
problem Efficiently calculating matrix-based Renyi's entropy for large-scale applications.
method Develops randomized approximations for matrix-based Renyi's entropy with arbitrary α orders.
result Achieves a significant reduction in time complexity from O(n^3) to O(n^2sm), where s, m << n.
New bound limits generalization gap for large models, independent of model complexity.
problem Understanding generalization gap in large-scale machine learning models.
method Established a model-independent upper bound for generalization gap using Rényi entropy.
result Generalization gap can be maintained with arbitrarily large models if data entropy is sufficient.
We extend common entropy concept and propose algorithms to distinguish causation from correlation.
problem Discovering the simplest latent variable for conditional independence of observed variables.
method Renyi common entropy, iterative algorithm, constraint-based methods modification.
result Improved constraint-based methods for causal inference in small samples.
New algorithms improve deep RL with entropy-based action selection and environment exploration.
problem Improving deep reinforcement learning algorithms for better sample efficiency and effectiveness.
method Proposes Tsallis entropy Actor-Critic (TAC), Renyi entropy Actor-Critic (RAC), and Ensemble Actor-Critic (EAC) algorithms.
result Empirically, TAC, RAC, and EAC outperform SAC and other algorithms in benchmark control tasks.
In this paper, we quantify the statistical coherence between financial time series by means of the Renyi entropy. With the help of Campbell's coding theorem we show that the Renyi entropy selectively emphasizes only certain sectors of the underlying empirical distribution while strongly suppressing others. This accentu…
The paper proves the concavity of entropy power for diffusion equations and applies it to new inequalities.
problem Proving concavity of p-Rényi entropy power for diffusion equations. method Analyzing positive solutions to doubly nonlinear diffusion equations and applying Lp-Sobolev and Gagliardo-Nirenberg inequalities. result New proofs and improvements of Lp-Gagliardo-Nirenberg inequalities. A new method for VAEs improves latent space disentanglement without violating probability laws.
problem Improving latent space disentanglement in VAEs without violating probability laws.
method Developed a Renyi VAE with a conditional distribution not learned, using Singular Value Decomposition for evaluation.
result Improved latent space disentanglement without violating probability laws.
Maximizes Rényi entropy for efficient exploration in reward-free RL.
problem Challenges of exploration in reward-free reinforcement learning.
method Maximizes Rényi entropy over state-action space in exploration phase; uses batch RL for planning phase.
result Effective and sample-efficient exploration leading to superior policies.
Proposes an active RBI framework using Rényi information measures for more informed decision-making.
problem Optimal latent variable estimates in real-time settings with streaming noisy observations.
method Unified inference and query selection steps through Rényi entropy and α-divergence; new objective called Momentum for exploration.
result Analytically demonstrates superior performance compared to conventional methods like mutual information.
The paper proposes a method to adapt models from source to target domains by calibrating their predictive uncertainties.
problem Inferring class labels for unlabeled target domain given a related labeled source dataset.
method The approach involves calibrating predictive uncertainties quantified as Renyi entropy, using variational Bayes learning and sample variance regularization.
result The proposed method effectively adapts models across three domain-adaptation tasks.
Paper introduces RCaI, a risk-sensitive control method using Rényi divergence.
problem Risk-sensitive control in reinforcement learning.
method RCaI extends CaI using Rényi divergence variational inference.
result Risk-sensitive optimal policy can be obtained by solving a soft Bellman equation.
Study introduces new curvature conditions for Lorentzian spaces using Rényi entropy.
problem Developing synthetic curvature conditions for Lorentzian spaces.
method Introducing timelike curvature-dimension conditions and measure-contraction properties using Rényi entropy.
result Equivalence of new curvature conditions to entropic counterparts.
New IP analysis for deep neural networks using Rényi's entropy and tensor kernels.
problem Estimating mutual information in high-dimensional hidden layers of deep neural networks.
method Matrix-based Rényi's entropy coupled with tensor kernels for convolutional layers.
result First comprehensive IP analysis of large-scale DNNs and CNNs.
We prove local Poincaré inequalities under various curvature-dimension conditions which are stable under the measured Gromov-Hausdorff convergence. The first class of spaces we consider is that of weak CD(K,N) spaces as defined by Lott and Villani. The second class of spaces we study consists of spaces where we have a …
With the help of transfer entropy, we analyze information flows between communities of complex networks. We show that the transfer entropy provides a coherent description of interactions between communities, including non-linear interactions. To put some flesh on the bare bones, we analyze transfer entropies between co…
The paper analyzes ESS metrics and their connections to entropy families.
problem Improving the accuracy of effective sample size approximations.
method Examined Rényi and Tsallis entropy families and their relationship to ESS.
result Proved that ESS functions in the Huggins-Roy family meet theoretical conditions.
The paper characterizes curvature-dimension conditions and related inequalities on Riemannian manifolds.
problem Curvature-dimension conditions and related inequalities on Riemannian manifolds.
method Information-theoretic approach to study curvature-dimension condition, rigidity theorems, and entropy differential inequalities.
result Equivalence of curvature-dimension condition and entropy differential inequalities on Riemannian manifolds.
All you need is log
problem Comparing multiple probability distributions
method Characterizing multi-distribution Rényi divergences
result Found a canonical multi-distribution Rényi calculus
A new family of multi-distribution divergences is characterized for fairness and other problems.
problem Comparing more than two probability distributions in a consistent way.
method Characterized a new family of multi-way coincidence divergences.
result The new family of multi-distribution Rényi divergences is necessary and arises from several independent routes.
We present simple and computationally efficient nonparametric estimators of Rényi entropy and mutual information based on an i.i.d. sample drawn from an unknown, absolutely continuous distribution over Rd. The estimators are calculated as the sum of p-th powers of the Euclidean lengths of the edges of the `genera…
The paper improves semi-supervised learning using f-divergences and α-Rényi divergences.
problem Improving semi-supervised learning with noisy pseudo-labels.
method Inspired by f-divergences and α-Rényi divergences, the paper develops new empirical risk functions and regularization techniques. result The new methods show better performance than traditional self-training methods, especially in noisy pseudo-label scenarios.
Introduces TSI, a variance-based measure for persistence barcodes.
problem Capturing structural variability in persistence barcodes.
method Variance-based scalar measure, TSI, and complementary TSigI.
result TSI captures structural variability complementary to entropy.
In this paper we consider the space of those probability distributions which maximize the q-Rényi entropy. These distributions have the same parameter space for every q, and in the q=1 case these are the normal distributions. Some methods to endow this parameter space with Riemannian metric is presented: the seco…
Bayesian Monte-Carlo method assesses uncertainty in shear stress entropy models.
problem Uncertainty in evaluating shear stress entropy models remains an open question.
method Bayesian Monte-Carlo (BMC) uncertainty method to evaluate four entropy models.
result FOCB statistic index determines certainty of entropy models in shear stress estimation.
In the framework of Multifractal Diffusion Entropy Analysis we propose a method for choosing an optimal bin-width in histograms generated from underlying probability distributions of interest. The method presented uses techniques of Rényi's entropy and the mean squared error analysis to discuss the conditions under whi…
New GAN loss functions improve image generation quality and stability.
problem Improving the performance of GANs in generating high-quality images.
method Introducing least kth-order GAN (LkGAN) and Rényi-centric GAN loss functions. result The proposed loss functions lead to better image quality and stability.
New estimator uses k-nearest neighbor distances for density functionals.
problem Estimating general density functionals from data.
method Asymptotically unbiased estimator using inverse Laplace transform.
result Established L2-consistency and mean squared error convergence. The notion of utility maximising entropy (u-entropy) of a probability density, which was introduced and studied by Slomczynski and Zastawniak (Ann. Prob 32 (2004) 2261-2285, arXiv:math.PR/0410115 v1), is extended in two directions. First, the relative u-entropy of two probability measures in arbitrary probability space…
Generative models' evaluation scores can be misleading, leading to inflated grades.
problem Misleading evaluation scores for generative models.
method Analyzed and compared various scores for evaluating synthetic vs. ground-truth data.
result The Eden score avoids grade inflation and better aligns with human perception.
Long memory and volatility clustering are two stylized facts frequently related to financial markets. Traditionally, these phenomena have been studied based on conditionally heteroscedastic models like ARCH, GARCH, IGARCH and FIGARCH, inter alia. One advantage of these models is their ability to capture nonlinear dynam…
We consider the minimum error entropy (MEE) criterion and an empirical risk minimization learning algorithm in a regression setting. A learning theory approach is presented for this MEE algorithm and explicit error bounds are provided in terms of the approximation ability and capacity of the involved hypothesis space w…
SMOTE is one of the oversampling techniques for balancing the datasets and it is considered as a pre-processing step in learning algorithms. In this paper, four new enhanced SMOTE are proposed that include an improved version of KNN in which the attribute weights are defined by mutual information firstly and then they …
The paper generalizes Bayesian Cramér-Rao inequality using information geometry of relative α-entropy.
problem Establishing a lower bound for the variance of an unbiased estimator for the α-escort distribution.
method Proposes a general Riemannian metric based on relative α-entropy to derive a generalized Bayesian Cramér-Rao inequality.
result Establishes a lower bound for the variance of an unbiased estimator for the α-escort distribution.
Suggests stopping criteria for feature selection using mutual information.
problem Automatic determination of optimal feature subset size and stopping criterion.
method Monitoring conditional mutual information (CMI) among groups of variables using Renyi's α-entropy.
result Easy to implement stopping criteria for feature selection.
Paper extends information theory for efficient probabilistic modeling.
problem Efficient and data-efficient non-parametric density estimation.
method Structured generative model (SGM) using Rényi's information.
result SGM improves mutual information estimation and generative adversarial networks.
New framework controls generalization for heavy-tailed data in RLHF and SGLD.
problem Heavy-tailed data in modern learning pipelines.
method Tail-dependent information-theoretic framework for sub-Weibull data.
result Sharp generalization bounds for heavy-tailed data.