The paper proves entropy power properties on Riemannian manifolds and Ricci flows.
problem Entropy power on Riemannian manifolds and Ricci flows.
method Proving concavity and convexity of Shannon entropy power for heat and conjugate heat equations on Riemannian manifolds and Ricci flows.
result Entropy power rigidity models on Einstein or quasi Einstein manifolds and shrinking Ricci solitons.
The paper proves properties of Renyi entropy power on Riemannian manifolds.
problem Properties of Renyi entropy power on Riemannian manifolds.
method Proof of concavity, rigidity models, Aronson-Benilan estimates, NIW formula, entropy isoperimetric inequality.
result Rigidity models and intrinsic relationships for Renyi entropy power.
Ancient flows by curvature powers in 2D have finite entropy.
problem Existence of non-homothetic ancient flows by powers of curvature in R2. method Determined Morse indices and kernels of the linearized operator of shrinkers. Constructed flows using unstable eigenfunctions.
result Existence of ancient flows with finite entropy.
The paper proves the concavity of entropy power for diffusion equations and applies it to new inequalities.
problem Proving concavity of p-Rényi entropy power for diffusion equations. method Analyzing positive solutions to doubly nonlinear diffusion equations and applying Lp-Sobolev and Gagliardo-Nirenberg inequalities. result New proofs and improvements of Lp-Gagliardo-Nirenberg inequalities. We find the entropy's infinite-size behavior in complex manifold sections.
problem Determining entropy behavior in complex manifold sections.
method Analyzing entanglement entropy in tensor powers of hermitian line bundles.
result Asymptotic formula for expected entanglement entropy.
The paper extends entropy formulas to super Ricci flows on metric measure spaces.
problem Entropy formulas for super Ricci flows on metric measure spaces.
method Extending Perelman's W-entropy and Shannon entropy power to super Ricci flows. result Equivalence between volume non-local collapsing property and lower boundedness of W-entropy on RCD(0,N) spaces. Entropy regularization improves power k-means for high-dimensional data.
problem Power k-means' tendency to get stuck in local minima and performance in high dimensions.
method Entropy regularization to learn feature relevance, combined with majorization-minimization algorithm.
result Consistent learning and scalable algorithm with closed-form updates and convergence guarantees.
We investigate entropy as a financial risk measure. Entropy explains the equity premium of securities and portfolios in a simpler way and, at the same time, with higher explanatory power than the beta parameter of the capital asset pricing model. For asset pricing we define the continuous entropy as an alternative meas…
A measure called relative cluster entropy distinguishes between correlated and uncorrelated sequences.
problem Distinguishing between sequences with different correlation degrees.
method Minimum relative entropy principle applied to cluster partitions of power-law correlated sequences.
result Optimal Hurst exponents are selected for market price series, indicating non-markovianity.
In this paper, we prove the concavity of p-entropy power of probability densities solving the p-heat equation on closed Riemannian manifold with nonnegative Ricci curvature. As applications, we give new proofs of Lp-Euclidean Nash inequality and Lp-Euclidean Logarithmic Sobolev inequality, moreover, an improv…
Paper proves generalized Talagrand inequality for Sinkhorn distance.
problem Proving a generalized Talagrand inequality for Sinkhorn distance.
method Using entropy power inequality and infinitesimal displacement convexity of optimal transport map.
result Extends previous results of Gaussian Talagrand inequality for Sinkhorn distance to strongly log-concave case.
The paper calculates bounds for risk metrics and entropies under partial information constraints.
problem Analyzing risk metrics and entropies for unimodal, symmetric distributions with limited information.
method Develops lower and upper bounds for worst-case distortion riskmetrics and weighted entropy for unimodal, symmetric distributions with known mean and variance.
result Sharp upper bounds for distortion riskmetrics and weighted entropy for symmetric distributions.
LLMs learn peaked distributions slowly due to power-law losses.
problem Slow convergence of loss in training large language models.
method Systematic analysis of toy models and empirical evaluation of LLMs.
result Power-law time scaling with an exponent of 1/3 for learning peaked distributions.
The paper proposes a method to identify high-quality financial patterns using entropy.
problem Extracting reliable short-term patterns from noisy financial data.
method Entropy-assisted framework for clustering and pruning patterns.
result High-quality patterns with low local entropy and historical profitability.
Improved reasoning model by sampling from power distribution without additional training.
problem Efficiently sampling from a sharpened distribution to improve reasoning models.
method Entropy-Cut Metropolis-Hastings algorithm that identifies key decision points for resampling.
result The method consistently improves reasoning models across various datasets.
The notion of utility maximising entropy (u-entropy) of a probability density, which was introduced and studied by Slomczynski and Zastawniak (Ann. Prob 32 (2004) 2261-2285, arXiv:math.PR/0410115 v1), is extended in two directions. First, the relative u-entropy of two probability measures in arbitrary probability space…
Paper characterizes embeddability of function spaces into Lp-type RKBS via metric entropy.
problem Characterizing embeddability of function spaces into Lp-type RKBS. method Establishes a connection between metric entropy growth and embeddability.
result A bound on metric entropy growth allows embedding into Lp-type RKBS. New methods for inferring, predicting, and estimating continuous-time, discrete-event processes.
problem Inferring, predicting, and estimating entropy rate of continuous-time, discrete-event processes.
method Bayesian structural inference extended with neural networks.
result Methods are competitive for prediction and entropy-rate estimation with state-of-the-art.
Researchers explore gauge freedom in entropies of q-Gaussian measures.
problem Exploring the gauge freedom of entropies in q-Gaussian measures. method Introducing a refined q-logarithmic function to demonstrate gauge freedom. result Different escort expectations can lead to the same entropy but different relative entropies.
On a Riemannian manifold, lower Ricci curvature bounds are known to be characterized by geodesic convexity properties of various entropies with respect to the Kantorovich-Rubinstein-Wasserstein square distance from optimal transportation. These notions also make sense in a (nonsmooth) metric measure setting, where they…
This article proposes a method to quantify the structure of a bipartite graph using a network entropy per link. The network entropy of a bipartite graph with random links is calculated both numerically and theoretically. As an application of the proposed method to analyze collective behavior, the affairs in which parti…
Coupled entropy corrects flaws in Tsallis entropy for complex systems.
problem Misinterpretation of generalized temperature and entropy.
method Derived from generalized Pareto and Student's t distributions.
result Provides balanced measure of uncertainty for complex systems.
The entropy density is an intuitive and powerful concept to study the complicated nonlinear processes derived from physical systems. We develop the minimum entropy density method (MEDM) to detect the structure scale of a given time series, which is defined as the scale in which the uncertainty is minimized, hence the p…
The paper characterizes curvature-dimension conditions and related inequalities on Riemannian manifolds.
problem Curvature-dimension conditions and related inequalities on Riemannian manifolds.
method Information-theoretic approach to study curvature-dimension condition, rigidity theorems, and entropy differential inequalities.
result Equivalence of curvature-dimension condition and entropy differential inequalities on Riemannian manifolds.
This work addresses two main issues of the standard Kernel Entropy Component Analysis (KECA) algorithm: the optimization of the kernel decomposition and the optimization of the Gaussian kernel parameter. KECA roughly reduces to a sorting of the importance of kernel eigenvectors by entropy instead of by variance as in K…
Behavior of the entropy numbers of classes of multivariate functions with mixed smoothness is studied here. This problem has a long history and some fundamental problems in the area are still open. The main goal of this paper is to develop a new method of proving the upper bounds for the entropy numbers. This method is…
The paper studies entropy calibration in language models and finds that miscalibration improves slowly with scale.
problem The problem is whether language model entropy calibration improves with scale and if it's possible to calibrate without reducing log loss.
method The authors study a simplified theoretical setting to characterize miscalibration scaling behavior and measure it empirically in language models ranging from 0.5B to 70B parameters.
result The observed scaling behavior of miscalibration is similar to theoretical predictions, indicating slow improvement with scale. The authors also prove theoretically that it is possible to reduce entropy while preserving log loss if access to a black box predicting future entropy is available.
New method speeds up lead-lag detection between asynchronous time series.
problem Slow inference of lead-lag networks between long time series.
method Derive asymptotic distribution of Transfer Entropy and introduce time-shifted time series.
result Statistically validated lead-lag networks between time series.
A new test assesses text similarity between two groups of documents.
problem Comparing similarity between two groups of documents.
method Neural network-based language models estimate entropy, and a test statistic derived from an estimation-and-inference framework is used.
result The proposed test maintains the nominal Type one error rate while offering greater power compared to existing methods.
The matrix-based Renyi's α-order entropy functional was recently introduced using the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the current theory in the matrix-based Renyi's α-order entropy functional only defines the entropy of a single…
Transformers model contextual relations using probabilistic measures, revealing their expressive power.
problem Lack of clear understanding of Transformer's ability to model contextual relations.
method Introduced a measure-theoretic framework connecting softmax attention and entropy-regularized optimal transport.
result Transformer architectures can approximate arbitrary contextual relations, and the choice of normalization affects how these relations are represented.
Information theory provides principled ways to analyze different inference and learning problems such as hypothesis testing, clustering, dimensionality reduction, classification, among others. However, the use of information theoretic quantities as test statistics, that is, as quantities obtained from empirical data, p…
This review explores entropy applications in data analysis and machine learning.
problem Characterizing probability mass distributions in data analysis and machine learning.
method Review of various entropy types and their applications.
result Entropy's versatility in data analysis and machine learning.
The ability of many powerful machine learning algorithms to deal with large data sets without compromise is often hampered by computationally expensive linear algebra tasks, of which calculating the log determinant is a canonical example. In this paper we demonstrate the optimality of Maximum Entropy methods in approxi…
Paper compares Rényi min-entropy vs Shannon entropy for feature selection in machine learning.
problem Feature selection in machine learning to improve model performance.
method Proposes an algorithm based on conditional Rényi min-entropy for feature selection, comparing it to Shannon-based mutual information.
result Rényi-based algorithm tends to outperform Shannon-based in real datasets.
Modernizes Thurston's proof of entropy theorem for traintrack maps.
problem Proving the entropy theorem for traintrack maps using Thurston's methods.
method Modernizes Thurston's original proof, fills gaps, and proves ergodicity.
result A cohesive proof of the traintrack theorem, including ergodicity.
The role of kernels is central to machine learning. Motivated by the importance of power-law distributions in statistical modeling, in this paper, we propose the notion of power-law kernels to investigate power-laws in learning problem. We propose two power-law kernels by generalizing Gaussian and Laplacian kernels. Th…
The Model-X knockoff procedure has recently emerged as a powerful approach for feature selection with statistical guarantees. The advantage of knockoff is that if we have a good model of the features X, then we can identify salient features without knowing anything about how the outcome Y depends on X. An important dra…
Entropy measure quantifies volatility correlation and risk diversity in asset portfolios.
problem Quantifying volatility correlation and risk diversity in asset portfolios.
method Kullback-Leibler cluster entropy DC[P∥Q] for empirical and model probability distributions of realized volatility. result Portfolio built on diversity indexes derived from Kullback-Leibler entropy measure of realized volatility exhibits better performance.
Develops a framework for analyzing multi-agent and many-body systems with feedback loops.
problem Optimal order of multi-agent and general many-body systems
method Derive macroscopic properties and optimal degree of order
result Optimal degree of order balances productivity, stability, and adaptability
Bayesian Monte-Carlo method assesses uncertainty in shear stress entropy models.
problem Uncertainty in evaluating shear stress entropy models remains an open question.
method Bayesian Monte-Carlo (BMC) uncertainty method to evaluate four entropy models.
result FOCB statistic index determines certainty of entropy models in shear stress estimation.
Paper proposes a Renyi entropy-based method for tuning hierarchical topic models.
problem Tuning hierarchical topic models, especially determining the number of topics at each level, is challenging.
method The paper introduces a Renyi entropy-based metric for quality assessment and a practical tuning concept.
result The proposed method can estimate the number of topics for two hierarchical levels in hARTM model.
An energy based approach for stabilizing a mechanical system has offered a simple yet powerful control scheme. However, since it does not impose such strong constraints on parameter space of the controller, finding appropriate parameter values for an optimal controller is known to be hard. This paper intends to generat…
The minimum error entropy (MEE) criterion has been verified as a powerful approach for non-Gaussian signal processing and robust machine learning. However, the implementation of MEE on robust classification is rather a vacancy in the literature. The original MEE only focuses on minimizing the Renyi's quadratic entropy …
Analyzes how BPE tokenisation affects corpus statistics and model entropy in transformer models.
problem Understanding how natural language properties relate to tokenisation schemes in transformer models.
method Analyzes Shannon entropy of corpora under Zipfian distribution, investigates BPE transformations, trains language models, and uses attention diagnostics.
result Transformer models trained on BPE-tokenised corpora increasingly agree with Zipfian predictions as BPE depth increases, indicating reduced local token dependencies.
MESMOC optimizes constrained multi-objective problems efficiently.
problem Constrained multi-objective optimization with expensive function evaluations.
method Max-value Entropy Search in the output space.
result MESMOC selects high-quality Pareto solutions efficiently.
New method calibrates reference distributions for bounded support.
problem Lack of principled method for bounded-support statistical reference distributions.
method Formulated maximum entropy on projective space of nonnegative measures.
result Prescribed acceptance region uniquely determines deformation parameter.
Proposes a cross entropy loss for better ranking algorithms.
problem Improving the theoretical understanding and performance of ranking algorithms.
method Introduces a cross entropy-based loss function that is a convex bound on NDCG and consistent with NDCG.
result Empirically, the proposed method outperforms existing algorithms in quality and robustness.