New algorithm estimates semi-continuous data density using entropy maximization.
problem Estimating density functions for semi-continuous data.
method Maximum entropy principle, requiring only constraint function samples.
result Estimate has significantly less bias compared to existing methods.
Generic low-entropy hypersurfaces in 4-6D flow with only generic singularities.
problem Analyzing mean curvature flow of low-entropy hypersurfaces.
method Proving flow encounters only generic singularities for specific entropy conditions.
result Proves flow encounters only generic singularities for low-entropy initial data.
Entropy data replaces classical charts for smooth manifolds.
problem Establishing smooth structures on topological manifolds.
method Using entropy data to define admissible coordinate functions and reconstruct smooth atlases.
result Entropy-smooth structures are equivalent to classical smooth structures and stable under perturbations.
This review explores entropy applications in data analysis and machine learning.
problem Characterizing probability mass distributions in data analysis and machine learning.
method Review of various entropy types and their applications.
result Entropy's versatility in data analysis and machine learning.
The article presents a new entropy model for assessing stock market interest.
problem Assessing investor interest and market sentiment in exchange-traded securities.
method Intrinsic entropy model using actual trading data, without exogenous factors.
result Empirical evidence supports the model's ability to predict trading activity.
Information theory provides principled ways to analyze different inference and learning problems such as hypothesis testing, clustering, dimensionality reduction, classification, among others. However, the use of information theoretic quantities as test statistics, that is, as quantities obtained from empirical data, p…
The paper presents a method to estimate joint interventional distributions from marginal interventional data.
problem Estimating joint interventional distributions from marginal interventional data.
method The paper extends the Causal Maximum Entropy method to use interventional data and employs Lagrange duality to prove the solution lies in the exponential family.
result The method allows for causal feature selection and inference of joint interventional distributions.
REMEDI improves neural entropy estimation across various tasks.
problem Challenges in estimating information theoretic quantities in high-dimensional data.
method Combines minimization of cross-entropy with estimation of deviation from data density.
result Improves accuracy in entropy estimation on synthetic and natural data.
In this paper, we develop the notion of entropy for uniform hypergraphs via tensor theory. We employ the probability distribution of the generalized singular values, calculated from the higher-order singular value decomposition of the Laplacian tensors, to fit into the Shannon entropy formula. We show that this tensor …
Structured entropy improves classification performance on structured targets.
problem Cross-entropy loss fails to account for target variable structure.
method Proposes structured entropy, a generalization of entropy using random partitions.
result Structured cross-entropy loss yields better results on classification problems with known structure.
The study evaluates different probability models for uncertainty visualization using entropy calculations.
problem Choosing the right probability model affects memory use, run time, and accuracy in uncertainty visualization.
method Entropy calculation on ensemble data to compare various probability models (uniform, Gaussian, histogram, quantile).
result Models matching the ensemble data distribution have the lowest entropy, indicating better accuracy.
Low-entropy surfaces can be flowed into spheres and cylinders.
problem Proving mean curvature flow for low-entropy hypersurfaces.
method Low-entropy density drop argument and recent work on hypersurfaces.
result Closed hypersurfaces with entropy ≤ 2 can be flowed into spherical and cylindrical shapes.
Causal discovery is a fundamental problem in statistics and has wide applications in different fields. Transfer Entropy (TE) is a important notion defined for measuring causality, which is essentially conditional Mutual Information (MI). Copula Entropy (CE) is a theory on measurement of statistical independence and is …
We consider the problem of identifying the causal direction between two discrete random variables using observational data. Unlike previous work, we keep the most general functional model but make an assumption on the unobserved exogenous variable: Inspired by Occam's razor, we assume that the exogenous variable is sim…
Quantum machine learning uses quantum cross entropy to minimize loss, but measurement loss affects this process.
problem Quantum machine learning's loss minimization through cross entropy is affected by measurement outcomes.
method Defined quantum cross entropy, proved its lower bounds, and investigated its relation to quantum fidelity and likelihood.
result Quantum cross entropy is lower-bounded by negative log-likelihood when derived from quantum data, but measurement outcomes can cause loss.
We present a procedure for effective estimation of entropy and mutual information from small-sample data, and apply it to the problem of inferring high-dimensional gene association networks. Specifically, we develop a James-Stein-type shrinkage estimator, resulting in a procedure that is highly efficient statistically …
Entropy corrections improve GBM's predictive accuracy for non-log-normal distributions.
problem Log-normal distribution limitations in GBM predictions.
method Entropy corrections to geometric Brownian motion (GBM).
result Improved predictive accuracy for non-log-normal distributions.
Entropy rate of sequential data-streams naturally quantifies the complexity of the generative process. Thus entropy rate fluctuations could be used as a tool to recognize dynamical perturbations in signal sources, and could potentially be carried out without explicit background noise characterization. However, state of…
We study the problem of discovering the simplest latent variable that can make two observed discrete variables conditionally independent. The minimum entropy required for such a latent is known as common entropy in information theory. We extend this notion to Renyi common entropy by minimizing the Renyi entropy of the …
New method synthesizes and analyzes probability measures using entropy-regularized optimal transport.
problem Synthesize and analyze probability measures with entropy-regularized optimal transport.
method Entropy-regularized Wasserstein-2 cost and Sinkhorn divergence for synthesis and analysis.
result Computed barycentric coefficients and their stability for classification of corrupted point cloud data.
Persistent entropy detects phase transitions in complex systems.
problem Detecting phase transitions in complex systems.
method Established a general theorem for persistent entropy to reliably detect phase transitions, introduced operational framework for finite-time computations.
result Persistent entropy exhibits an asymptotically non-vanishing gap across phases, robust numerical signatures across experiments.
Efficiently constructs sparse ROMs for high-dimensional data using causation entropy.
problem Creating effective reduced-order models for high-dimensional dynamical data.
method Uses causation entropy to identify important terms and construct ROMs with varying sparsity.
result Demonstrates the effectiveness of causation entropy in constructing sparse ROMs for chaotic systems with skewed statistics.
MGD combines maximum entropy and diffusion methods for efficient sampling.
problem Generating samples from limited information in high dimensions.
method Moment Guided Diffusion (MGD) using stochastic differential equations.
result MGD efficiently samples maximum entropy distributions in finite time.
EDD uses entropy of distance distributions to cluster unlabeled data.
problem Challenges in clustering unlabeled high-dimensional data.
method EDD employs Shannon entropy to quantify distance distribution peaks.
result EDD detects varying degrees of clustering sensitivity.
Entropy asymmetry affects regularization in ERM, leading to biased solutions.
problem Analyzing the impact of relative entropy asymmetry in ERM regularization.
method Examined Type-I and Type-II ERM-RER, comparing their solutions and properties.
result Type-II ERM-RER regularization introduces a strong bias against training data.
EntProp increases entropy of clean samples to generate out-of-distribution data for better DNN performance.
problem Improving deep neural networks' accuracy and robustness to out-of-distribution data.
method High entropy propagation using data augmentation and free adversarial training.
result EntProp achieves higher standard accuracy and robustness with lower training cost.
Study examines how war impacts financial entropy.
problem Analyzing sudden events' impact on financial markets.
method Investigates entropy changes in Polish stock market data.
result Entropy analysis reveals market changes due to extreme events.
Review of correlation-based financial networks and entropy measures.
problem Understanding the dynamics of financial markets through correlation networks.
method Analysis of empirical correlation matrices and entropy measures.
result Entropy measures help in continuous monitoring of financial networks.
The paper corrects biases in estimating intrinsic dimension and differential entropy.
problem Systematic bias in estimating intrinsic dimension and differential entropy.
method A bias-corrected estimator for both measures is proposed, highlighting shared steps and useful consequences.
result Simultaneous estimation of differential entropy and intrinsic dimension provides complementary perspectives on underlying manifolds.
The construction of efficient and effective decision trees remains a key topic in machine learning because of their simplicity and flexibility. A lot of heuristic algorithms have been proposed to construct near-optimal decision trees. ID3, C4.5 and CART are classical decision tree algorithms and the split criteria they…
New method estimates robust multi-period portfolios using entropy.
problem Lack of general agreement on building robust multi-period portfolios.
method Detrended cluster entropy approach to estimate portfolio weights.
result Portfolio weights are estimated reliably from real-world data at varying time horizons.
Data transformation, e.g. feature transformation and selection, is an integral part of any machine learning procedure. In this paper we introduce an information-theoretic model and tools to assess the quality of data transformations in machine learning tasks. In an unsupervised fashion, we analyze the transfer of infor…
Entropy-based model for hierarchical learning from multiscale data.
problem Learning from data with auxiliary information and multiscale target functions.
method Entropy-based hierarchical learning model with multiscale entropies.
result Entropy-based model yields stronger guarantees than uniform convergence bounds.
We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier. Entropy-SGD works by optimizing the bound's p…
New bound limits generalization gap for large models, independent of model complexity.
problem Understanding generalization gap in large-scale machine learning models.
method Established a model-independent upper bound for generalization gap using Rényi entropy.
result Generalization gap can be maintained with arbitrarily large models if data entropy is sufficient.
The matrix-based Renyi's α-order entropy functional was recently introduced using the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the current theory in the matrix-based Renyi's α-order entropy functional only defines the entropy of a single…
The intrinsic entropy model accurately estimates stock market volatility.
problem Accurately estimating historical volatility of stock market indices.
method Incorporates traded volumes alongside OHLC prices in daily data.
result Intrinsic entropy model delivers reliable estimates with lower coefficient of variation.
Study of focal-entropy for class-imbalanced classification.
problem Understanding the focal-loss in class-imbalanced settings.
method Distributional viewpoint and information-theoretic analysis of focal-entropy.
result Focal-entropy minimizer exists and departs from data distribution.
Generative models' ELBOs converge to entropy sums, proving for various models.
problem Proving convergence of ELBOs to entropy sums for various generative models.
method Proofs for individual models under realistic conditions.
result ELBOs of various generative models converge to entropy sums at all stationary points.
The paper proposes a method to identify high-quality financial patterns using entropy.
problem Extracting reliable short-term patterns from noisy financial data.
method Entropy-assisted framework for clustering and pruning patterns.
result High-quality patterns with low local entropy and historical profitability.
The need to estimate smooth probability distributions (a.k.a. probability densities) from finite sampled data is ubiquitous in science. Many approaches to this problem have been described, but none is yet regarded as providing a definitive solution. Maximum entropy estimation and Bayesian field theory are two such appr…
Paper proposes a new uncertainty measure for active learning in neural networks.
problem Efficiently selecting informative data points in limited labeled data scenarios.
method BalEntAcq, a new uncertainty measure based on balanced entropy, approximated by Beta distributions.
result BalEntAcq outperforms existing uncertainty measures in active learning.
Gathering the most information by picking the least amount of data is a common task in experimental design or when exploring an unknown environment in reinforcement learning and robotics. A widely used measure for quantifying the information contained in some distribution of interest is its entropy. Greedily minimizing…
New analysis shows entropy term cancels out in likelihood-based OOD detection.
problem Curious likelihood values for out-of-distribution data.
method Decomposed average likelihood into KL divergence and entropy terms.
result Entropy term explains OOD behaviour and cancels out in expectation.
The paper studies the growth of closed geodesics on hyperbolic surface amalgams.
problem Understanding the growth of closed geodesics on hyperbolic surface amalgams.
method Analyzing topological and volume entropies, and their dependence on geometric data.
result Entropy can increase exponentially with pasting length in the absence of a lower bound on the systole.
Study detects anomalies in financial markets using GNN and nonextensive entropy.
problem Detecting anomalies in global financial markets with many correlated assets.
method Used Graph Neural Networks (GNN) with nonextensive entropy to measure uncertainty.
result Anomalies are statistically different for nonextensive entropy parameters before, during, and after a crisis.
ED-VAE improves VAEs by explicitly including entropy components in ELBO.
problem Limitations of traditional VAEs with ELBO in generating high-quality samples and interpreting latent spaces.
method Introduces ED-VAE, a re-formulation of ELBO that includes entropy and cross-entropy components.
result Significantly enhances model flexibility and improves interpretability and generative performance.
In this paper, we propose a general framework to learn a robust large-margin binary classifier when corrupt measurements, called anomalies, caused by sensor failure might be present in the training set. The goal is to minimize the generalization error of the classifier on non-corrupted measurements while controlling th…