New methods for inferring, predicting, and estimating continuous-time, discrete-event processes.
problem Inferring, predicting, and estimating entropy rate of continuous-time, discrete-event processes.
method Bayesian structural inference extended with neural networks.
result Methods are competitive for prediction and entropy-rate estimation with state-of-the-art.
This paper improves entropy bounds for ranking time-series complexity.
problem Ranking the complexity of time series processes.
method Building on information theoretic bounds, the paper improves the upper bound of conditional differential entropy using Hadamard's inequality and covariance matrix properties.
result The improved bounds can be used to rank the complexity of time series processes.
Entropy corrections improve GBM's predictive accuracy for non-log-normal distributions.
problem Log-normal distribution limitations in GBM predictions.
method Entropy corrections to geometric Brownian motion (GBM).
result Improved predictive accuracy for non-log-normal distributions.
Unified approach to structured prediction combining entropy regularization and neuro-symbolic logic.
problem Structured prediction challenges due to large output spaces and insufficient labeled data.
method Neuro-symbolic entropy regularization loss that restricts entropy regularization to valid structures.
result Models predict more accurately and are more likely to be valid.
Study predicts price predictability in ultra-high frequency financial data using entropy tests.
problem Tackles predictability of ultra-high frequency financial data.
method Develops statistical tests based on Shannon entropy and Kullback-Leibler divergence to analyze predictability.
result Degree of randomness increases with aggregation level in transaction time.
News novelty predicts negative stock market returns.
problem Negative stock market returns due to increased news novelty.
method Quantified news novelty using entropy measure from recurrent neural network applied to a large news corpus.
result Entropy exposure carries a negative risk premium, indicating that assets positively correlated with entropy hedge aggregate news risk.
Researchers calculate Shannon entropy rates of hidden Markov processes efficiently.
problem No finite expression exists for Shannon entropy rates of hidden Markov processes.
method Developed an efficient method to calculate entropy rates and identify minimal predictive features.
result Entropy rates can be accurately calculated for hidden Markov processes.
Proposes a new method to measure epistemic uncertainty in Bayesian neural networks.
problem Measuring epistemic uncertainty in Bayesian neural networks for out-of-distribution detection.
method Proposes measuring disagreement between logits and their pre-softmax counterparts as an epistemic uncertainty measure.
result Proposed epistemic uncertainty scores outperform mutual information and equal predictive entropy performance.
ProSelfLC improves robustness of deep neural networks by automatically deciding trust in predictions.
problem Training robust deep neural networks requires addressing issues like label noise and low entropy predictions.
method ProSelfLC progressively increases trust in predicted labels over time, considering entropy and learning time.
result ProSelfLC demonstrates improved robustness in both clean and noisy settings through empirical validation.
Bayesian Entropy Neural Networks enforce constraints on deep learning predictions.
problem Deep learning models lack well-defined constraints in their outputs.
method Bayesian Entropy Neural Networks (BENN) using Maximum Entropy principles and the method of multipliers.
result BENN improves model robustness and reliability across various applications.
In this work, we study the credit assignment problem in reward augmented maximum likelihood (RAML) learning, and establish a theoretical equivalence between the token-level counterpart of RAML and the entropy regularized reinforcement learning. Inspired by the connection, we propose two sequence prediction algorithms, …
This study proposes hidden state curiosity to enhance RL models' resilience against noise.
problem Curiosity traps in RL models distract agents from discovering novel experiences.
method Proposed hidden state curiosity based on the Free Energy Principle to reward agents for KL divergence between predictive priors and posteriors.
result Agents with hidden state curiosity are more resilient against curiosity traps compared to those with prediction error curiosity.
A new reinforcement learning method reduces action complexity for robust control.
problem Deep reinforcement learning's susceptibility to spurious correlations.
method Minimizing trajectory entropy to encourage simple, predictable actions.
result Trajectory Entropy Reinforcement Learning achieves superior performance and robustness.
Order-flow entropy predicts price magnitude without directionality.
problem Predicting price magnitude in financial markets.
method Real-time order-flow entropy computed from a 15-state Markov transition matrix.
result Order-flow entropy predicts the magnitude of intraday returns with high accuracy.
We propose a novel information-theoretic approach for Bayesian optimization called Predictive Entropy Search (PES). At each iteration, PES selects the next evaluation point that maximizes the expected information gained with respect to the global maximum. PES codifies this intractable acquisition function in terms of t…
New entropy functionals for curved spaces help predict shape behavior.
problem Understanding entropy behavior in curved spaces.
method Introduced new entropy functionals for submanifolds of Cartan-Hadamard manifolds.
result Obtained sharp lower bounds on these entropies for certain closed hypersurfaces and observed a novel rigidity phenomenon.
The paper extends explainability methods to uncertainty-aware models, revealing feature impacts on predictive entropy and likelihood.
problem Understanding the factors contributing to uncertainty in probabilistic models.
method Adapting permutation feature importance, partial dependence plots, and individual conditional expectation plots to measure feature impacts on predictive entropy and likelihood.
result Novel insights into model behaviour and feature impacts on uncertainty are obtained.
The paper proposes a method to identify high-quality financial patterns using entropy.
problem Extracting reliable short-term patterns from noisy financial data.
method Entropy-assisted framework for clustering and pruning patterns.
result High-quality patterns with low local entropy and historical profitability.
Paper proves Jeffrey's update rule minimizes relative entropy.
problem Improving Bayesian learning algorithms.
method More concise proof of Jeffrey's update rule.
result Jeffrey's update rule reduces relative entropy.
This paper uses SampEn to measure and predict oil price volatility.
problem Measuring and predicting volatility in international oil prices.
method Sample Entropy (SampEn) compared with standard deviation; machine learning algorithms used.
result SampEn effectively predicts traditional volatility measures, especially during financial crises.
ALIEN improves uncertainty estimation of language models by refining entropy-based methods.
problem Overconfidence in uncertainty estimation for language models, especially for difficult inputs.
method ALIEN refines entropy-based uncertainty by aligning it with prediction reliability, using a lightweight uncertainty head.
result ALIEN consistently outperforms strong baselines in detecting incorrect predictions and achieving the lowest calibration error.
Examines predictability and complexity of economic time series using symbolic dynamics and entropy.
problem Understanding the predictability and complexity of economic time series.
method Symbolic dynamics and Information theory (entropy and uncertainty).
result Economic time series are complex and can be expressed in terms of information production.
Analyzing mobility behavior of users is extremely useful to create or improve existing services. Several research works have been done in order to study mobility behavior of users that mainly use users' significant locations. However, these existing analysis are extremely intrusive because they require the knowledge of…
We show that the predictability of letters in written English texts depends strongly on their position in the word. The first letters are usually the least easy to predict. This agrees with the intuitive notion that words are well defined subunits in written languages, with much weaker correlations across these units t…
Study improves materials discovery for high-entropy alloys using sparse linear models.
problem Inefficient materials discovery due to combinatorial explosion in alloy compositions.
method Sparse mixed linear modeling with anchor-based guidance for feature selection and prediction.
result Developed a method that balances predictive performance and interpretability for materials discovery.
Paper compares Rényi min-entropy vs Shannon entropy for feature selection in machine learning.
problem Feature selection in machine learning to improve model performance.
method Proposes an algorithm based on conditional Rényi min-entropy for feature selection, comparing it to Shannon-based mutual information.
result Rényi-based algorithm tends to outperform Shannon-based in real datasets.
Empirical evidence suggests that even the most competitive markets are not strictly efficient. Price histories can be used to predict near future returns with a probability better than random chance. Many markets can be considered as {\it favorable games}, in the sense that there is a small probabilistic edge that smar…
Tent adapts models during testing by minimizing entropy of predictions.
problem Adapting models to new data during testing with limited information.
method Test entropy minimization (tent) and online channel-wise affine transformations.
result Reduces generalization error on various datasets and benchmarks.
In this paper, we obtain generic bounds on the variances of estimation and prediction errors in time series analysis via an information-theoretic approach. It is seen in general that the error bounds are determined by the conditional entropy of the data point to be estimated or predicted given the side information or p…
JES optimizes expensive functions by considering joint entropy over input and output spaces.
problem Optimizing expensive functions with limited evaluations.
method Joint Entropy Search (JES) considers joint entropy over input and output spaces.
result JES outperforms other information-theoretic methods in Bayesian optimization.
Entropy Search (ES) and Predictive Entropy Search (PES) are popular and empirically successful Bayesian Optimization techniques. Both rely on a compelling information-theoretic motivation, and maximize the information gained about the argmax of the unknown function; yet, both are plagued by the expensive computatio…
Predicts trainability of deep neural networks using reconstruction entropy.
problem Predicting the initial conditions for trainability of deep neural networks.
method Cascade of auxiliary networks to reconstruct input from activation layers, computing relative entropy.
result Predicts trainability of deep feedforward networks on various datasets with a single epoch.
Paper proposes AXE loss for non-autoregressive machine translation, improving performance.
problem Challenges in training non-autoregressive models due to lack of autoregressive factors and cross entropy loss penalties.
method Proposes aligned cross entropy (AXE) loss function using a differentiable dynamic program for better word order alignment.
result AXE-based training improves performance on major WMT benchmarks and sets a new state of the art for non-autoregressive models.
This paper studies Fenchel-Young losses, a generic way to construct convex loss functions from a regularization function. We analyze their properties in depth, showing that they unify many well-known loss functions and allow to create useful new ones easily. Fenchel-Young losses constructed from a generalized entropy, …
The paper proposes using entropy to assess model confidence.
problem Discarded information in probability distributions hinders model confidence assessment.
method Entropy methods applied to assess model confidence.
result Theoretical explanation of confidence degradation in Naive Bayes model.
Gradient descent biases linear models in next-token prediction towards data entropy.
problem Optimization bias in next-token prediction models.
method Analysis of gradient descent on linear models with sparse conditional distributions.
result Gradient descent selects parameters that equate token logits differences to log-odds in the data subspace.
A recent proposal by Ryu and Takayanagi for a holographic interpretation of entanglement entropy in conformal field theories dual to supergravity on anti-de Sitter (adS) is generalized to include entanglement entropy of black holes living on the boundary of adS. The generalized proposal is verified in boundary dimensio…
Language models can predict numeric values as strings.
problem Regression tasks with numeric predictions.
method Causal sequence decoding models trained for next-token prediction.
result Decoder-based heads perform as well as standard heads in numeric regression tasks.
This research improves neural network uncertainty estimates and reliability.
problem Lack of inherent uncertainty estimates and variability in softmax scores.
method Ensemble-based Dirichlet modeling with method of moments estimator.
result Improved stability and predictive uncertainty estimates.
In our previous studies we have investigated the structural complexity of time series describing stock returns on New York's and Warsaw's stock exchanges, by employing two estimators of Shannon's entropy rate based on Lempel-Ziv and Context Tree Weighting algorithms, which were originally used for data compression. Suc…
Estimate relaxation times in nonextensive systems using gradient flow for Tsallis entropy maximization.
problem Estimating relaxation times in financial market dynamics.
method Developing a method using EGF for maximizing Tsallis entropy.
result Longer relaxation times for nonextensive systems compared to Shannon entropy.
New methods improve Bayesian inference and decision-making in online learning.
problem Current Bayesian deep learning does not fully utilize joint predictives for sequential decision-making.
method Proposes new evaluation settings for active learning and active sampling, focusing on marginal and joint cross-entropies.
result Initial experiments suggest challenges in applying current BDL inference techniques in high-dimensional spaces.
Analyzes how BPE tokenisation affects corpus statistics and model entropy in transformer models.
problem Understanding how natural language properties relate to tokenisation schemes in transformer models.
method Analyzes Shannon entropy of corpora under Zipfian distribution, investigates BPE transformations, trains language models, and uses attention diagnostics.
result Transformer models trained on BPE-tokenised corpora increasingly agree with Zipfian predictions as BPE depth increases, indicating reduced local token dependencies.
Bitcoin's price direction is better predicted without additional drivers during high volatility.
problem Predicting Bitcoin's price direction using various determinants.
method Continuous local transfer entropy for feature selection and deep learning classification model.
result Bitcoin's price direction can be better predicted without additional drivers during high volatility.
Improves adversarial robustness of DEQ models by regulating neural dynamics.
problem Limited adversarial robustness of DEQ models.
method Interprets DEQs as neural dynamics, uses entropy reduction and random intermediate states.
result Significantly increases adversarial robustness of DEQ models.
A new method reduces compounding errors in model-based reinforcement learning.
problem Compounding errors in long horizon predictions from model-based reinforcement learning.
method Maximum Entropy Model Rollouts (MEMR) with non-uniform sampling and prioritized experience replay.
result Significantly reduces computation requirements compared to other model-based methods.
Quantum ML predicts data with improved speed and accuracy.
problem Predicting data using maximum likelihood in a quantum setting.
method Quantum states embedding and minimization of quantum relative entropy.
result Unified framework for classical and quantum LLMs with performance guarantees.
Entropy-based GP adaptive design improves failure probability estimation.
problem Limited accuracy in failure probability estimation due to model evaluation costs.
method Entropy-based Gaussian process (GP) adaptive design combined with multifidelity importance sampling (MFIS).
result More accurate failure probability estimates and higher confidence.