The problem of determining the joint probability distributions for correlated random variables with pre-specified marginals is considered. When the joint distribution satisfying all the required conditions is not unique, the "most unbiased" choice corresponds to the distribution of maximum entropy. The calculation of t…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,341 papers · 148 categories
Trend · papers per month
48 results for “marginal entropy”
New results on max-entropy distributions with succinct descriptions and stability.
problem Understanding the complexity and stability of max-entropy distributions.
method Polynomial-time algorithms and bounds on bit complexity.
result Polynomial bit complexity of ε-optimal dual solutions to max-entropy convex programs.
Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithmsstat.ML
Paper studies Fenchel-Young losses for classifier construction.
problem Creating effective loss functions for classifiers.
method Analyzes Fenchel-Young losses from generalized entropies, formulates conditions for separation margins and sparse support.
result Fenchel-Young losses can induce predictive distributions with separation margins and sparse support.
Bayesian models use hyperparameters to indirectly assign priors, and this work shows how these priors can be derived from maximum entropy principles.
problem Understanding the assumptions and dependencies in Bayesian hierarchical models.
method Demonstrates how canonical distributions and maximum entropy principles can be used to derive marginal priors in hierarchical models.
result Marginal priors in hierarchical models derived from maximum entropy principles have different constraints compared to the original priors.
This work improves policy optimization by maximizing entropy of state distribution, leading to better exploration.
problem Lack of exploration in state space when maximizing policy entropy.
method Proposes maximizing the entropy of a lower bound approximation to the state weighting distribution, based on latent space representation.
result Entropy regularization based on marginal state distribution achieves superior state space coverage and better performance in various domains.
Proposes SOVR loss to improve adversarial robustness by increasing logit margins.
problem Adversarial training's difficulty in robustness against sophisticated attacks.
method Introduces SOVR loss function that switches from cross-entropy to one-vs-the-rest loss for important samples.
result SOVR loss increases logit margins of important samples, improving robustness against Auto-Attack.
This note improves code quality for equientropic channels by maximizing marginal entropy.
problem Improving code quality for equientropic channels with finite bit usage.
method Characterizes code quality by an upper bound on expected minimum error probability for equientropic channels, showing random coding maximizes marginal entropy.
result For equientropic channels, random coding maximizes marginal entropy and achieves minimal upper bound on expected minimum error probability.
A framework estimates categorical distributions under constraints, ensuring generality and uniqueness.
problem Estimating categorical distributions summarizing sample data under marginal constraints.
method Theoretical framework + Iterative Proportional Fitting (IPF) to estimate the distribution.
result A unique categorical distribution of Maximum Entropy under marginal constraints exists and is estimated.
New risk bound derived for multi-category margin classifiers.
problem Guaranteed risk dependency on categories, sample size, and margin parameter.
method Derived a new risk bound using Rademacher complexity and chaining method.
result Improved dependency on categories over state of the art.
The paper presents a method to estimate joint interventional distributions from marginal interventional data.
problem Estimating joint interventional distributions from marginal interventional data.
method The paper extends the Causal Maximum Entropy method to use interventional data and employs Lagrange duality to prove the solution lies in the exponential family.
result The method allows for causal feature selection and inference of joint interventional distributions.
New method uncovers zero entropy in dependent observations after finite samples.
problem Understanding uncertainty reduction in dependent observations.
method Minimum list entropy coupling, greedy algorithm.
result Zero entropy achieved with O(log(1/P_min)) samples for dependent observations.
This work improves adversarial robustness by boosting model ensembles with margin maximization.
problem Single models are insufficient for defending against adversarial attacks.
method Margin-boosting approach to learn ensembles with maximum margin.
result Our algorithm outperforms existing ensembling techniques and large models trained end-to-end.
Paper proposes a new uncertainty measure for active learning in neural networks.
problem Efficiently selecting informative data points in limited labeled data scenarios.
method BalEntAcq, a new uncertainty measure based on balanced entropy, approximated by Beta distributions.
result BalEntAcq outperforms existing uncertainty measures in active learning.
A new classifier updates sequentially using maximum margin principles.
problem Sequential data collection and partial labeling.
method Maximum margin classifier with Maximum Entropy Discrimination principle, kernel representation, and regularization.
result Improved performance compared to non-sequential classifiers.
Proposes MGCE for improved classification performance.
problem Optimizing between robustness and optimization difficulty in classification.
method Minimax formulation of GCE leading to convex optimization over margins.
result MGCE achieves strong accuracy and better calibration, especially in noisy labels.
Refinement revisited with connections to Bayes error, conditional entropy and calibrated classifiersstat.ML
The concept of refinement from probability elicitation is considered for proper scoring rules. Taking directions from the axioms of probability, refinement is further clarified using a Hilbert space interpretation and reformulated into the underlying data distribution setting where connections to maximal marginal diver…
MEC-Cox: A Machine-Learning-Assisted Generalized Entropy Calibration Method for Estimating ATT Marginal Hazard-Ratio
problem Estimating ATT marginal hazard-ratio in externally controlled survival trials
method Machine-learning-assisted generalized entropy calibration for IPW Cox regression
result Reduces bias, increases efficiency, and improves coverage
Proposes L-Softmax loss for CNNs to improve feature discriminativeness.
problem Lack of explicit feature discriminativeness in cross-entropy loss.
method Introduces L-Softmax loss that encourages intra-class compactness and inter-class separability.
result Deeply learned features with L-Softmax loss are more discriminative, boosting performance.
A new framework for causal inference from entropy minimization.
problem Identifying causal relationships between discrete random variables from data.
method Minimum entropy coupling problem solved via a greedy algorithm.
result The greedy algorithm finds a local minimum and is within an additive error of the global optimum.
In this paper, we present a novel and general framework called {\it Maximum Entropy Discrimination Markov Networks} (MaxEnDNet), which integrates the max-margin structured learning and Bayesian-style estimation and combines and extends their merits. Major innovations of this model include: 1) It generalizes the extant …
MIM learns joint distributions with mutual information and low divergence.
problem Learning joint distributions over observations and latent variables.
method Probabilistic auto-encoder with three design principles: low divergence, high mutual information, and low marginal entropy.
result MIM learns representations with high mutual information, consistent encoding and decoding distributions, effective latent clustering, and comparable data log likelihood to VAE.
A new algorithm maximizes entropy or mutual information for efficient inference of nonstationary Gaussian processes.
problem Nonstationary dynamics in real-world phenomena pose challenges to accurate modeling.
method LISAL algorithm that adaptively maximizes entropy or mutual information on induced latent dynamics and marginal likelihood.
result Efficient inference of nonstationary Gaussian processes for large-scale real-world applications.
Develops a Best-of-Both-Worlds algorithm for linear contextual bandits with Tsallis entropy.
problem Linear contextual bandits with i.i.d. contexts.
method Follow-The-Regularized-Leader (FTRL) with Tsallis entropy.
result Achieves $O\left(\log(T)^{\frac{1+β}{2+β}}T^{\frac{1}{2+β}}
ight)$ regret under margin condition.
New methods improve Bayesian inference and decision-making in online learning.
problem Current Bayesian deep learning does not fully utilize joint predictives for sequential decision-making.
method Proposes new evaluation settings for active learning and active sampling, focusing on marginal and joint cross-entropies.
result Initial experiments suggest challenges in applying current BDL inference techniques in high-dimensional spaces.
New training scheme reduces adversarial examples by increasing decision boundary margin.
problem Vulnerability of neural networks to adversarial examples.
method Differential training using a loss function on feature differences.
result Differential training significantly reduces adversarial examples.
A new copula, the checkerboard copula, maximizes entropy and preserves dependence.
problem Choosing copula for non-continuous marginal distributions.
method Introducing the checkerboard copula, maximizing Shannon entropy.
result Checkerboard copula maximizes entropy and preserves dependence.
Improved sampling from complex distributions with reduced bias.
problem Reducing bias in high-dimensional sampling algorithms.
method Hierarchical entropy analysis to weaken assumptions and expand scope.
result Bias reduction in low-dimensional marginals scales with lower dimension, not full dimension.
The paper introduces a minimax approach to supervised learning problems.
problem Optimal decision rule minimizing worst-case expected loss over probability distributions.
method Generalization of maximum entropy principle applied to constrained distributions.
result Developed a new linear classifier called the maximum entropy machine.
Study finds optimal martingale coupling between two distributions with minimal entropy.
problem Finding the optimal martingale coupling between two distributions with minimal relative entropy.
method Solving a dual problem to find the log-density of the optimal coupling, which represents the marginal and martingale constraints.
result The log-density of the optimal coupling is given by a triplet of real functions representing the marginal and martingale constraints.
The paper improves itemset quality assessment by incorporating background knowledge.
problem Assessing the quality of discovered itemsets is challenging due to many patterns being explainable by background knowledge.
method The authors introduce a maximum entropy approach to efficiently infuse additional background knowledge such as row margins, lazarus counts, and bounds of ones.
result More sophisticated models that incorporate background knowledge fit the data better and improve frequency prediction of itemsets.
The article introduces gamma-Psi-dimensions for margin multi-category classifiers.
problem Margin multi-category classifiers' generalization performance under minimal learnability hypotheses.
method Derives gamma-Psi-dimensions, handles capacity measures, and establishes upper bounds on metric entropies and Rademacher complexity.
result Gamma-Psi-dimensions improve over fat-shattering dimension and offer a promising alternative for multi-class to binary transitions.
A new test for independence using mutual information.
problem Testing independence between two multivariate random vectors.
method MINT test based on mutual information decomposition and efficient entropy estimators.
result The test has nominal size and provides local power analyses.
New framework uses entropy to improve robust color transfer.
problem Improving color transfer between images.
method Entropy regularisation in robust optimal transport.
result Shows Grogan et al's method is a robust optimal transport framework.
We define a generalized likelihood function based on uncertainty measures and show that maximizing such a likelihood function for different measures induces different types of classifiers. In the probabilistic framework, we obtain classifiers that optimize the cross-entropy function. In the possibilistic framework, we …
Default-ERM shortcut learning persists even without additional information.
problem Default-ERM shortcut learning in perception tasks despite stable feature sufficiency.
method Studied linear perception task; developed margin control (MARG-CTRL) loss functions.
result Margin control mitigates shortcut learning on various tasks.
Study on convergence rates for optimal transport with regularization.
problem Convergence analysis of divergence-regularized optimal transport.
method Novel methodology using quantization and martingale couplings.
result Sharp rates for various divergences and transport costs.
Exact partial information decompositions for Gaussian systems based on dependency constraintscond-mat.stat-mech
New method calculates partial information for Gaussian systems based on dependency constraints.
problem Quantifying information sharing in multivariate Gaussian systems.
method Constructing maximum entropy models based on dependency constraints and deriving closed-form solutions.
result Closed-form solutions for Gaussian systems show differences in redundancy and synergy estimates compared to existing methods.
Proposes a method to robustly classify and detect anomalies.
problem Learning robust binary classifiers in the presence of corrupted measurements.
method Geometric-Entropy-Minimization regularized Maximum Entropy Discrimination (GEM-MED) method.
result Improved performance in classification accuracy and anomaly detection rate.
Multithreshold Entropy Linear Classifier (MELC) is a recent classifier idea which employs information theoretic concept in order to create a multithreshold maximum margin model. In this paper we analyze its consistency over multithreshold linear models and show that its objective function upper bounds the amount of mis…
New method tightens bounds on causation probabilities using independent datasets.
problem Challenging point identification of causation probabilities without strong assumptions.
method Imposes counterfactual consistency between SCMs constructed from independent datasets and uses conditional mutual information.
result Significantly tighter bounds on causation probabilities are established.
PFES uses entropy of Pareto-frontier for multi-objective Bayesian optimization.
problem Bayesian optimization for multi-objective problems, especially trade-off among objectives.
method Pareto-frontier entropy search (PFES) incorporating trade-off relation.
result PFES effectively incorporates dependency among objectives conditioned on Pareto-frontier.
Stability result for a popular algorithm in optimal transport.
problem Stability of the Iterative Proportional Fitting Procedure in time and metric.
method Uniform stability analysis in the 1-Wasserstein metric.
result Quantitative stability result for entropy-regularized Optimal Transport and Schrödinger bridges.
Paper tackles long-tailed labels in classification problems.
problem Imbalanced or long-tailed label distribution in real-world classification problems.
method Logit adjustment applied post-hoc or during training to encourage a large relative margin between rare and dominant labels.
result Unified and generalised techniques for coping with long-tailed labels, improving generalisation and performance.
Toward a Theory of Marginally Efficient Marketscond-mat.stat-mech
Empirical evidence suggests that even the most competitive markets are not strictly efficient. Price histories can be used to predict near future returns with a probability better than random chance. Many markets can be considered as {\it favorable games}, in the sense that there is a small probabilistic edge that smar…
New method resolves nonidentifiability in mixture models.
problem Nonidentifiability in marginal models of mixture models.
method Introducing an effective temperature to generalize the marginal likelihood.
result Maximization of the generalized likelihood leads to unique results.
Study shows uniform-time chaos propagation in mean field Langevin dynamics.
problem Understanding the convergence of marginal distributions in mean field dynamics.
method Assumed functional convexity of energy, used -convergence and Wasserstein metrics.
result Uniform-in-time propagation of chaos proved in both -Wasserstein and relative entropy.
The Sinkhorn flow converges to a Wasserstein mirror gradient flow from the Sinkhorn algorithm.
problem Optimizing joint distributions using the Sinkhorn algorithm.
method Wasserstein mirror gradient flow derived from the Sinkhorn algorithm.
result The Sinkhorn flow converges to a Wasserstein mirror gradient flow.
Conservation laws improve diffusion model training by optimizing likelihood.
problem Training diffusion models with denoising objectives.
method Developed conservation laws based on GEXIT functions for memoryless noise processes.
result Unified characterization of diffusion model likelihood, reducing training to learning marginal posteriors.