AECF improves multimodal inference robustness and calibration.
problem Robustness and calibration issues in multimodal systems with missing inputs.
method Adaptive Entropy-Gated Contrastive Fusion (AECF) layer.
result Improves masked-input mAP by +18 pp at a 50% drop rate.
The paper studies entropy calibration in language models and finds that miscalibration improves slowly with scale.
problem The problem is whether language model entropy calibration improves with scale and if it's possible to calibrate without reducing log loss.
method The authors study a simplified theoretical setting to characterize miscalibration scaling behavior and measure it empirically in language models ranging from 0.5B to 70B parameters.
result The observed scaling behavior of miscalibration is similar to theoretical predictions, indicating slow improvement with scale. The authors also prove theoretically that it is possible to reduce entropy while preserving log loss if access to a black box predicting future entropy is available.
MEC-Cox: A Machine-Learning-Assisted Generalized Entropy Calibration Method for Estimating ATT Marginal Hazard-Ratio
problem Estimating ATT marginal hazard-ratio in externally controlled survival trials
method Machine-learning-assisted generalized entropy calibration for IPW Cox regression
result Reduces bias, increases efficiency, and improves coverage
For many applications it is critical to know the uncertainty of a neural network's predictions. While a variety of neural network parameter estimation methods have been proposed for uncertainty estimation, they have not been rigorously compared across uncertainty measures. We assess four of these parameter estimation m…
MEC improves efficiency and robustness in semi-supervised inference.
problem Efficient inference with limited labeled data and robust uncertainty quantification.
method Machine-Learning-Assisted Generalized Entropy Calibration (MEC) using cross-fitted, calibration-weighted PPI.
result MEC achieves semiparametric efficiency bounds under weaker assumptions and provides near-nominal coverage.
Proposes squentropy loss for improved classification accuracy and model calibration.
problem Theoretical and empirical evidence for cross-entropy loss is lacking.
method Introduces squentropy loss as the sum of cross-entropy and average square loss over incorrect classes.
result Squentropy loss outperforms cross-entropy and rescaled square losses in classification accuracy and model calibration.
Building accurate language models that capture meaningful long-term dependencies is a core challenge in natural language processing. Towards this end, we present a calibration-based approach to measure long-term discrepancies between a generative sequence model and the true distribution, and use these discrepancies to …
Unsupervised domain adaptation (UDA) aims at inferring class labels for unlabeled target domain given a related labeled source dataset. Intuitively, a model trained on source domain normally produces higher uncertainties for unseen data. In this work, we build on this assumption and propose to adapt from source to targ…
The concept of refinement from probability elicitation is considered for proper scoring rules. Taking directions from the axioms of probability, refinement is further clarified using a Hilbert space interpretation and reformulated into the underlying data distribution setting where connections to maximal marginal diver…
This paper tackles over-certainty in test-time adaptation models, proposing a solution to improve calibration.
problem Over-certainty in predictions caused by domain shifts, leading to misplaced trust.
method Introduces a certainty regularizer that dynamically adjusts pseudo-label confidence based on backbone entropy and logit norm.
result Achieves state-of-the-art performance in terms of Expected Calibration Error and Negative Log Likelihood, while maintaining accuracy.
AdaDEM decouples EM into two parts to improve class overlap and uncertainty.
problem Improper EM limits its effectiveness in various machine learning tasks.
method Decouple EM into CADF and GMC, and AdaDEM normalizes CADF reward and uses MEC.
result AdaDEM outperforms classical EM and improves performance in noisy and dynamic environments.
Ancient solutions and translators identified for Lagrangian flow.
problem Characterizing ancient solutions and translators of Lagrangian mean curvature flow.
method Analyzing almost calibrated, exact, ancient solutions with specific geometric properties.
result All ancient solutions with entropy less than 3 are special Lagrangian, planes, or translators in \(\mathbb{C}^2\).
Proposes an accuracy-preserving calibration method for DNNs.
problem Calibration of deep neural networks (DNNs) to measure prediction reliability.
method Uses Concrete distribution on the probability simplex to calibrate DNNs without accuracy loss.
result The proposed method outperforms previous methods in accuracy-preserving calibration tasks.
New method calibrates reference distributions for bounded support.
problem Lack of principled method for bounded-support statistical reference distributions.
method Formulated maximum entropy on projective space of nonnegative measures.
result Prescribed acceptance region uniquely determines deformation parameter.
Temperature scaling improves model uncertainty but not diversity in LLMs.
problem Improving the calibration and stochasticity of probabilistic models.
method Investigates theoretical properties of temperature scaling in classification and LLMs.
result Temperature scaling increases model uncertainty but not diversity in LLMs.
Entropy based ideas find wide-ranging applications in finance for calibrating models of portfolio risk as well as options pricing. The abstracted problem, extensively studied in the literature, corresponds to finding a probability measure that minimizes relative entropy with respect to a specified measure while satisfy…
The study examines label smoothing to improve confidence calibration in fine-tuned LLMs.
problem Improving confidence calibration in fine-tuned large language models (LLMs) after instruction tuning.
method Examine various open-sourced LLMs, label smoothing, and custom kernel design.
result Label smoothing is effective in maintaining confidence calibration but faces challenges in large vocabulary LLMs.
Paper proposes ensemble distillation for well-calibrated structured prediction.
problem Well-calibrated predictions are hard to achieve in structured prediction.
method Ensemble distillation framework for structured prediction.
result Ensemble distillation produces well-calibrated models with similar performance and calibration benefits to ensembles.
Estimates proper calibration errors and refinement terms in probabilistic predictions.
problem Lack of a general estimator for proper calibration errors and refinement terms with known statistical properties.
method Proposes a method for consistent, asymptotically unbiased estimation of proper calibration errors and refinement terms.
result Proves the relation between refinement and f-divergences, implying information monotonicity in neural networks.
The accuracy of least squares calibration using option premiums and particle filtering of price data to find model parameters is determined. Derivative models using exponential Lévy processes are calibrated using regularized weighted least squares with respect to the minimal entropy martingale measure. Sequential impor…
This work investigates square loss in overparametrized neural networks, revealing its advantages in robustness and calibration.
problem Theoretical understanding of square loss in overparametrized neural networks.
method Systematic investigation of square loss in the NTK regime for both separable and non-separable classes.
result Square loss shows fast convergence rates and robustness guarantees for overparametrized neural networks.
The resolution and calibration of pure spectra of minority components in measurements of chemical mixtures without prior knowledge of the mixture is a challenging problem. In this work, a combination of band target entropy minimization (BTEM) and target partial least squares (T-PLS) was used to obtain estimates for sin…
In this paper we study the probabilistic properties of the posteriors in a speech recognition system that uses a deep neural network (DNN) for acoustic modeling. We do this by reducing Kaldi's DNN shared pdf-id posteriors to phone likelihoods, and using test set forced alignments to evaluate these using a calibration s…
Boosted decision trees typically yield good accuracy, precision, and ROC area. However, because the outputs from boosting are not well calibrated posterior probabilities, boosting yields poor squared error and cross-entropy. We empirically demonstrate why AdaBoost predicts distorted probabilities and examine three cali…
Focal loss improves deep neural networks' accuracy and calibration.
problem Miscalibration in deep neural networks.
method Using focal loss and temperature scaling to improve model calibration.
result Focal loss leads to state-of-the-art calibrated models without sacrificing accuracy.
Resultant improves likelihood-based U-OOD detection across various tasks.
problem Improving likelihood-based U-OOD detection performance.
method Resultant combines post-hoc prior and dataset entropy-mutual calibration techniques.
result Resultant achieves new state-of-the-art U-OOD detection performance.
Study proves stability and uniqueness for a specific type of flow.
problem Volume-preserving mean curvature flow stability and uniqueness.
method New gradient flow calibrations for volume preservation, stability estimate in distributional solutions.
result Strong solutions are calibrated and stable under certain conditions.
New method improves model calibration by adjusting confidence based on prediction correctness.
problem Improving model confidence alignment with true class probabilities.
method Post-hoc calibration objective using transformed samples for training.
result Competitive calibration performance on in-distribution and out-of-distribution test sets.
We present a scoring approach for speaker verification that mimics the standard PLDA-based backend process used in most current speaker verification systems. However, unlike the standard backends, all parameters of the model are jointly trained to optimize the binary cross-entropy for the speaker verification task. We …
Efficient approach improves prediction calibration for domain shifts.
problem Improving uncertainty-aware predictions for domain shifts.
method Combining entropy-encouraging and adversarial calibration losses.
result Substantially outperforms existing approaches in domain drift calibration.
Proposes MGCE for improved classification performance.
problem Optimizing between robustness and optimization difficulty in classification.
method Minimax formulation of GCE leading to convex optimization over margins.
result MGCE achieves strong accuracy and better calibration, especially in noisy labels.
The top-k error is often employed to evaluate performance for challenging classification tasks in computer vision as it is designed to compensate for ambiguity in ground truth labels. This practical success motivates our theoretical analysis of consistent top-k classification. Surprisingly, it is not rigorously und…
We obtain the maximum entropy distribution for an asset from call and digital option prices. A rigorous mathematical proof of its existence and exponential form is given, which can also be applied to legitimise a formal derivation by Buchen and Kelly. We give a simple and robust algorithm for our method and compare our…
Uncertainty estimates help to identify ambiguous, novel, or anomalous inputs, but the reliable quantification of uncertainty has proven to be challenging for modern deep networks. In order to improve uncertainty estimation, we propose On-Manifold Adversarial Data Augmentation or OMADA, which specifically attempts to ge…
In the existing financial literature, entropy based ideas have been proposed in portfolio optimization, in model calibration for options pricing as well as in ascertaining a pricing measure in incomplete markets. The abstracted problem corresponds to finding a probability measure that minimizes the relative entropy (al…
Focal loss reduces model curvature for better calibration.
problem Improving model confidence in classification problems.
method Geometric interpretation of focal loss to reduce curvature.
result Focal loss reduces the curvature of the loss surface, enhancing model calibration.
A new approach for test-time adaptation detects and reacts to distribution shifts.
problem Improving test-time accuracy under distribution shifts.
method Online self-training with a detection tool based on entropy values and betting martingales.
result The classifier's entropy values match those of the source domain, building invariance to distribution shifts.
In a recent work, we presented a discriminative backend for speaker verification that achieved good out-of-the-box calibration performance on most tested conditions containing varying levels of mismatch to the training conditions. This backend mimics the standard PLDA-based backend process used in most current speaker …
ALIEN improves uncertainty estimation of language models by refining entropy-based methods.
problem Overconfidence in uncertainty estimation for language models, especially for difficult inputs.
method ALIEN refines entropy-based uncertainty by aligning it with prediction reliability, using a lightweight uncertainty head.
result ALIEN consistently outperforms strong baselines in detecting incorrect predictions and achieving the lowest calibration error.
ECLIPSE detects AI hallucinations in finance with high accuracy.
problem Hallucinations in AI-generated answers limit safe deployment in finance.
method Combines entropy estimation and perplexity decomposition to measure model evidence use.
result ECLIPSE achieves ROC AUC of 0.89 and average precision of 0.90 on financial QA dataset.
New framework assesses LM uncertainty without thresholding.
problem Uncertainty quantification for LMs, especially comparing different measures.
method Rank-Calibration framework to assess uncertainty and confidence measures.
result Higher uncertainty correlates with lower generation quality.
ABC improves uncertainty quantification in LLMs for clinical diagnostics.
problem Overconfident and poorly calibrated estimates of LLMs in clinical domains.
method Approximate Bayesian Computation (ABC) for likelihood-free inference.
result Improves accuracy by up to 46.9%, reduces Brier scores by 74.4%, and enhances calibration.
The paper decomposes probabilistic scores into reliability, uncertainty, and information loss.
problem Understanding the reliability and uncertainty of probabilistic predictions.
method Developed decomposition identities for proper losses, quantifying reliability, residual uncertainty, and information gain.
result A three-term identity for classification scores, revealing miscalibration, grouping term, and feature-level uncertainty.
We propose a generic framework to calibrate accuracy and confidence of a prediction in deep neural networks through stochastic inferences. We interpret stochastic regularization using a Bayesian model, and analyze the relation between predictive uncertainty of networks and variance of the prediction scores obtained by …
OrthoGrad improves neural calibration by constraining gradient updates orthogonally.
problem Overconfidence in neural networks, leading to poor uncertainty estimates.
method Orthogonal gradient updates to optimize for decision boundaries and reduce overconfidence.
result Significant improvements in test loss, predictive entropy, and confidence measures.
In this paper we study several issues related to the generation of superpotential induced by background Ramond-Ramond fluxes in compactification of Type IIA string theory on Calabi-Yau four-folds. Identifying BPS solitons with D-branes wrapped over calibrated submanifolds in a Calabi-Yau space, we propose a general for…
In this paper, we present a machine learning approach for estimating the number of incident wavefronts in a direction of arrival scenario. In contrast to previous works, a multilayer neural network with a cross-entropy objective is trained. Furthermore, we investigate an online training procedure that allows an adaptio…
Generative Cross-Entropy improves classification with fewer labels.
problem Limited sample efficiency of cross-entropy loss in data-scarce scenarios.
method Proposes Generative Cross-Entropy (GenCE), a new loss function that incorporates generative principles into a standard discriminative network.
result Generative Cross-Entropy outperforms traditional cross-entropy loss across various datasets and conditions.