Confidence intervals are a popular way to visualize and analyze data distributions. Unlike p-values, they can convey information both about statistical significance as well as effect size. However, very little work exists on applying confidence intervals to multivariate data. In this paper we define confidence interval…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,694 papers · 148 categories
Trend · papers per month
48 results for “confidence information”
New confidence intervals improve treatment effect estimation in randomized experiments.
problem Improving confidence intervals for treatment effects in randomized experiments.
method Systematic exploitation of negative dependence or variance adaptivity.
result Achieved nonasymptotic confidence intervals with the same effective sample size as asymptotic ones.
The paper proposes using entropy to assess model confidence.
problem Discarded information in probability distributions hinders model confidence assessment.
method Entropy methods applied to assess model confidence.
result Theoretical explanation of confidence degradation in Naive Bayes model.
In this paper we study leveraging confidence information induced by adversarial training to reinforce adversarial robustness of a given adversarially trained model. A natural measure of confidence is (i.e. how confident is about its prediction?). We start by analyzing an adversarial training…
New method improves certified robustness for classifier confidence.
problem Certifying confidence in classifier predictions.
method Randomized smoothing with modified Neyman-Pearson lemma.
result Certified radii for prediction confidence improved.
Pathology reports contain useful information such as the main involved organ, diagnosis, etc. These information can be identified from the free text reports and used for large-scale statistical analysis or serve as annotation for other modalities such as pathology slides images. However, manual classification for a hug…
Recursive PAC-Bayes: A Frequentist Approach to Sequential Prior Updates with No Information Losscs.LG
New PAC-Bayes method updates priors without losing confidence information.
problem Lack of sequential prior updates in PAC-Bayes without losing confidence information.
method Recursive PAC-Bayes decomposition of expected loss.
result Sequential prior updates with no information loss.
We integrate information-theoretic concepts into the design and analysis of optimistic algorithms and Thompson sampling. By making a connection between information-theoretic quantities and confidence bounds, we obtain results that relate the per-period performance of the agent with its information gain about the enviro…
LLMs generate answers under incomplete context, and their uncertainty should scale with missing information.
problem Evaluating the quality of LLM answers under incomplete context.
method A controlled framework with varying context availability, and two uncertainty measures (sampling-based confidence and response entropy) evaluated on SQuAD.
result Response entropy increases with context removal and explains more variance in accuracy than confidence, suggesting it is a more responsive uncertainty measure.
Mahalanobis distance detects anomalies well, but not for classification.
problem Detecting anomalies in neural classifier outputs.
method Analyzes Mahalanobis distance-based anomaly detection method, revealing its reliance on information not useful for classification.
result Combining Mahalanobis and ODIN methods improves anomaly detection performance and robustness.
Information Forestscs.LG
We describe Information Forests, an approach to classification that generalizes Random Forests by replacing the splitting criterion of non-leaf nodes from a discriminative one -- based on the entropy of the label distribution -- to a generative one -- based on maximizing the information divergence between the class-con…
A new concept of confidence in learning is defined and analyzed.
problem Understanding and quantifying trust in learning processes.
method Formal axioms, continuum measures, vector fields, loss functions.
result Confidence can be represented and optimized in learning.
This study evaluates methods to measure traffic forecasting model confidence.
problem Lack of consensus on uncertainty types and techniques for traffic forecasting models.
method Reviews and compares different uncertainty estimation techniques using real traffic data.
result Empirical evidence shows benefits and caveats of various techniques.
Mixed likelihood GPs improve model performance in human-in-the-loop experiments.
problem Lack of auxiliary information in traditional GPs for human responses.
method Propose mixed likelihood variational GPs to leverage auxiliary information.
result Modeling performance improvements across diverse applications.
Bayes-assisted confidence sequences improve efficiency for bounded means.
problem Efficient uncertainty quantification for bounded IID means without parametric assumptions.
method Bayesian working predictive model selects adaptive martingale updates maximizing predictive log-growth.
result Asymptotically log-optimal performance with informative priors reducing width and sampling effort.
Exploits class similarity for better machine learning models with confidence labels and projective loss functions.
problem Poor model performance due to confusing similar classes.
method Exploits class similarity with confidence labels and projective loss functions.
result Improved model performance on noisy labels.
Novel framework for unbiased confidence estimates in object detection.
problem Unbiased confidence estimates for safety-critical object detection.
method Combines regression output with additional information for calibration.
result Calibrated confidence estimates for image location and scale.
Typical dimensionality reduction (DR) methods are often data-oriented, focusing on directly reducing the number of random variables (features) while retaining the maximal variations in the high-dimensional data. In unsupervised situations, one of the main limitations of these methods lies in their dependency on the sca…
Typical dimensionality reduction methods focus on directly reducing the number of random variables while retaining maximal variations in the data. In this paper, we consider the dimensionality reduction in parameter spaces of binary multivariate distributions. We propose a general Confident-Information-First (CIF) prin…
Efficient method for high confidence level inference using parallel stochastic optimization.
problem Uncertainty quantification for online estimation.
method Small number of independent multi-runs to construct t-based confidence intervals.
result Rigorous theoretical guarantee for exact coverage of confidence intervals.
GINNs: Graph-Informed Neural Networks for Multiscale Physicsphysics.comp-ph
GINNs combine deep learning with PGMs for physics-based multiscale systems.
problem Intrinsic computational bottlenecks and lack of sufficient data for QoI estimation.
method Hybrid approach combining deep learning with probabilistic graphical models, informed by structured priors for CVs.
result GINNs produce tight confidence intervals for non-Gaussian QoIs.
New method calibrates confidence for object detection and segmentation models.
problem Intrinsically miscalibrated confidence estimates in object detection and segmentation models.
method Introduces multivariate confidence calibration for object detection and segmentation, extending ECE.
result Improves calibration, positively impacts segmentation quality.
The process of data mining with differential privacy produces results that are affected by two types of noise: sampling noise due to data collection and privacy noise that is designed to prevent the reconstruction of sensitive information. In this paper, we consider the problem of designing confidence intervals for the…
The paper tackles best arm identification with minimal regret in experiments.
problem Identifying the best arm with minimal regret in experiments.
method Information-theoretic techniques and Double KL-UCB algorithm.
result Achieves asymptotic optimality in identifying the best arm with minimal regret.
RLFA estimates misstated monetary fraction with weighted sampling without replacement.
problem Estimating misstated monetary fraction with given accuracy and confidence.
method Developed new confidence sequences for weighted average of unknown values using randomized weighted sampling and side information.
result Adaptive methods improve accuracy of estimates based on side information's predictive power.
In many countries information on expectations collected through consumer confidence surveys are used in macroeconomic policy formulation. Unfortunately, before doing so, the consistency of responses is often not taken into account, leading to biases creeping in and affecting the reliability of the indices hence created…
Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confide…
Robust Conformal Prediction under Distribution Shift via Physics-Informed Structural Causal Modelcs.LG
CP improves robustness against distribution shift using physics-informed structural causal models.
problem Uncertainty in machine learning predictions under distributional shift.
method Physics-informed structural causal model (PI-SCM) to upper bound coverage difference.
result PI-SCM improves coverage robustness across confidence levels and test domains.
New method estimates grouping loss in neural networks to improve confidence scores.
problem Improving confidence scores in neural networks to reflect true posterior probabilities.
method Proposed an estimator to approximate the grouping loss.
result Modern neural networks exhibit grouping loss, especially in distribution shifts.
New metrics CWSA and CWSA+ improve model evaluation under confidence thresholds.
problem Lack of metrics capturing model reliability under confidence thresholds.
method Introducing CWSA and CWSA+ metrics that reward confident accuracy and penalize overconfident mistakes.
result CWSA and CWSA+ outperform classical metrics in trust-sensitive tests.
Optimal best-arm identification with known number of optimal arms.
problem Identifying the best arm in a multi-armed bandit with multiple optimal arms under fixed confidence.
method Deriving a new information-theoretic lower bound and proposing a modified stopping rule.
result Achieving asymptotic instance-optimality with a new lower bound and new stopping rule.
Precise estimation of uncertainty in predictions for AI systems is a critical factor in ensuring trust and safety. Deep neural networks trained with a conventional method are prone to over-confident predictions. In contrast to Bayesian neural networks that learn approximate distributions on weights to infer prediction …
Paper introduces methods for more reliable probabilistic predictions with confidence intervals.
problem Inaccurate labeling of datasets due to unreliable probabilistic predictions from weak labeling functions.
method Proposes a methodology to provide confidence intervals for label probabilities using uncertainty sets of distributions.
result Improves reliability of probabilistic predictions and provides confidence intervals for label probabilities.
VR game data for P300 BCI with raccoon vs demon stimuli.
problem Developing confidence metrics for P300 BCI.
method Multiclass labeled P300 dataset in VR game context.
result Estimation of model's confidence in stimulus predictions.
The paper improves confidence regions for band-limited functions using tighter norm bounds and majority voting.
problem Constructing reliable confidence regions for band-limited functions from noisy data.
method Improved norm bounds using Hoeffding's inequality and empirical Bernstein bound, majority voting to aggregate intervals.
result Confidence intervals retain their simultaneous coverage guarantee even when aggregated from random subsamples.
Unified framework for active learning problems using information theory.
problem Combining level set estimation and Bayesian optimization.
method Information-theoretic criterion and acquisition function.
result Unified framework achieves state-of-the-art performance.
New method uses weak labels to create valid confidence sets for predictions.
problem Lack of labeled data in machine learning models.
method Developed a conformal prediction framework to provide valid predictive confidence sets using weakly labeled data.
result New coverage definition allows for tighter and more informative (but valid) confidence sets.
Optimizes black-box functions with varying costs across multiple sources.
problem Optimizing black-box functions with varying costs across multiple sources.
method Uses Augmented Gaussian Process and Gaussian Process to model fidelity and location-dependent costs, respectively. Uses Confidence Bound acquisition function to select sources and locations.
result The approach significantly outperforms existing methods on Hyperparameters Optimization tasks.
Upper Confidence Bound (UCB) method is arguably the most celebrated one used in online decision making with partial information feedback. Existing techniques for constructing confidence bounds are typically built upon various concentration inequalities, which thus lead to over-exploration. In this paper, we propose a n…
New bounds on generalization error using information density moments.
problem Bounding the generalization error of randomized learning algorithms.
method Derives bounds on average and tail probabilities of generalization error using mth central moments of the information density.
result Explicit bounds on generalization error are derived, showing better dependence on confidence level with higher-order information density moments.
TeLeS improves ASR confidence estimation by considering temporal alignment and lexical errors.
problem Inaccurate confidence scores from E2E ASR models, especially for overconfident predictions.
method Proposes TeLeS, a novel confidence score that considers temporal alignment and lexical errors, and uses shrinkage loss to handle data imbalance.
result TeLeS generalizes well across different languages and ASR models, leading to significant WER reduction.
Modified AUC improves CNN training by considering model confidence.
problem Improving binary classifier performance metrics.
method Proposes a modified AUC metric that incorporates model confidence into BCE loss for CNN training.
result Demonstrates improved performance on three datasets: MNIST, prostate MRI, and brain MRI.
New algorithm for precise changepoint localization without assumptions.
problem Offline changepoint localization in arbitrary distributions.
method Distribution-free algorithm CONformal CHangepoint localization (CONCH) using exchangeability arguments.
result Derives principled score functions for informative and small confidence sets with normalized length shrinking to zero.
Proposes an active RBI framework using Rényi information measures for more informed decision-making.
problem Optimal latent variable estimates in real-time settings with streaming noisy observations.
method Unified inference and query selection steps through Rényi entropy and α-divergence; new objective called Momentum for exploration.
result Analytically demonstrates superior performance compared to conventional methods like mutual information.
New method for rigorous confidence intervals in off-policy evaluation.
problem Evaluate new policies from off-policy data without executing them.
method Variational framework using kernel Bellman loss.
result Efficient method for tight confidence intervals in various settings.
In training speech recognition systems, labeling audio clips can be expensive, and not all data is equally valuable. Active learning aims to label only the most informative samples to reduce cost. For speech recognition, confidence scores and other likelihood-based active learning methods have been shown to be effectiv…
Generative AI reduces IR evaluation costs but introduces errors; this work provides reliable CIs.
problem Generating relevance annotations using AI introduces errors that affect IR evaluation metrics.
method Proposes two methods: prediction-powered inference and conformal risk control to place reliable CIs around IR metrics.
result Proposed methods accurately capture both variance and bias in evaluation based on AI-generated annotations.
Entropy-based decoding improves DLM sampling efficiency.
problem Decoding strategy challenges in flexible DLMs.
method Entropy sum-based confidence-based decoding.
result Entropy sum-based decoding achieves -accuracy with iterations.