PRCD-MAP learns to trust imperfect priors in causal discovery, improving accuracy and robustness.
problem Tackles the brittle trade-off between blind trust and rejection of external priors in causal discovery.
method Proposes PRCD-MAP, a soft prior-consumption layer that assigns per-edge trust to imperfect priors and modulates regularization in a MAP objective.
result Enjoys a population-level safety guarantee and outperforms existing methods on real-world causal discovery tasks.
The paper proposes calibration to improve algorithm performance using machine learning predictions.
problem Improving real-world performance of online algorithms with machine learning predictions.
method Calibration as a tool to bridge the gap between prediction uncertainty and algorithm design.
result Calibrated advice leads to more effective guidance in high-variance settings and significant performance improvements in real-world data.
Improves model calibration by addressing adversarially unrobust inputs.
problem Miscalibrated predictions and lack of adversarial robustness in neural networks.
method AR-AdaLS, an adaptive label smoothing method that considers adversarial robustness.
result AR-AdaLS improves model calibration, even under distributional shifts.
The paper addresses decision making with partially calibrated forecasts, offering a robust approach.
problem Developing a decision-making strategy for forecasts that are only partially calibrated.
method A minimax approach to mapping predictions to actions, considering worst-case distributions.
result The minimax optimal decision rule is to trust predictions and act accordingly, even for partially calibrated forecasts.
In order to cope with the increased data volumes generated by modern radio interferometers such as LOFAR (Low Frequency Array) or SKA (Square Kilometre Array), fast and efficient calibration algorithms are essential. Traditional radio interferometric calibration is performed using nonlinear optimization techniques such…
The paper addresses the difficulty of decision makers trusting AI-assisted predictions and proposes a method to improve confidence values.
problem Decision makers struggle to trust AI-assisted predictions based on confidence values.
method The paper investigates why decision makers have difficulties and proposes a method to construct more useful confidence values.
result Multicalibration with respect to the decision maker's confidence on her own predictions is a sufficient condition for alignment, leading to better decisions.
New framework for evaluating multiclass classifier calibration.
problem Ensuring classifiers are well-calibrated for trustworthy predictions.
method Utility Calibration framework that measures calibration error relative to a utility function.
result Unified and robust interpretation of existing calibration metrics.
The study examines when to trust confidence thresholding in pseudo-labelling regression.
problem Calibrated probabilities from classifiers used for pseudo-labelling need careful handling to avoid bias in downstream regression.
method Developed a diagnostic apparatus to predict and bound the bias induced by confidence thresholding, derived a closed-form expression for the attenuation bias.
result The bias can be predicted from the residual score variance V∗, motivating a structural separation between classifier features and downstream controls. New metrics CWSA and CWSA+ improve model evaluation under confidence thresholds.
problem Lack of metrics capturing model reliability under confidence thresholds.
method Introducing CWSA and CWSA+ metrics that reward confident accuracy and penalize overconfident mistakes.
result CWSA and CWSA+ outperform classical metrics in trust-sensitive tests.
The paper studies calibration in ML models for wireless networks, showing key theoretical and practical insights.
problem Ensuring ML models in wireless networks deliver well-calibrated confidence scores for reliable decision-making.
method Theoretical analysis and simulation-based experiments using Platt scaling and isotonic regression.
result Well-calibrated models improve the system's minimum achievable OP and are part of a broader class of predictors.
SC unifies ICL calibration methods and improves LLM performance.
problem Systematic biases in LLM predictions leading to unstable performance.
method Supervised Calibration (SC) learns optimal affine transformations in logit space.
result SC delivers state-of-the-art performance across multiple datasets.
Modern machine learning methods including deep learning have achieved great success in predictive accuracy for supervised learning tasks, but may still fall short in giving useful estimates of their predictive {\em uncertainty}. Quantifying uncertainty is especially critical in real-world settings, which often involve …
New methods improve confidence set calibration in complex models.
problem Challenges in maintaining confidence set coverage in complex models.
method TRUST and TRUST++ methods using simulated data for calibration.
result Methods achieve distribution-free conditional coverage and robust inference.
People who design, use, and are affected by autonomous artificially intelligent agents want to be able to \emph{trust} such agents -- that is, to know that these agents will perform correctly, to understand the reasoning behind their actions, and to know how to use them appropriately. Many techniques have been devised …
Causal discovery can be a powerful tool for investigating causality when a system can be observed but is inaccessible to experiments in practice. Despite this, it is rarely used in any scientific or medical fields. One of the major hurdles preventing the field of causal discovery from having a larger impact is that it …
New approach calibrates predictions for better decision-making.
problem Achieving reliable predictions for multi-class problems is hard.
method Introduces decision calibration, a new approach to calibrate predictions.
result Designs a recalibration algorithm that makes predictions reliable for decision-making.
A macroeconomic model based on the economic variables (i) assets, (ii) leverage (defined as debt over asset) and (iii) trust (defined as the maximum sustainable leverage) is proposed to investigate the role of credit in the dynamics of economic growth, and how credit may be associated with both economic performance and…
Recently, Deep Neural Networks (DNNs) have been achieving impressive results on wide range of tasks. However, they suffer from being well-calibrated. In decision-making applications, such as autonomous driving or medical diagnosing, the confidence of deep networks plays an important role to bring the trust and reliabil…
This paper reviews metrics to assess AI model calibration accuracy.
problem AI model probabilities do not always match their true accuracy.
method Comprehensive review of 82 probability calibration metrics.
result Identified 4 classifier families and 1 object detection family of metrics.
New calibration measure SCDL improves trust in AI predictions.
problem Improving trust in AI predictions by ensuring they are both actionable and testable.
method Introducing SCDL, a new calibration measure that is fully actionable and testable.
result SCDL is the first calibration measure that is fully actionable and testable.
GP model calibration improves optimization algorithm performance.
problem GP model uncertainty calibration issues degrade optimization performance.
method Kernel validation procedure to calibrate GP predictions.
result Proper calibration enhances optimization algorithm convergence.
This paper tackles over-certainty in test-time adaptation models, proposing a solution to improve calibration.
problem Over-certainty in predictions caused by domain shifts, leading to misplaced trust.
method Introduces a certainty regularizer that dynamically adjusts pseudo-label confidence based on backbone entropy and logit norm.
result Achieves state-of-the-art performance in terms of Expected Calibration Error and Negative Log Likelihood, while maintaining accuracy.
A framework assesses the trustworthiness of probabilistic classifiers using local calibration error.
problem Assessing the trustworthiness of probabilistic classifiers beyond traditional metrics.
method I-trustworthy framework linking local calibration to trustworthiness; Kernel Local Calibration Error (KLCE) method for hypothesis testing.
result The effectiveness of the proposed test statistic demonstrated through simulated and real-world datasets.
This research improves uncertainty estimation for medical predictions, enhancing model trust and decision support.
problem Improving model uncertainty estimation for rare medical conditions.
method Developed and refined heuristics for selecting uncertainty estimation techniques, distinguishing them by clinical use-case. Also, compared ensembles vs. auto-encoders for detecting out-of-domain examples.
result Auto-encoders outperform ensembles in detecting out-of-domain examples, highlighting their importance for medical tabular data.
The paper certifies AI reliability via sampling and calibration, providing exact guarantees.
problem Ensuring trust in black-box AI systems' outputs.
method Self-consistency sampling and conformal calibration.
result Reliability levels derived from these methods offer finite-sample guarantees.
With rapid adoption of deep learning in critical applications, the question of when and how much to trust these models often arises, which drives the need to quantify the inherent uncertainties. While identifying all sources that account for the stochasticity of models is challenging, it is common to augment prediction…
Statsformer validates and adapts LLM-derived semantic priors for improved supervised learning.
problem Unreliable semantic priors from LLMs can degrade supervised learning performance.
method Adapts LLM-derived feature scores into a family of learner-specific prior-injection mechanisms, calibrating their influence using out-of-fold validation.
result Improves prediction performance by adaptively downweighting unreliable LLM priors, ensuring a guardrailed statistical learning system.
The paper proposes a framework to calibrate multi-agent simulation models from output series using Bayesian optimization.
problem Calibrating multi-agent simulation models from observable output series.
method Novel eligibility set concept, two-sample Kolmogorov-Smirnov test with Bonferroni correction, Bayesian optimization (BO), and trust-region BO (TuRBO).
result Demonstrated the efficiency of the proposed framework using numerical experiments.
Proposes a new trust framework for AI models to maximize utility.
problem Concerns over bias and discrimination in predictive models.
method Introduces a novel trust framework inspired by philosophy, focusing on maximizing Bayes utility.
result Properly-ranked models are inherently U-trustworthy. Benchmark assesses fairness in algorithmic uncertainty, revealing consistent and calibrated estimates improve fairness.
problem Challenges in managing uncertainty in fairness evaluations for predictive algorithms.
method Introduces FairlyUncertain, an axiomatic benchmark for evaluating uncertainty in fairness.
result Consistent and calibrated uncertainty estimates improve fairness without explicit fairness interventions.
New research shows calibration error is flawed when dealing with model uncertainty.
problem Current model evaluation techniques conflate model uncertainty with aleatoric uncertainty.
method Posterior predictive checks to evaluate deep learning models.
result Calibration error and variants are incorrect when model uncertainty is present.
New trust matrix quantifies breakdowns in deep neural networks.
problem Understanding trust breakdowns in deep learning models.
method Introduces trust matrix and conditional trust densities to analyze deep neural networks.
result Trust matrices reveal areas needing improvement for deep neural networks.
Supporting model interpretability for complex phenomena where annotators can legitimately disagree, such as emotion recognition, is a challenging machine learning task. In this work, we show that explicitly quantifying the uncertainty in such settings has interpretability benefits. We use a simple modification of a cla…
Stablecoin system improves resilience to extreme market events.
problem Vulnerability of stablecoins to extreme volatility and adversarial attacks.
method MVF-Composer uses multi-agent simulations to stress-test and down-weight manipulative signals.
result Reduces peak peg deviation by 57% and mean recovery time by 3.1x under adversarial conditions.
Two multifidelity trust-region methods use low-fidelity models for efficient optimization.
problem Efficiently solving complex optimization problems with limited data.
method Sketched Trust-Region (STR) and SVD Trust-Region (SVDTR) methods using low-fidelity models.
result Potential gain in efficiency demonstrated through numerical examples.
Study recommends PLM choices for minimizing calibration error in NLP tasks.
problem Minimizing calibration error in PLM-based NLP predictions.
method Compared various options for PLM encoding, size, uncertainty quantifier, and fine-tuning loss.
result Recommendations for a well-calibrated PLM-based prediction pipeline.
Trust-aware MAB improves learning performance by accounting for human deviation.
problem Learning performance suffers when humans deviate from recommended policies due to lack of trust.
method Integrates a dynamic trust model into MAB framework, establishing minimax regret and proposing a two-stage trust-aware procedure.
result Proves near-optimal statistical guarantees for trust-aware MAB algorithms.
Formalizes weak and strong verification for LLMs, controlling errors without assumptions.
problem Balancing cost and reliability in reasoning with LLMs.
method Formalizes weak-strong verification policies, introduces metrics, develops online algorithm.
result Optimal policies admit a two-threshold structure, and calibration and sharpness govern value of weak verifiers.
This work shows fine-tuning improves LLM uncertainty estimates efficiently.
problem Ensuring high-stakes predictions from large language models.
method Fine-tuning on a small dataset of correct and incorrect answers.
result Fine-tuned LLMs produce good generalization and tractable uncertainty estimates.
Safe autonomous decisions made with machine learning predictions using Conformal Decision Theory.
problem Safe decisions from imperfect machine learning predictions.
method Conformal Decision Theory framework for producing safe decisions.
result Safe decisions with provable statistical guarantees of low risk.
SPI uses synthetic data to improve predictive inference efficiency.
problem Inefficient predictive inference with scarce calibration data.
method Integrates synthetic data to align nonconformity scores and improve coverage guarantees.
result SPI yields substantially tighter and more informative prediction sets.
We explore trust in a relatively new area of data science: Automated Machine Learning (AutoML). In AutoML, AI methods are used to generate and optimize machine learning models by automatically engineering features, selecting models, and optimizing hyperparameters. In this paper, we seek to understand what kinds of info…
Optimizes data acquisition in high-dimensional Bayesian optimization.
problem Suboptimal data acquisition in high-dimensional Bayesian optimization tasks.
method Utility-calibrated variational inference to align approximations with BO goals.
result Optimal data acquisition decisions under a limited computational budget.
Study explores fairness in financial deep learning through multi-scale trust quantification.
problem Ensuring fairness in financial deep learning models, especially under regulatory compliance.
method Conducts multi-scale trust quantification on a deep neural network for credit card default prediction.
result Demonstrates the feasibility and utility of multi-scale trust quantification for financial deep learning fairness.
Accurately learning from user data while providing quantifiable privacy guarantees provides an opportunity to build better ML models while maintaining user trust. This paper presents a formal approach to carrying out privacy preserving text perturbation using the notion of dx-privacy designed to achieve geo-indistingui…
Pipeline combines ETF preprocessing with tabular model for cross-modal inference.
problem Transferability of tabular models across different modalities.
method Fixed comparison object, ETF preprocessing, in-context inference.
result Pipeline is broadly competitive, runs faster, and produces well-calibrated probabilities.
New method calibrates diffusion models for image regression tasks.
problem Ensuring reliability of diffusion models for critical applications.
method Risk-Controlling Prediction Sets (RCPS) with convex optimization.
result Calibrated entrywise intervals and risk control with minimal mean interval length.
DeFi doesn't fully remove trust, showing run risk and personal character's importance.
problem The need for trust in DeFi despite its code-based approach.
method Natural experiment revealing identities of DeFi participants, including a criminal.
result DeFi doesn't fully remove trust, showing run risk and personal character's relevance.