New metrics improve understanding of predictive system reliability.
problem Evaluating conditional coverage of predictive systems.
method Casting conditional coverage estimation as a classification problem, using excess risk of the target coverage (ERT) metrics.
result Modern classifiers provide higher statistical power for estimating conditional coverage.
CPA framework assesses conditional validity of conformal prediction.
problem Challenges in evaluating conditional validity of conformal prediction.
method Reframes conditional coverage evaluation as a supervised learning task.
result Establishes convergence rates and proves CVI consistency.
The paper addresses the reliability of conformal prediction under covariate shift.
problem Ensuring reliable prediction sets under covariate shift.
method Derives upper bounds on training-conditional coverage.
result Offers PAC guarantees for conformal prediction methods.
Enhanced metrics improve generative model evaluation reliability.
problem Lack of reliable quality metrics for generative models.
method Introduce Clipped Density and Clipped Coverage metrics.
result Metrics prevent out-of-distribution samples from biasing quality scores.
Develop a framework to evaluate the reliability of probabilistic emulation of physical systems.
problem Evaluating the reliability of probabilistic forecasts in physical systems.
method Developing a framework to assess the reliability of probabilistic emulation across diverse 2D spatiotemporal systems.
result CRPS-trained ensembles achieve more reliable uncertainties on single-step prediction and autoregressive rollouts.
Method certifies edge predictions with cloud-level reliability.
problem Ensuring reliability of edge intelligence models.
method Conformal alignment-based cascading mechanism.
result Certifies conditional coverage with user control over risk level.
DS-CP improves reliability of uncertainty quantification for large language models under domain shift.
problem Overconfident and factually incorrect outputs (hallucinations) from large language models.
method Adapts conformal prediction to large language models under domain shift by reweighting calibration samples.
result DS-CP delivers more reliable coverage than standard conformal prediction, especially under substantial distribution shifts.
CF-GNN provides reliable uncertainty estimates for graph data.
problem Lack of rigorous uncertainty estimates in GNNs.
method Conformal prediction extended to graph-based models (CF-GNN).
result CF-GNN produces prediction sets with provable coverage probability.
Proposes a new method for reliable treatment effect interval estimates.
problem Uncertainty quantification in treatment effect estimation.
method Conformal inference for counterfactuals and individual treatment effects.
result Achieves desired coverage with short intervals in various settings.
The paper certifies AI reliability via sampling and calibration, providing exact guarantees.
problem Ensuring trust in black-box AI systems' outputs.
method Self-consistency sampling and conformal calibration.
result Reliability levels derived from these methods offer finite-sample guarantees.
New method improves conditional coverage of conformal prediction.
problem Improving conditional coverage in conformal prediction.
method Trainable transformation of conformity scores to improve conditional coverage.
result Highly adaptive to local data structure, outperforming existing methods.
Devising indicative evaluation metrics for the image generation task remains an open problem. The most widely used metric for measuring the similarity between real and generated images has been the Fréchet Inception Distance (FID) score. Because it does not differentiate the fidelity and diversity aspects of the genera…
PS-DME evaluates model performance and reliability after data-dependent selection.
problem Evaluating model performance and reliability when data is used for selection and evaluation.
method Post-selection distributional model evaluation (PS-DME) using e-values to control false coverage rate.
result PS-DME provides reliable comparison of model configurations across different reliability levels.
The paper introduces recklessness to improve recommendation quality and quantity.
problem The reliability/coverage dilemma in recommender systems limits the number of recommended items.
method Incorporates a new term (recklessness) into matrix factorization-based recommender systems to address the dilemma.
result Recklessness improves the quantity and quality of recommendations by allowing for risk regulation.
Unified framework for reliable uncertainty quantification in RL.
problem Uncertainty quantification in high-stakes reinforcement learning.
method Unified conformal prediction framework integrating distributional RL and conformal calibration.
result Significantly improved coverage and reliability over standard methods.
TQA improves prediction intervals for time series data by adjusting quantiles for both cross-sectional and longitudinal coverage.
problem Constructing reliable prediction intervals for cross-sectional time series data.
method Temporal Quantile Adjustment (TQA) method that adjusts the quantile in Conformal Prediction to account for both cross-sectional and longitudinal coverage.
result TQA improves longitudinal coverage while preserving cross-sectional coverage, as validated through extensive experimentation.
Adaptive Quantum Conformal Prediction improves reliability of quantum machine learning predictions.
problem Quantum machine learning lacks robust uncertainty quantification methods.
method Adaptive Conformal Inference applied to quantum conformal prediction to maintain validity over time.
result AQCP achieves target coverage levels and is more stable than standard quantum conformal prediction.
New method provides reliable high-confidence prediction intervals for high-impact events.
problem High-impact events require very high confidence prediction intervals, but classical methods provide uninformative intervals.
method Bridge extreme value statistics and conformal prediction to provide reliable and informative prediction intervals.
result Provides reliable and informative prediction intervals with high-confidence coverage.
LPCI provides valid prediction intervals for longitudinal data.
problem Current conformal prediction methods for time series data lack cross-sectional coverage when applied to longitudinal datasets.
method Modeling residual data as a quantile fixed-effects regression problem, constructing prediction intervals with a trained quantile regressor.
result LPCI achieves valid cross-sectional coverage and outperforms existing benchmarks in terms of longitudinal coverage rates.
New method for time series prediction with uncertainty quantification.
problem Uncertainty quantification for multi-dimensional time series predictions.
method Flow-based conformal prediction for time series.
result Significantly smaller prediction sets with target coverage.
SPLICE generates accurate time-series imputations with reliable prediction intervals.
problem Lack of reliability guarantees in time-series imputation models.
method Modular framework combining latent generative imputation with distribution-free prediction intervals.
result SPLICE achieves lowest mean Load-only MSE and best CRPS on various datasets.
The paper improves off-policy evaluation in contextual bandits using conformal prediction.
problem Quantifying the performance of a target policy using data from a different behavior policy.
method Proposes a novel algorithm based on a PAC-valid conformal prediction framework to construct probably approximately correct prediction intervals.
result Establishes PAC-type bounds on coverage, improving theoretical guarantees.
New methods for distributed CP improve reliability in healthcare.
problem Post-hoc calibration of models in safety-critical domains.
method Quantile-based and histogram-based distributed conformal prediction via message passing.
result Improved reliability and coverage guarantees in decentralized settings.
HCC extends conformal prediction to handle class hierarchies, improving prediction reliability.
problem Uncertainty quantification in classification models with class hierarchy considerations.
method Formulates HCC as a constrained optimization problem, ensuring coverage guarantees with a smaller subset of candidate solutions.
result HCC produces more reliable prediction sets by leveraging class hierarchy information.
New method for probabilistic prediction sets with conditional validity.
problem Marginal coverage guarantee of existing methods.
method Combines conformal methods with approximate conditional validity.
result Consistently outperforms existing approaches in conditional coverage.
Develops optimal decision-making framework for uncertain counterfactuals.
problem Ensuring reliability of predictions in high-stakes decisions.
method Policy-Coupled Risk-Averse Conformal Prediction (PC-RACP).
result Optimal prediction sets for counterfactual decisions with valid coverage.
BCCP uses bandit feedback to provide reliable predictions with limited labeled data.
problem Limited labeled data and bandit feedback challenge online set-valued classification.
method BCCP uses stochastic gradient descent to train model and make set-valued inferences with unbiased estimation of true label.
result BCCP offers coverage guarantees on a class-specific granularity.
The paper introduces a method to assess machine translation quality with confidence intervals.
problem Evaluating the uncertainty and quality of machine translation.
method Utilizes conformal predictive distributions to produce prediction intervals with guaranteed coverage.
result The method outperforms a baseline on six language pairs in terms of coverage and sharpness.
CTI produces efficient prediction intervals with guaranteed coverage.
problem Efficient and reliable uncertainty quantification in regression.
method CTI estimates conditional density for interval length, then thresholds intervals based on this density.
result CTI achieves smaller prediction sets with guaranteed coverage compared to existing methods.
New methods adapt conformal prediction to unknown subpopulation shifts.
problem Failure of conformal prediction under unknown subpopulation shifts.
method Proposes new methods that adapt conformal prediction to unknown subpopulation shifts without explicit subpopulation labels.
result Ensures valid coverage guarantees without explicit knowledge of subpopulation structure.
A new framework bridges classical and machine learning methods for reliable inference from complex models.
problem Intractable likelihood functions in complex systems make classical statistics ineffective for likelihood-free inference.
method Likelihood-Free Frequentist Inference (LF2I) framework that combines classical statistics and machine learning.
result Valid confidence sets with near finite-sample validity can be constructed for any parameter value.
Conformal prediction fails to cover minority classes in imbalanced datasets, but a class-conditional fix improves coverage.
problem Conformal prediction fails to cover minority classes in imbalanced datasets, leading to poor performance on rare labels.
method Class-conditional conformal prediction to improve coverage of minority classes.
result Class-conditional conformal prediction restores minority coverage to target with a modest increase in prediction-set size.
CP improves robustness against distribution shift using physics-informed structural causal models.
problem Uncertainty in machine learning predictions under distributional shift.
method Physics-informed structural causal model (PI-SCM) to upper bound coverage difference.
result PI-SCM improves coverage robustness across confidence levels and test domains.
New method combines HQR and WACI for better time series prediction intervals.
problem Challenges in creating reliable prediction intervals for time series forecasting.
method Combining Heteroscedastic Quantile Regression (HQR) with Width-Adaptive Conformal Inference (WACI).
result Combined approach meets or surpasses typical benchmarks for validity and efficiency.
Improved multivariate conformal prediction by standardizing residuals.
problem Weak conditional coverage in heteroskedastic multivariate settings.
method Natural extension of univariate normalization to multivariate setting, whitening residuals and standardizing local variance.
result Standardized residuals yield asymptotic conditional coverage under certain distributions.
CAOS aggregates multiple one-shot predictors for efficient uncertainty quantification.
problem Lack of principled uncertainty quantification in one-shot prediction.
method CAOS, a conformal framework that aggregates multiple one-shot predictors and uses a leave-one-out calibration scheme.
result CAOS produces smaller prediction sets with reliable coverage compared to split conformal baselines.
A new method for active learning works well across all label budgets.
problem Active learning methods perform poorly in both low and high label budgets.
method Uncertainty Herding: a simple, computationally fast method that optimizes uncertainty coverage.
result Uncertainty Herding nearly optimizes distribution-level coverage and performs well across various active learning tasks.
New method calibrates classifier probabilities with guaranteed coverage.
problem Inaccurate probability estimates by classifiers in high-risk applications.
method Adaptive temperature scaling algorithm for conformal prediction.
result Improves calibration error measures and standard metrics across various tasks.
Though black-box predictors are state-of-the-art for many complex tasks, they often fail to properly quantify predictive uncertainty and may provide inappropriate predictions for unfamiliar data. Instead, we can learn more reliable models by letting them either output a prediction set or abstain when the uncertainty is…
New method prevents invalid inference after causal discovery.
problem Invalid inference after causal discovery.
method Developed tools for valid post-causal-discovery inference.
result Our method provides reliable coverage while achieving more accurate causal discovery.
Random forests are stable and provide reliable prediction intervals.
problem Stability and reliability of random forest prediction intervals.
method Established stability under mild conditions and proved coverage bounds.
result Non-asymptotic lower and upper bounds for prediction interval coverage.
Bayesian optimization enhanced with conformal prediction for better outcome reliability.
problem Uncertainty and model misspecification in Bayesian optimization.
method Conformal prediction to provide coverage guarantees and Bayesian optimization to select queries.
result Significant improvement in query coverage without sacrificing sample-efficiency.
Decision-calibrated prediction sets improve power system operations by reducing unnecessary costs.
problem Balancing operating costs and reliability in power systems with renewable uncertainty.
method Learn conditional prediction sets as sub-level sets of norm-based score functions, calibrate uncertainty sets based on reliability of downstream decisions.
result Decision-calibrated sets lead to more efficient operations with smaller uncertainty sets and lower costs compared to standard coverage-based calibration.
A reliable and accurate forecasting model for crop yields is of crucial importance for efficient decision-making process in the agricultural sector. However, due to weather extremes and uncertainties, most forecasting models for crop yield are not reliable and accurate. For measuring the uncertainty and obtaining furth…
The paper proposes a method for distribution-free prediction sets that adapt to unknown temporal changes.
problem Distribution-free prediction sets require reliable calibration data, which is often unavailable in real-world settings with temporal changes.
method The method selects an adaptive window to construct prediction sets, optimizing a bias-variance tradeoff.
result The method provides sharp coverage guarantees and is shown to be adaptive to temporal drift through numerical experiments.
A framework to quantify deployment risk in ML systems, especially for rare states.
problem Under-supported rare states in ML models lead to unreliable performance in unseen data.
method Blind-Spot Mass (B_n(tau)) using Good-Turing unseen-species estimation.
result Identifies and quantifies the risk of under-supported states in ML models.
Unified framework for causal inference with reliable uncertainty quantification.
problem Causal inference under unobserved confounding with unreliable uncertainty quantification.
method Deconditional Gaussian Process (DGP) framework for uncertainty-aware causal learning.
result Strong predictive performance and informative uncertainty quantification.
Contrast uses normalizing flows to create precise prediction regions for multi-dimensional outputs.
problem Generating reliable prediction regions for multi-dimensional outputs in supervised and unsupervised learning.
method Contrast uses normalizing flows to define nonconformity scores based on distances in latent space, creating sharp prediction regions.
result Contrast maintains guaranteed coverage probability and outperforms existing methods in generating accurate prediction regions.