Proposes a new criterion for reliable uncertainty estimation in deep neural networks.
problem Inability of existing approaches to provide reliable uncertainty estimates for deep neural networks.
method Develops a density uncertainty layer architecture that satisfies the proposed criterion.
result Density uncertainty layers provide more reliable uncertainty estimates and robust out-of-distribution detection.
GCAE uses density estimation to achieve reliable disentanglement in latent space.
problem Disentangled learning representations suffer from reliability issues.
method GCAE uses Gaussian Channel Autoencoder with Dual Total Correlation (DTC) to avoid the curse of dimensionality.
result GCAE achieves highly competitive and reliable disentanglement scores.
AL-SPCE improves reliability analysis for complex systems with active learning and SPCE.
problem Efficiently analyzing reliability of complex, computationally expensive models with intrinsic randomness.
method Active learning framework using stochastic polynomial chaos expansions (SPCE) to reduce computational burden.
result AL-SPCE maintains high accuracy in reliability estimates while significantly improving efficiency.
Paper proposes real-time VaR estimation using quantile regression forest with conformal calibration.
problem Real-time estimation of Value at Risk (VaR) in rapidly changing markets.
method Quantile regression forest trained offline, real-time VaR estimates via observed risk factors, conformalized estimator for reliability.
result The proposed method provides reliable real-time VaR estimates.
Survey and framework for efficient active learning in structural reliability.
problem Efficiently solving complex structural reliability problems.
method Generalized modular framework combining surrogate model, reliability estimation algorithm, learning function, and stopping criterion.
result 39 strategies for solving 20 reliability benchmark problems, highlighting the importance of surrogates and algorithms.
New method improves reliability of selecting individuals based on predicted treatment effects.
problem Reliability of selecting individuals based on predicted conditional average treatment effects (CATE) is unreliable.
method Denoised Conformal Alignment, combining proxy errors, variance estimation, and Benjamini-Hochberg selection.
result Significantly improved power in selecting individuals while maintaining false discovery rate control.
Study defines and optimizes bank reliability using LR and PSO.
problem Lack of reliability concept in financial services.
method Logistic Regression (LR) for initial estimation, Particle Swarm Optimization (PSO) for optimization.
result Optimal financial ratios maximize bank reliability.
This study compares two methods for uncertainty estimation in CNNs, finding Conformal Prediction more reliable.
problem CNNs often overestimate uncertainty, leading to unreliable predictions.
method Bayesian approximation via Monte Carlo Dropout and Conformal Prediction.
result Conformal Prediction produces more reliable uncertainty estimates than Monte Carlo Dropout.
We present a study on reinforcement learning (RL) from human bandit feedback for sequence-to-sequence learning, exemplified by the task of bandit neural machine translation (NMT). We investigate the reliability of human bandit feedback, and analyze the influence of reliability on the learnability of a reward estimator,…
PS-DME evaluates model performance and reliability after data-dependent selection.
problem Evaluating model performance and reliability when data is used for selection and evaluation.
method Post-selection distributional model evaluation (PS-DME) using e-values to control false coverage rate.
result PS-DME provides reliable comparison of model configurations across different reliability levels.
A new estimator improves financial econometrics by providing reliable inference.
problem Poor performance of standard regression methods in financial economics with thick-tailed predictors.
method Developed an unbiased, consistent, and asymptotically normal estimator for linear regression.
result The new method delivers reliable inference under heteroskedasticity and quantile regression.
To use machine learning in high stakes applications (e.g. medicine), we need tools for building confidence in the system and evaluating whether it is reliable. Methods to improve model reliability often require new learning algorithms (e.g. using Bayesian inference to obtain uncertainty estimates). An alternative is to…
Estimating individual level treatment effects (ITE) from observational data is a challenging and important area in causal machine learning and is commonly considered in diverse mission-critical applications. In this paper, we propose an information theoretic approach in order to find more reliable representations for e…
CPA framework assesses conditional validity of conformal prediction.
problem Challenges in evaluating conditional validity of conformal prediction.
method Reframes conditional coverage evaluation as a supervised learning task.
result Establishes convergence rates and proves CVI consistency.
Bayesian method improves reliability of BERT for hate speech detection.
problem Reliable detection of hate speech in user-generated content.
method Bayesian approach using Monte Carlo dropout in transformer models.
result Monte Carlo dropout provides well-calibrated reliability estimates.
In this paper we focus on the problem of assigning uncertainties to single-point predictions. We introduce a cost function that encodes the trade-off between accuracy and reliability in probabilistic forecast. We derive analytic formula for the case of forecasts of continuous scalar variables expressed in terms of Gaus…
Improves reliability of medical diagnosis uncertainty estimates.
problem Label uncertainty in medical diagnosis.
method Post-hoc alpha-calibration method for neural network classifiers. result Significantly enhances reliability of uncertainty estimates.
In many instances, information on engineering systems can be obtained through measurements, monitoring or direct observations of system performances and can be used to update the system reliability estimate. In structural reliability analysis, such information is expressed either by inequalities (e.g. for the observati…
The aim of the present paper is to develop a strategy for solving reliability-based design optimization (RBDO) problems that remains applicable when the performance models are expensive to evaluate. Starting with the premise that simulation-based approaches are not affordable for such problems, and that the most-probab…
Active Kriging Monte Carlo simulation method with conformal certification for failure probability estimation
problem Failure probability estimation in structural reliability analysis
method Active learning framework with conformal prediction
result Improved uncertainty quantification and reliability of failure probability estimates
Empirical error estimates improve graph sparsification reliability.
problem Uncertainty in sparsification error limits downstream computations reliability.
method Data-driven approach to compute empirical error estimates.
result Empirical error estimates provide theoretical guarantees and are computationally feasible.
In the field of structural reliability, the Monte-Carlo estimator is considered as the reference probability estimator. However, it is still untractable for real engineering cases since it requires a high number of runs of the model. In order to reduce the number of computer experiments, many other approaches known as …
Bayesian framework improves deep classifier reliability.
problem Overconfident models under dataset shift.
method Bayesian inference with out-of-distribution data augmentation.
result Reliable uncertainty estimates for deep classifiers.
Paper improves ETF tail-risk monitoring reliability.
problem Unreliable ETF risk monitoring under degraded data.
method Combines quality checks, prediction, scoring, and adjustment.
result Improves tail-risk monitoring, especially during stressed periods.
Proposes glocal hypergradient estimation for hyperparameter optimization.
problem Combining reliability and efficiency in hyperparameter optimization.
method Uses Koopman operator theory to approximate global hypergradients from local ones.
result Achieves both reliability and efficiency in hyperparameter optimization.
Method quantifies sensitivity of reliability analysis to uncertainty sources.
problem Computational expense in reliability analysis of complex models.
method Gaussian process surrogate model, active learning, sensitivity analysis.
result Reduces main source of error in estimating rare event probabilities.
Improved VAE model enhances uncertainty estimation for out-of-distribution samples.
problem VAEs assign higher likelihood to out-of-distribution inputs.
method INCPVAE integrates noise contrastive prior into VAEs for reliable uncertainty estimation.
result INCPVAE outperforms standard VAEs in uncertainty estimation for OOD inputs.
Framework for uncertainty estimation in training parameters.
problem Estimating uncertainty in training parameters.
method Marginalizing hyperparameters as random variables, investigating various forms of marginalisation.
result Some marginalisations can reliably estimate uncertainty without extensive tuning.
New method calibrates uncertainty estimates for image classifiers without labeled data.
problem Uncertainty estimates for modern classifiers are unreliable without labeled calibration data.
method Calibrates uncertainty estimates using unlabeled examples for distribution shifts.
result Proposes a method that provides excellent uncertainty estimates under natural distribution shifts.
A new framework evaluates LLMs by considering judge reliability.
problem Evaluating LLMs without ground truth labels can lead to biased results.
method Introduces judge-specific discrimination parameters and estimates model quality and judge reliability.
result Improves agreement with human preferences and produces calibrated uncertainty quantification.
A new method estimates rare failure events in complex systems.
problem Estimating the probability of rare failure events in non-linear systems.
method Stochastic Spectral Embedding (SSE) combined with modifications for efficient rare event estimation.
result Rare failure probability decomposed into conditional probabilities for easier computation.
F-PACOH improves meta-learners' reliability in uncertain regions.
problem Overconfident uncertainty estimates in meta-learning.
method Meta-learning priors as stochastic processes in function space, directly steering predictions towards high epistemic uncertainty.
result Significantly outperforms other meta-learners in Bayesian Optimization.
New method combines neural networks with Monte Carlo for complex system reliability.
problem Estimating small failure probabilities in complex systems.
method Subset Simulation with Hamiltonian Neural Networks.
result High acceptance rates and computational efficiency in low-probability regions.
The ACCRU framework improves probabilistic forecasts by capturing input-dependent uncertainty.
problem Uncertainty in deterministic predictions, especially for skewed and non-Gaussian errors.
method Neural network trained with a loss function balancing accuracy and reliability to learn input-dependent, non-Gaussian uncertainty distributions.
result Improves probabilistic forecasts relative to existing methods, capturing skewed and non-Gaussian errors.
DeepSKA provides interpretable, reliable neural approximations for SRNs.
problem Estimating expected outputs in SRNs is computationally challenging and unreliable.
method Jointly achieves interpretability, reliability, and computational gains with DeepSKA.
result Delivers accurate predictions and substantial efficiency improvements across SRNs.
LogDet estimator improves entropy estimation in neural networks.
problem Inconsistent observations and diversified interpretation in neural networks.
method Proposes LogDet estimator for reliable entropy approximation.
result LogDet estimator overcomes distributional diversity issues.
The paper highlights the importance of model misspecification in uncertainty estimation.
problem The reliability of uncertainty estimates in machine learning models under model misspecification.
method Thought experiments and literature review.
result Model misspecification should be given more attention in uncertainty estimation.
New method improves reliability of depth estimation models.
problem Uncertainty quantification in large-scale vision models.
method Parameter-efficient Bayesian neural networks with PEFT methods.
result Combining PEFT methods with Bayesian inference enhances predictive performance.
We consider the problem of accurately estimating the reliability of workers based on noisy labels they provide, which is a fundamental question in crowdsourcing. We propose a novel lower bound on the minimax estimation error which applies to any estimation procedure. We further propose Triangular Estimation (TE), an al…
As a result of social network popularity, in recent years, hate speech phenomenon has significantly increased. Due to its harmful effect on minority groups as well as on large communities, there is a pressing need for hate speech detection and filtering. However, automatic approaches shall not jeopardize free speech, s…
Entropy-based GP adaptive design improves failure probability estimation.
problem Limited accuracy in failure probability estimation due to model evaluation costs.
method Entropy-based Gaussian process (GP) adaptive design combined with multifidelity importance sampling (MFIS).
result More accurate failure probability estimates and higher confidence.
The paper presents a method to assign accurate and calibrated uncertainties to deterministic model predictions.
problem Assigning uncertainties to deterministic model predictions.
method Transforming deterministic predictions into probabilistic ones using a cost function that balances accuracy and reliability.
result The method improves the reliability of probabilistic predictions without sacrificing accuracy.
Latent truth discovery, LTD for short, refers to the problem of aggregating ltiple claims from various sources in order to estimate the plausibility of atements about entities. In the absence of a ground truth, this problem is highly challenging, when some sources provide conflicting claims and others no claims at all.…
Obtaining reliable uncertainty estimates of neural network predictions is a long standing challenge. Bayesian neural networks have been proposed as a solution, but it remains open how to specify their prior. In particular, the common practice of an independent normal prior in weight space imposes relatively weak constr…
Improves neural network performance by dynamically adjusting model weights based on source reliability.
problem Training neural networks on data from unreliable sources leads to poor performance.
method Dynamic re-weighting strategy using likelihood tempering to adjust model weights based on estimated source reliability.
result Significant improvement in model performance when trained on mixtures of reliable and unreliable data sources.
Algorithm optimizes and infers performance online, improving reliability.
problem Balancing optimization and statistical inference in complex systems.
method Online algorithm that provides consistent performance variance and confidence intervals.
result Online consistent estimator for performance variance and asymptotic confidence intervals.
Proposes a method to estimate treatment effects using instruments.
problem Estimating treatment effects from observational data is challenging when unconfoundedness is violated.
method Leverages instruments to estimate bounds on conditional average treatment effect (CATE) through a mapping to a discrete representation space and a two-step procedure.
result Demonstrates theoretical validity and reduced estimation variance in finite-sample settings.
The paper proposes a method for better uncertainty estimation in neural networks.
problem Estimating predictive uncertainty in neural networks is crucial but challenging.
method The paper proposes a function-space variational inference method to infer a posterior distribution over functions.
result The proposed method leads to state-of-the-art uncertainty estimation and predictive performance.