This work proposes using Conformal Prediction to improve OOD detection scores and vice versa.
problem Improper evaluation of OOD detection scores due to finite sample size.
method Defining new conformal AUROC and FRP@TPR95 metrics and using OOD scores as non-conformity scores.
result Improved evaluation metrics and better interpretation of OOD scores.
Neural networks classify OOD images by their nearest neighbor in training data.
problem Understanding out-of-distribution prediction behavior of neural networks.
method Nearest category generalization (NCG) measure to assess OOD prediction accuracy.
result Adversarially robust networks have higher NCG accuracy than natural training, indicating local regularization impacts decision regions.
Gaussian process detects OOD data without needing OOD samples.
problem Overconfident predictions from DNNs on OOD data.
method Gaussian process for uncertainty quantification and decision boundary.
result Outperforms state-of-the-art methods in OOD detection.
ProHOC detects OOD samples in class hierarchies, predicting them to correct internal nodes.
problem Binary OOD detection ignores semantic relationships between OOD and ID classes.
method Probabilistic hierarchical model using multi-depth networks trained for ID classification.
result ProHOC effectively classifies OOD samples to their correct internal nodes in class hierarchies.
PostNet predicts uncertainty without OOD data, improving OOD detection and calibration.
problem Accurate uncertainty estimation for safe systems.
method PostNet uses Normalizing Flows to learn individual posterior distributions over predicted probabilities.
result PostNet achieves state-of-the-art results in OOD detection and uncertainty calibration.
When presented with Out-of-Distribution (OOD) examples, deep neural networks yield confident, incorrect predictions. Detecting OOD examples is challenging, and the potential risks are high. In this paper, we propose to detect OOD examples by identifying inconsistencies between activity patterns and class predicted. We …
OOD detection methods often misidentify OOD points, leading to ineffective safety improvements.
problem Improving model safety through OOD detection methods often leads to incorrect identification of out-of-distribution points.
method Re-examine popular OOD detection procedures based on predictive uncertainty or features of supervised models trained on in-distribution data.
result Popular OOD detection methods incorrectly conflate high uncertainty and far feature-space distance with being out-of-distribution.
Proposes CSG model to separate semantic and variation factors for OOD prediction.
problem Out-of-distribution examples cause conventional models to mix semantic and variation factors, leading to poor performance.
method Causal Semantic Generative model (CSG) based on causal reasoning, using variational Bayes for efficient learning and prediction.
result CSG can identify semantic factor and improve OOD prediction performance.
Study shows gMPNNs struggle with OOD link prediction in larger test graphs.
problem Inductive out-of-distribution link prediction in larger test graphs.
method Theoretical analysis and development of a gMPNN with structural pairwise embeddings.
result Structural node embeddings from gMPNNs converge to random guessing as test graphs grow.
Framework reduces simplicity bias in NNs, improving OOD generalization and robustness.
problem Simplicity bias in deep learning models leads to biased predictions and poor OOD generalization.
method Proposes a framework that regularizes conditional mutual information to encourage use of diverse features.
result Demonstrates effectiveness in various settings, enhancing OOD generalization and robustness.
Proposes a new method to measure epistemic uncertainty in Bayesian neural networks.
problem Measuring epistemic uncertainty in Bayesian neural networks for out-of-distribution detection.
method Proposes measuring disagreement between logits and their pre-softmax counterparts as an epistemic uncertainty measure.
result Proposed epistemic uncertainty scores outperform mutual information and equal predictive entropy performance.
By design, discriminatively trained neural network classifiers produce reliable predictions only for in-distribution samples. For their real-world deployments, detecting out-of-distribution (OOD) samples is essential. Assuming OOD to be outside the closed boundary of in-distribution, typical neural classifiers do not c…
Bayesian neural networks fail at OOD detection, revealing fundamental issues.
problem The ability of Bayesian neural networks to detect out-of-distribution data.
method Empirical study of Bayesian inference in neural networks, including infinite-width and finite-width cases using Hamiltonian Monte Carlo.
result Bayesian inference in common neural network architectures does not lead to good OOD detection.
HOoD detects near-out-of-distribution groups in correlated biomedical assays.
problem Detecting near-out-of-distribution cases in biased or incomplete data.
method Projects correlated measurements through a trained model and uses permutation-based hypothesis tests.
result HOoD reliably identifies OoD groups, outperforming other detectors.
Discriminative neural networks offer little or no performance guarantees when deployed on data not generated by the same process as the training distribution. On such out-of-distribution (OOD) inputs, the prediction may not only be erroneous, but confidently so, limiting the safe deployment of classifiers in real-world…
Improves uncertainty estimation and OOD detection in neural networks.
problem Accurate uncertainty estimation and OOD detection in neural networks.
method Investigates one-vs-all and distance-based logit representations for probabilities.
result One-vs-all formulations improve calibration without additional complexity.
In-N-Out improves model robustness to out-of-distribution data.
problem Learning robust models with few in-distribution labeled examples.
method Pre-training with auxiliary information and self-training with pseudolabels.
result In-N-Out outperforms auxiliary inputs or outputs alone on both in-distribution and OOD error.
Framework detects out-of-distribution inputs in regression and survival analysis.
problem Limited OOD detection for regression and survival analysis.
method Model-aware and subspace-aware variable prioritization.
result Consistent improvements over existing methods in synthetic and real data.
Bayesian neural networks fail at out-of-distribution detection, revealing fundamental issues.
problem Out-of-distribution detection with Bayesian neural networks.
method Study of Bayesian inference with function space priors and comparison to Gaussian processes.
result Bayesian inference with function space priors does not lead to good OOD detection.
Discriminatively trained neural classifiers can be trusted, only when the input data comes from the training distribution (in-distribution). Therefore, detecting out-of-distribution (OOD) samples is very important to avoid classification errors. In the context of OOD detection for image classification, one of the recen…
NADS improves OoD detection accuracy by 57%.
problem Uncertainty in machine learning models when encountering out-of-distribution data.
method NADS searches for a distribution of architectures that perform well on a given task, optimizing a stochastic OoD detection objective.
result NADS achieves up to 57% improvement in accuracy over state-of-the-art methods.
Framework tackles OOD challenges in molecule property prediction by modeling environments.
problem Challenges in modeling OOD samples for molecule property prediction.
method Soft causal learning framework incorporating chemistry theories and cross-attention mechanisms.
result Demonstrates well generalization ability on seven datasets.
ProSub uses angles in feature space to classify data as in- or out-of-distribution.
problem Open-set semi-supervised learning with unknown classes.
method Probabilistic approach based on angles in feature space, estimating conditional distributions of scores.
result ProSub achieves state-of-the-art performance on benchmark problems.
Deep neural networks are known to achieve superior results in classification tasks. However, it has been recently shown that they are incapable to detect examples that are generated by a distribution which is different than the one they have been trained on since they are making overconfident prediction for Out-Of-Dist…
As deep learning methods form a critical part in commercially important applications such as autonomous driving and medical diagnostics, it is important to reliably detect out-of-distribution (OOD) inputs while employing these algorithms. In this work, we propose an OOD detection algorithm which comprises of an ensembl…
New method detects out-of-distribution samples in regression tasks.
problem Detecting instances far from training data in regression models.
method Estimating predictor entropy based on nearest neighbors and generative models.
result A new method for robust OOD detection in regression tasks.
Proposes DCV-ROOD framework for robust OOD detection evaluation.
problem Ensuring reliable OOD detection methods under diverse conditions.
method Dual Cross-Validation (DCV) adapted for OOD detection evaluation.
result Demonstrates fast convergence to true performance of OOD detection methods.
Study shows different trajectory prediction models generalize better under OoD conditions.
problem Comparing trajectory prediction models' robustness across different datasets.
method Training models on Argoverse 2 and testing on Waymo Open Motion, and vice versa, with various augmentation strategies.
result Smallest model with highest inductive bias performs best in OoD generalization.
New framework detects out-of-distribution samples efficiently.
problem Detecting samples from different distributions in deep neural networks.
method Statistical hypothesis testing framework combining evidence from entire network.
result Framework maintains Type I Error and achieves comparable results to state-of-the-art methods.
It has recently been shown that ReLU networks produce arbitrarily over-confident predictions far away from the training data. Thus, ReLU networks do not know when they don't know. However, this is a highly important property in safety critical applications. In the context of out-of-distribution detection (OOD) there ha…
Paper tackles SCOD problem with optimal strategy and empirical validation.
problem Designing reliable prediction models abstaining from uncertain predictions.
method Bayes classifier for ID data and stochastic linear selector in 2D space.
result POSCOD method outperforms existing OOD methods.
A simple method flags images as out-of-distribution based on their distance to nearest neighbors.
problem Detecting images not aligned with a trained model's in-distribution data.
method Flag images as OOD if their average distance to K nearest neighbors is large in the classifier's representation space.
result Simple methods can outperform more complex ones when considering learned representations.
Paper aims to ensure reliable detection of out-of-distribution data with certifiable worst-case guarantees.
problem Deep neural networks are overconfident with OOD inputs, posing safety risks.
method Enforces low confidence and bounds in an l∞-ball around OOD points using interval bound propagation (IBP). result Certifiable worst-case guarantees for OOD detection are possible without significant loss in accuracy.
Robustness to out-of-distribution (OOD) data is an important goal in building reliable machine learning systems. Especially in autonomous systems, wrong predictions for OOD inputs can cause safety critical situations. As a first step towards a solution, we consider the problem of detecting such data in a value-based de…
A new OOD detection method OTOD uses optimal transport theory to improve model performance.
problem Detecting unknown samples in real-world machine learning models.
method OTOD uses optimal transport theory to calculate an OOD score combining features, logits, and softmax probability space.
result OTOD outperforms state-of-the-art methods by significant margins on benchmarks.
Robust reinforcement learning agents generalize well to out-of-distribution settings using pretrained representations.
problem Achieving sample-efficient reinforcement learning agents that generalize to real-world settings.
method Trained 240 representations and 10,000 RL policies on a simulated robotic setup, evaluating different pretrained VAE-based representations' effects on OOD generalization.
result Many reinforcement learning agents are surprisingly robust to realistic distribution shifts, including sim-to-real cases.
CGRL improves graph neural networks' OOD generalization by blocking spurious correlations.
problem Graph Neural Networks struggle with out-of-distribution data due to learning spurious correlations.
method Formulates a causal graph, uses backdoor adjustment, and introduces a loss replacement strategy.
result Significantly improves OOD generalization of GNNs, stabilizing mutual information learning.
TULiP estimates uncertainty for deep learning models safely.
problem Reliable uncertainty estimation for deep learning models in the open world.
method TULiP considers a hypothetical perturbation, bounds its effect, and computes uncertainty from sampled predictions.
result TULiP achieves state-of-the-art performance in OOD detection benchmarks.
WOOD detects out-of-distribution samples using Wasserstein distance.
problem Detecting samples from different distributions in neural networks.
method WOOD defines a Wasserstein-distance-based score to evaluate dissimilarity and solves an optimization problem.
result WOOD consistently outperforms other OOD detection methods.
FROB model improves robustness and reliable confidence for few-shot OoD detection.
problem Challenges in few-shot classification and OoD detection due to limited samples and adversarial attacks.
method FROB model combines support boundary generation and few-shot Outlier Exposure (OE) for improved robustness and reliable confidence.
result FROB achieves generalization to unseen OoD and maintains robustness independent of few-shot number.
Bayesian methods improve OoD detection in deep networks.
problem Detecting Out-of-Distribution (OoD) inputs in deep neural networks.
method Three Bayesian inference approaches applied to VAE weights.
result Improved OoD detection scores over benchmarks.
New method tackles OOD robustness with a single additional variable.
problem Out-of-distribution generalization with unobserved confounders.
method Identifiability assumptions using a single additional variable.
result Superior empirical performance on benchmark tasks.
ATOM improves robust OOD detection by mining informative auxiliary examples.
problem Robust OOD detection in open-world settings is challenging due to adversarial inputs.
method ATOM combines adversarial training with outlier mining to improve robustness.
result ATOM achieves state-of-the-art performance in OOD detection, reducing FPR by up to 57.99%.
Paper investigates learnability of OOD detection under various conditions.
problem Learnability of OOD detection under different scenarios.
method Investigates PAC learning theory, proves impossibility theorems, and provides necessary and sufficient conditions.
result Some conditions for learnability of OOD detection may not hold in practical scenarios.
Paper investigates OOD detection learnability under various conditions.
problem Learnability of OOD detection under diverse and unknown test data.
method PAC learning theory applied to OOD detection, proving impossibility theorems and necessary conditions.
result Some conditions for learnability hold in practical scenarios.
New method improves OOD detection without sacrificing generalization.
problem Improving OOD detection models that also generalize well.
method Decouples uncertainty learning from Bayesian perspective.
result Achieves state-of-the-art OOD detection performance.
LoD improves model safety by integrating unlabeled wild data, reducing OOD misclassification.
problem Improving model safety and reliability using unlabeled wild data containing both in-distribution and out-of-distribution samples.
method Intentionally label-noisifying unlabeled wild data to enable joint learning of labeled ID and OOD data, distinguishing losses between ID and OOD samples.
result LoD framework achieves superior OOD detection without requiring thresholds, improving model safety.
The paper improves uncertainty quantification for node classification using distance-based regularization.
problem Uncertainty in deep learning models, especially for node classification tasks.
method Graph posterior networks (GPNs) with UCE loss function, followed by a distance-based regularization.
result The proposed distance-based regularization outperforms state-of-the-art methods in OOD detection and misclassification detection.