Simple methods combine statistical tests for out-of-distribution detection.
problem Detecting data points not following the training distribution.
method Combining classical parametric tests (Rao's score test) and a typicality test.
result Combining Fisher's method of test statistics improves out-of-distribution detection accuracy.
Method enhances anomaly detection using contrastive learning and out-of-distribution data.
problem Improving anomaly detection in datasets with limited out-of-distribution data.
method Proposes a contrastive learning method that incorporates out-of-distribution data to enhance anomaly detection performance.
result The method significantly improves anomaly detection performance, even with limited out-of-distribution data.
Bayesian Gaussian Processes layer detects out-of-distribution data in medical imaging.
problem Detecting out-of-distribution data in medical imaging tasks.
method Parameter-efficient hierarchical convolutional Gaussian Processes in Wasserstein-2 space.
result Uncertainty estimates enable superior out-of-distribution detection compared to previous methods.
Deep learning models are known to be overconfident in their predictions on out of distribution inputs. This is a challenge when a model is trained on a particular input dataset, but receives out of sample data when deployed in practice. Recently, there has been work on building classifiers that are robust to out of dis…
Modern neural networks are very powerful predictive models, but they are often incapable of recognizing when their predictions may be wrong. Closely related to this is the task of out-of-distribution detection, where a network must determine whether or not an input is outside of the set on which it is expected to safel…
In-N-Out improves model robustness to out-of-distribution data.
problem Learning robust models with few in-distribution labeled examples.
method Pre-training with auxiliary information and self-training with pseudolabels.
result In-N-Out outperforms auxiliary inputs or outputs alone on both in-distribution and OOD error.
Deep learning provides a powerful tool for machine perception when the observations resemble the training data. However, real-world robotic systems must react intelligently to their observations even in unexpected circumstances. This requires a system to reason about its own uncertainty given unfamiliar, out-of-distrib…
Deep learning models are known to be overconfident in their predictions on out of distribution inputs. There have been several pieces of work to address this issue, including a number of approaches for building Bayesian neural networks, as well as closely related work on detection of out of distribution samples. Recent…
The paper shows strong correlation between in-distribution and out-of-distribution performance in various machine learning models.
problem Understanding reliability of machine learning systems in unseen environments.
method Empirical analysis of various models and distribution shifts on CIFAR-10, ImageNet, and other datasets.
result Out-of-distribution performance is strongly correlated with in-distribution performance across different models and distribution shifts.
In this work, we propose a method to reject out-of-distribution samples which can be adapted to any network architecture and requires no additional training data. Publicly available chest x-ray data (38,353 images) is used to train a standard ResNet-50 model to detect emphysema. Feature activations of intermediate laye…
Combining Bayesian deep learning and split conformal prediction affects out-of-distribution coverage.
problem Improving out-of-distribution coverage in multiclass image classification.
method Combining Bayesian deep learning with split conformal prediction methods.
result Combining methods can reduce out-of-distribution coverage in some cases.
Recent work has shown that deep generative models assign higher likelihood to out-of-distribution inputs than to training data. We show that a factor underlying this phenomenon is a mismatch between the nature of the prior distribution and that of the data distribution, a problem found in widely used deep generative mo…
RATIO improves neural network robustness and explainability.
problem Neural networks' lack of robustness to adversarial changes and uncertainty on out-distribution samples.
method RATIO: Adversarial Training on In- and Out-distribution.
result RATIO leads to robust models with reliable confidence estimates on out-distribution samples.
New framework detects out-of-distribution data by considering intrinsic ID attributes in outliers.
problem Deploying reliable machine learning systems requires effective out-of-distribution detection.
method Structured multi-view-based out-of-distribution detection learning (MVOL) framework.
result MVOL effectively utilizes both auxiliary OOD datasets and wild datasets with noisy in-distribution data.
XEnsemble improves DNN robustness against adversarial and out-of-distribution inputs.
problem Protecting DNN models from adversarial and out-of-distribution inputs.
method Diverse input denoising verifiers and disagreement-diversity ensemble learning.
result XEnsemble achieves high defense and detection success rates.
Study optimal ridge regularization for out-of-distribution prediction.
problem Optimal ridge regularization for predicting out-of-distribution data.
method Established conditions for optimal regularization under covariate and regression shifts, proving monotonic risk in data aspect ratio.
result Negative regularization can be optimal under shifts, even with isotropic or underparameterized training features.
Single deep model detects out-of-distribution data with single forward pass.
problem Detecting out-of-distribution data points in neural networks.
method Deterministic uncertainty quantification (DUQ) using gradient penalty for reliable detection.
result Single model outperforms or matches ensemble methods in out-of-distribution detection.
Unified study of out-of-distribution generalization across 172 datasets.
problem Measuring and improving robustness of transfer learning models.
method Collect and fine-tune 31k models on 172 dataset pairs, varying architectures and settings.
result In- and out-of-distribution accuracies increase jointly but their relation is dataset-dependent.
Bayesian layer improves image segmentation and out-of-distribution detection.
problem Outlier detection in image segmentation.
method Parameter-efficient hierarchical convolutional Gaussian Processes in Wasserstein-2 space.
result Uncertainty estimates improve out-of-distribution detection.
A new operator based on t-distributions improves NN classifiers' robustness to out-of-distribution samples.
problem NN classifiers assign extreme probabilities to out-of-distribution samples, leading to unreliable predictions.
method Derive a novel operator using t-distributions to model uncertainty more accurately.
result Classifiers using the new operator are more robust to out-of-distribution samples.
GRAM enhances deep RL for reliable real-world deployment.
problem Generalizing deep RL across in-distribution and out-of-distribution scenarios.
method Introduces a robust adaptation module and a joint training pipeline.
result GRAM achieves strong generalization performance in simulations and hardware.
Regularizes ML algorithms for robust multivariate analysis against distribution shifts.
problem Ensuring robustness of multivariate analysis algorithms against distribution shifts.
method Integrates a causal regularisation term into the loss function of multivariate analysis algorithms.
result Demonstrates improved out-of-distribution generalisation with reduced-rank regression and partial least squares.
SLUG method detects bias and out-of-distribution content in generative models.
problem Generative models can underrepresent certain groups and fail on out-of-distribution data.
method SLUG: A new uncertainty quantification method for VAEs combining Laplace approximations and stochastic trace estimators.
result SLUG's UQ score correlates with bias and out-of-distribution content.
Hierarchical VAEs detect out-of-distribution data by identifying low-level in-distribution features.
problem Out-of-distribution data often has in-distribution low-level features, leading to misleading likelihood estimates in deep generative models.
method Developed a fast, scalable, unsupervised likelihood-ratio score for out-of-distribution detection based on hierarchical variational autoencoders.
result Achieved state-of-the-art results on out-of-distribution detection across various data and model combinations.
The problem of detecting whether a test sample is from in-distribution (i.e., training distribution by a classifier) or out-of-distribution sufficiently different from it arises in many real-world machine learning applications. However, the state-of-art deep neural networks are known to be highly overconfident in their…
A simple method flags images as out-of-distribution based on their distance to nearest neighbors.
problem Detecting images not aligned with a trained model's in-distribution data.
method Flag images as OOD if their average distance to K nearest neighbors is large in the classifier's representation space.
result Simple methods can outperform more complex ones when considering learned representations.
Igeood detects out-of-distribution samples using information geometry.
problem Out-of-distribution (OOD) detection in machine learning systems.
method Igeood uses the Fisher-Rao geodesic distance to detect OOD samples from any pre-trained neural network.
result Igeood outperforms state-of-the-art methods on various network architectures and datasets.
New methods predict language model out-of-distribution behaviors using causal mechanisms.
problem Predicting how language models behave on unseen data.
method Two methods: counterfactual simulation and value probing.
result Both methods achieve high AUC-ROC and outperform causal-agnostic approaches in out-of-distribution settings.
Quantum models learn unitary actions on entangled states from product states.
problem Generalization to out-of-distribution data in quantum machine learning.
method Proved out-of-distribution generalization for learning unitary actions.
result Learned unitary actions on entangled states from product states.
Sharp analysis of out-of-distribution error in overparameterized models with importance weights.
problem Understanding and quantifying the degradation of performance in overparameterized models when faced with underrepresented data.
method Sharp analysis of an overparameterized Gaussian mixture model with spurious features and cost-sensitive interpolating solutions incorporating importance weights.
result Characterization of a novel tradeoff between worst-case robustness and average accuracy as a function of importance weight magnitude.
NECO detects out-of-distribution data using neural collapse properties.
problem Detecting out-of-distribution data in machine learning models.
method NECO leverages neural collapse geometric properties to identify OOD data.
result NECO achieves state-of-the-art results on OOD detection tasks.
Novel neural network models quantify uncertainty for deep classifiers.
problem Deep networks' overconfidence and ignorance about uncertainty.
method Variational autoencoders and GANs generate out-of-distribution samples.
result Better uncertainty estimates for in- and out-of-distribution samples.
Bayesian framework improves deep classifier reliability.
problem Overconfident models under dataset shift.
method Bayesian inference with out-of-distribution data augmentation.
result Reliable uncertainty estimates for deep classifiers.
The effective application of neural networks in the real-world relies on proficiently detecting out-of-distribution examples. Contemporary methods seek to model the distribution of feature activations in the training data for adequately distinguishing abnormalities, and the state-of-the-art method uses Gaussian distrib…
ACNML method improves uncertainty estimation for deep networks.
problem Uncertainty estimation and calibration for deep neural networks under distribution shift.
method Approximate Bayesian inference to approximate CNML distribution.
result ACNML compares favorably to prior techniques for uncertainty estimation.
Modern deep artificial neural networks have achieved great success in the domain of computer vision and beyond. However, their application to many real-world tasks is undermined by certain limitations, such as overconfident uncertainty estimates on out-of-distribution data or performance deterioration under data distri…
Improves regression models' performance on covariate shift.
problem Out-of-distribution generalization for regression.
method Spectrally adapting the weights of a pre-trained neural regression model.
result Spectral adaptation improves out-of-distribution performance.
Study on how kernel regression models generalize to out-of-distribution data.
problem Understanding generalization in machine learning models under distributional shifts.
method Replica method from statistical physics to derive analytical formula for generalization error.
result Identified overlap matrix as key determinant of generalization performance under distribution shift.
Detects out-of-distribution sentences in Neural Machine Translation.
problem Identifying sentences from a different language than the training data.
method Developed a new uncertainty measure for long sequences of words in Transformers.
result Shows ability to identify Dutch sentences as German input.
Accuracy on in-distribution data correlates with out-of-distribution data when data is noisy or contains nuisance features.
problem Correlation between in-distribution and out-of-distribution accuracy in noisy or feature-rich data.
method Analyzes the impact of noise and nuisance features on model performance.
result Accuracy on in-distribution and out-of-distribution data can become negatively correlated in noisy or feature-rich data.
A new method approximates pNML for faster out-of-distribution detection.
problem Detecting out-of-distribution examples efficiently.
method Influence functions approximation of pNML for neural networks.
result The approximation effectively detects out-of-distribution examples.
Study complex-valued VAEs for radar OOD detection.
problem Detecting out-of-distribution signals in complex radar environments.
method Proposed and compared several detection metrics for CVAE.
result CVAE-MSE and latent-based scores outperform ANMF-Tyler.
Bayesian deep learning improves out-of-distribution detection but not always.
problem Improving the reliability of deep learning models in uncertain or novel data.
method Validation of likelihood-based Bayesian models for out-of-distribution detection.
result Bayesian deep learning models can marginally outperform conventional neural networks in certain conditions.
Proposes training neural networks to predict uncertainty for out-of-distribution inputs.
problem Poor uncertainty predictions for out-of-distribution inputs limit model robustness.
method Generates pseudo-inputs in low-density regions and trains a Bayesian framework.
result Yields robust and interpretable uncertainty predictions.
Paper tackles SCOD problem with optimal strategy and empirical validation.
problem Designing reliable prediction models abstaining from uncertain predictions.
method Bayes classifier for ID data and stochastic linear selector in 2D space.
result POSCOD method outperforms existing OOD methods.
PEOC uses policy entropy to detect untrained states in RL.
problem Detecting untrained states in reinforcement learning for safety.
method Policy entropy based one-class classifier.
result PEOC is highly competitive and reliable.
New method improves robustness of OOD detection models.
problem Detecting out-of-distribution inputs is critical for deep learning models.
method Proposes ALOE algorithm for robust training with adversarially crafted examples.
result ALOE substantially improves robustness of OOD detection on CIFAR-10 and CIFAR-100 datasets.
Normalizing flows fail to detect OOD data due to learning local pixel correlations.
problem Detecting out-of-distribution data in machine learning systems.
method Investigated why normalizing flows fail to distinguish between in- and out-of-distribution data, and modified flow architecture to improve OOD detection.
result Modifying flow architecture can improve OOD detection by biasing the flow towards learning semantic structure of the target data.