Paper proposes a framework to detect distribution shifts using embedding space geometry.
problem Detecting distribution shifts in candidate datasets to improve model generalizability.
method Non-parametric framework using embedding space geometry for two tests: robustness boundary and in-distribution/out-of-distribution classification.
result Both tests successfully detect distribution shifts in various scenarios for both synthetic and real-world datasets.
Detects changes in classifier scores to identify shifts in class priors.
problem Label shift changes in classification data.
method Sequential changepoint detection of classifier scores.
result Outperforms other detection procedures in label shift settings.
New method detects novel node categories in graphs with distribution shifts.
problem Detecting novel node categories in graphs with distribution shifts.
method Recall-Constrained Optimization with Selective Link Prediction (RECO-SLIP).
result RECO-SLIP outperforms existing methods in detecting novel node categories.
A new method detects distribution shifts faster than existing CTMs.
problem Detecting distribution shifts in data streams with contamination issues.
method Uses a fixed reference dataset to compare each new sample, avoiding contamination.
result Detects distribution shifts faster and more reliably than standard CTMs.
M-FISHER detects and adapts to streaming data shifts with statistical validity and stability.
problem Detecting and adapting to distributional shifts in streaming data.
method Constructs an exponential martingale from non-conformity scores and applies Ville's inequality for detection. Fisher-preconditioned updates for adaptation.
result Establishes M-FISHER as a principled approach for robust, anytime-valid detection and geometrically stable adaptation.
Detects domain shifts in datasets using interpretable feature subspaces.
problem Detecting subtle differences in dataset probability distributions.
method Localised density anomaly detection in high-dimensional feature spaces.
result Extracts interpretable feature subspaces for domain shifts.
Detects which features have shifted in data distributions.
problem Identifying which specific features have caused a distribution shift.
method Formalizes the problem as multiple conditional distribution hypothesis tests, proposes non-parametric and parametric statistical tests, and uses a test statistic based on the density model score function.
result Demonstrates methods for identifying when and where a shift occurs in multivariate time-series data.
Detects harmful shifts without labels for model performance.
problem Detecting distribution shifts without access to labels.
method Uses a proxy derived from predictions of an error estimator.
result High power and false alarm control under various shifts.
Bayesian method adapts to unknown distribution shifts in online learning.
problem Online learning with unknown and irregular distribution shifts.
method Bayesian inference with change-point detection and beam search.
result Improves adaptation to new data distributions over state-of-the-art methods.
This research examines how model explanations change under distribution shifts in tabular data.
problem Detecting distribution shifts in tabular data affecting model performance and explanations.
method Investigates the relationship between model performance and explanation characteristics under distribution shifts.
result Explanation shifts are a better indicator for detecting predictive performance changes than traditional distribution shift techniques.
Study evaluates AD methods for fraud detection in online credit card payments.
problem Fraud detection in online credit card payments using anomaly detection methods.
method Assessed several recent anomaly detection methods and compared them with standard supervised learning methods.
result LightGBM outperforms other methods but is more sensitive to distribution shifts.
Study detects concept shift in online data using martingales.
problem Detecting concept shift in online datasets.
method Exchangeable martingales and conformal prediction techniques.
result Decomposes concept shift into detectable components.
A new approach for test-time adaptation detects and reacts to distribution shifts.
problem Improving test-time accuracy under distribution shifts.
method Online self-training with a detection tool based on entropy values and betting martingales.
result The classifier's entropy values match those of the source domain, building invariance to distribution shifts.
Detects harmful distribution shifts in deployed models without false alarms.
problem Detecting harmful distribution shifts in deployed models without false alarms.
method Sequential tools for testing if the difference between source and target distributions leads to a significant increase in a risk function.
result Demonstrated the efficacy of the proposed framework through extensive empirical studies.
MADOD meta-learns invariant features for OOD detection across unseen domains.
problem Simultaneous covariate and semantic shifts in real-world machine learning applications.
method Meta-learning and G-invariance to learn robust, domain-invariant features.
result Superior performance in semantic OOD detection across unseen domains.
Adaptive monitoring for AI systems detects and diagnoses shifts in data distribution.
problem Continuous monitoring of AI systems to detect and address unsafe behavior.
method Weighted-conformal martingales (WCTMs) for online monitoring of AI systems.
result Improved performance over state-of-the-art baselines on real-world datasets.
This work uses adversarial learning to detect and correct feature shifts in various datasets.
problem Detecting and correcting feature shifts in real-world datasets.
method Adversarial learning applied to multiple discriminators to detect and correct feature shifts.
result Mainstream classifiers can effectively localize and correct feature shifts, outperforming existing techniques.
MAGDiff detects data shifts in neural networks without retraining.
problem Neural networks' sensitivity to data distribution shifts.
method Extracts MAGDiff representations from neural networks to detect shifts.
result MAGDiff representations improve data set shift detection.
Faced with distribution shift between training and test set, we wish to detect and quantify the shift, and to correct our classifiers without test set labels. Motivated by medical diagnosis, where diseases (targets) cause symptoms (observations), we focus on label shift, where the label marginal p(y) changes but the …
Online monitoring system for safety classifiers with shift detection and conformal adaptation
problem Detecting and adapting to distributional shifts in deployed safety classifiers
method Calibrated sequential statistics for online monitoring, conformal abstention for adaptation
result 86.6% valid detection with mean latency of 39.5 steps
Improved OOD detection across various shifts using multi-encoder fusion of RDMs.
problem Out-of-distribution detection across multiple types of distribution shifts.
method Statistical identification of encoder sensitivity, EncMin2L fusion, and Tippett minimum combination.
result Achieves AUROC ≥ 0.94 across four shift types, outperforming state-of-the-art detectors.
CSI detects novelty by contrasting shifted instances, outperforming existing methods.
problem Detecting samples from outside the training distribution.
method Contrastive learning with distributionally shifted augmentations.
result CSI outperforms existing methods in various novelty detection scenarios.
Unified framework for OOD detection and generalization using graph theory.
problem Challenges in out-of-distribution (OOD) generalization and detection in real-world machine learning models.
method Graph-theoretic framework to jointly tackle OOD generalization and detection.
result Empirical validation of theoretical underpinnings with competitive performance.
New approach models how explanations shift with distribution changes.
problem Model performance drops with changing input data distributions.
method Models explanation shifts and compares them to state-of-the-art techniques.
result Modeling explanation shifts better detects out-of-distribution behavior.
Online monitor detects classifier drift and adapts predictions.
problem Silent degradation of classifier accuracy under distributional shift.
method Sliding-window KS statistic with calibrated alarm thresholds.
result 86.6% valid detection across various shift conditions.
Paper tackles high-dimensional quantile regression with distribution shift using transfer learning.
problem Efficiency of knowledge transfer is severely impacted by distribution shift in high-dimensional regression.
method Proposes a novel transferable set and framework for three types of distribution shift: parameter, covariate, and residual.
result Establishes estimation error bounds and source detection consistency for the proposed method.
A new control chart detects shifts in binary data streams quickly and reliably.
problem Early detection of small shifts in multiple binary data streams.
method Cumulative Standardized Binomial EWMA (CSB-EWMA) chart with exact variance derivation.
result Adaptive control limits ensure robust detection across different data distributions.
In this paper, we study how the mean shift algorithm can be used to denoise a dataset. We introduce a new framework to analyze the mean shift algorithm as a denoising approach by viewing the algorithm as an operator on a distribution function. We investigate how the mean shift algorithm changes the distribution and sho…
Method distinguishes between failures and domain shifts in industrial data streams.
problem Confusing domain shifts with failures in industrial data.
method Modified Page-Hinkley changepoint detector and supervised domain-adaptation-based anomaly detection.
result Allows differentiation between failures and domain shifts.
Paper proposes methods to help autonomous vehicles adapt to unexpected driving scenarios.
problem Autonomous vehicles struggle with unexpected driving conditions.
method Robust imitative planning (RIP) and adaptive robust imitative planning (AdaRIP) methods to detect and adapt to distribution shifts.
result Methods outperform current state-of-the-art approaches in nuScenes prediction challenge.
TSFMs embed non-stationary time series data, revealing specific types of changes.
problem Understanding non-stationarity in TSFMs' embedding spaces.
method Examined mean shifts, variance changes, linear trends, and persistence in TSFMs.
result Different TSFMs exhibit distinct failure modes in detecting non-stationarity.
AutoML simplifies two-sample tests for detecting distribution shifts.
problem Detecting distribution shifts between datasets.
method Uses mean discrepancy of a witness function with squared loss minimization.
result AutoML simplifies and improves two-sample testing performance.
Proposes real-time risk monitoring for machine learning systems under unknown shifts.
problem Dynamic distribution shifts challenge real-world machine learning systems' risk assurances.
method Sequential hypothesis testing with 'testing by betting' to detect risk violations.
result Effective real-time risk monitoring under various unknown shifts.
Universal approach combines OOD detection scores for robustness.
problem Combining diverse OOD detection scores for robustness.
method Quantile normalization to p-values, meta-analysis, probabilistic interpretation.
result Significantly improved robustness and performance across diverse OOD detection scenarios.
We might hope that when faced with unexpected inputs, well-designed software systems would fire off warnings. Machine learning (ML) systems, however, which depend strongly on properties of their inputs (e.g. the i.i.d. assumption), tend to fail silently. This paper explores the problem of building ML systems that fail …
New dataset for industrial machine malfunction detection with domain shifts.
problem Challenges in detecting anomalies due to domain shifts in industrial sounds.
method Created a dataset with domain shifts for five types of industrial machines.
result Significant performance differences between source and target domains.
DetectShift framework detects and quantifies dataset shifts in various data types.
problem Frequent dataset shifts decrease supervised learning performance.
method DetectShift framework quantifies and tests for multiple dataset shifts in various data types.
result DetectShift framework effectively detects dataset shifts even in higher dimensions.
Proposes a new measure to evaluate stability of statistical parameters under distributional shifts.
problem Difficulty in transferring knowledge across data sets due to distributional changes.
method Introduces a measure of instability quantifying sensitivity of statistical parameters to Kullback-Leibler divergence and directional shifts.
result The proposed measure can elucidate the type of shifts a parameter is sensitive to and improve estimation accuracy under shifted distributions.
Balanced Activation improves object detection performance on long-tailed datasets.
problem Mismatch between training and testing label distributions in object detection.
method Introduces Balanced Activation (Balanced Softmax and Balanced Sigmoid) to address label distribution shift.
result Balanced Activation provides ~3% gain in mAP on LVIS-1.0 compared to state-of-the-art methods.
Estimates model performance under distribution shift using domain-invariant predictors.
problem Poor performance of models on test distributions different from training distributions.
method Uses domain-invariant predictors as a proxy for unknown target labels.
result Shows that the complexity of latent representations influences target risk.
New CPS model tackles conditional probability shift in machine learning.
problem Discrepancy between source and target distributions in machine learning.
method Conditional Probability Shift Model (CPSM) using multinomial regression and EM algorithm.
result Superior balanced classification accuracy on target data compared to existing methods.
Framework detects out-of-distribution inputs in regression and survival analysis.
problem Limited OOD detection for regression and survival analysis.
method Model-aware and subspace-aware variable prioritization.
result Consistent improvements over existing methods in synthetic and real data.
FSL-Net detects and localizes feature shifts in large, high-dimensional datasets.
problem Feature shifts between data sources lead to erroneous features in various applications.
method FSL-Net is a neural network trained on multiple datasets to localize feature shifts.
result FSL-Net accurately localizes feature shifts from unseen datasets without re-training.
Two-stage model improves credit scoring predictions.
problem Distribution shift in finance datasets.
method Two-stage model with out-of-distribution detection and domain knowledge.
result Highly reliable predictions for most datasets.
Enhanced regime shifts detection using unstructured text and financial data.
problem Detecting regime shifts in financial markets is challenging due to noisy and multicollinear data.
method Combines LLM reasoning on unstructured text and statistical validation on financial time series.
result Framework achieves F1 score of 0.82, outperforming pure data-driven methods.
Study evaluates predictive uncertainty in malware detection.
problem Detecting dataset shift and adversarial examples in malware detection.
method Re-designed and built 24 Android malware detectors, quantified their uncertainties with nine metrics.
result Predictive uncertainty helps reliable malware detection but not adversarial evasion attacks.
PITMonitor monitors model calibration over time with formal error guarantees.
problem Fixed-sample tests applied to models over time can lead to false alarms.
method PITMonitor uses mixture e-processes to detect distributional shifts in probability integral transforms.
result PITMonitor achieves competitive detection rates on river's FriedmanDrift benchmark.
A new framework detects adverse dataset shifts using outlier scores.
problem False alarms in dataset shift tests.
method Outlier scores to compare contamination rates at varying thresholds.
result Reduces the sensitivity to minor differences in predictive performance.