Paper tackles efficient risk estimation under dataset shift conditions.
problem Limited data from target population; auxiliary data available.
method Semiparametric efficiency theory; efficient and multiply robust estimators.
result Developed estimators for various dataset shift conditions.
New dataset for industrial machine malfunction detection with domain shifts.
problem Challenges in detecting anomalies due to domain shifts in industrial sounds.
method Created a dataset with domain shifts for five types of industrial machines.
result Significant performance differences between source and target domains.
FSL-Net detects and localizes feature shifts in large, high-dimensional datasets.
problem Feature shifts between data sources lead to erroneous features in various applications.
method FSL-Net is a neural network trained on multiple datasets to localize feature shifts.
result FSL-Net accurately localizes feature shifts from unseen datasets without re-training.
DetectShift framework detects and quantifies dataset shifts in various data types.
problem Frequent dataset shifts decrease supervised learning performance.
method DetectShift framework quantifies and tests for multiple dataset shifts in various data types.
result DetectShift framework effectively detects dataset shifts even in higher dimensions.
Study detects concept shift in online data using martingales.
problem Detecting concept shift in online datasets.
method Exchangeable martingales and conformal prediction techniques.
result Decomposes concept shift into detectable components.
FJS method improves multinomial classification accuracy.
problem Improving multinomial classification accuracy under dataset shift.
method Derive FJS representation and propose alternative methods.
result Factorizable joint shift is not fully identifiable without additional assumptions.
Paper proposes a framework to detect distribution shifts using embedding space geometry.
problem Detecting distribution shifts in candidate datasets to improve model generalizability.
method Non-parametric framework using embedding space geometry for two tests: robustness boundary and in-distribution/out-of-distribution classification.
result Both tests successfully detect distribution shifts in various scenarios for both synthetic and real-world datasets.
Machine learning and data mining techniques have been used extensively in order to detect credit card frauds. However purchase behaviour and fraudster strategies may change over time. This phenomenon is named dataset shift or concept drift in the domain of fraud detection. In this paper, we present a method to quantify…
Shifts dataset evaluates uncertainty in real-world tasks across modalities.
problem Lack of standard datasets for evaluating uncertainty estimation and robustness to distributional shift.
method Proposes Shifts Dataset for evaluation of uncertainty estimates and robustness to distributional shift across tabular, audio, text, and sensor data.
result Baseline results for tabular weather prediction, machine translation, and SDC vehicle motion prediction.
SJS model predicts label shifts in multinomial datasets.
problem Predicting label shifts in multinomial datasets.
method Sparse joint shift model for dataset shift.
result Valid predictions and class prior probabilities estimates.
Estimates calibration error under label shift without labels.
problem Ensuring model reliability in the face of dataset shift without access to labels.
method Importance re-weighting of the labeled source distribution to estimate calibration error under label shift.
result Effective and reliable CE estimation with respect to the shifted target distribution.
Extends Shifts dataset for MS lesion segmentation and marine vessel power estimation.
problem Distributional shift in training and deployment data for ML models.
method Develops new datasets for high-risk industrial applications.
result Demonstrates robustness and uncertainty estimation in new industrial tasks.
We might hope that when faced with unexpected inputs, well-designed software systems would fire off warnings. Machine learning (ML) systems, however, which depend strongly on properties of their inputs (e.g. the i.i.d. assumption), tend to fail silently. This paper explores the problem of building ML systems that fail …
BREEDS benchmarks assess model robustness to subpopulation shifts.
problem Measuring model robustness to novel subpopulation shifts.
method Controlled synthesis of realistic distribution shifts using class structure.
result Validated model sensitivity and effectiveness of robustness interventions.
This work uses adversarial learning to detect and correct feature shifts in various datasets.
problem Detecting and correcting feature shifts in real-world datasets.
method Adversarial learning applied to multiple discriminators to detect and correct feature shifts.
result Mainstream classifiers can effectively localize and correct feature shifts, outperforming existing techniques.
Study shows current image classification models lack robustness to real-world dataset shifts.
problem Robustness of current image classification models to natural distribution shifts in real datasets.
method Evaluation of 204 ImageNet models in 213 different test conditions.
result Little to no transfer of robustness from synthetic to natural distribution shifts.
Detects domain shifts in datasets using interpretable feature subspaces.
problem Detecting subtle differences in dataset probability distributions.
method Localised density anomaly detection in high-dimensional feature spaces.
result Extracts interpretable feature subspaces for domain shifts.
DSLOB creates synthetic LOB data for benchmarking forecasting algorithms under distributional shifts.
problem Challenges in dealing with out-of-distribution limit order book data.
method Multi-agent market simulator to create labeled synthetic LOB dataset with and without market stress.
result Demonstrates the need for robust forecasting algorithms to handle distributional shifts.
Recent interest in the external validity of prediction models (i.e., the problem of different train and test distributions, known as dataset shift) has produced many methods for finding predictive distributions that are invariant to dataset shifts and can be used for prediction in new, unseen environments. However, the…
Paper tackles robust classification under class-dependent domain shift.
problem Class-dependent domain shift in machine learning.
method Defined a simple optimization problem with an information theoretic constraint and solved it using neural networks.
result Demonstrated that the proposed method can learn robust classifiers that generalize well to unseen domains.
Proposes a framework to assess model robustness to dataset shifts.
problem Evaluating model robustness to changes in setting or population.
method Derives a debiased estimator for analyzing performance on worst-case distributions.
result Demonstrates the estimator can account for realistic shifts in complex distributions.
Identifies shifts in causal mechanisms between related datasets using ANMs.
problem Estimating the full causal structure from data is challenging; focus on identifying shifts in causal mechanisms.
method Assumes nonlinear additive noise models, uses Jacobian of score function for mixture distribution to identify shifts.
result Shows applicability of the approach on synthetic and real-world data.
FakeEdge tackles dataset shift in link prediction tasks.
problem Dataset shift between training and testing sets in link prediction.
method Model-agnostic technique to mitigate graph topological gap.
result Extensive experiments show FakeEdge's superiority on multiple datasets.
Paper proposes SJS model to estimate model performance under covariate and label shifts.
problem Estimating model performance when both covariates and labels shift.
method Sparse Joint Shift (SJS) model and SEES algorithm.
result SEES achieves significant shift estimation error improvements over existing approaches.
A new framework detects adverse dataset shifts using outlier scores.
problem False alarms in dataset shift tests.
method Outlier scores to compare contamination rates at varying thresholds.
result Reduces the sensitivity to minor differences in predictive performance.
Method identifies shifts leading to large model performance differences.
problem Detecting shifts in distribution that affect model performance.
method Parametric changes in causal mechanisms define robustness sets; worst-case optimization problem approximated as non-convex quadratic.
result Second-order approximation of worst-case loss for small shifts, leading to efficient algorithms.
Robust OPE framework uses human inputs to improve policy evaluation in changing environments.
problem Inaccurate policy evaluations due to shifts in environment properties.
method Adapts OPE methods to shifts on user-inputted covariates, providing more realistic utility estimates.
result Robust OPE framework yields less pessimistic policy evaluations and captures realistic dataset shifts.
A feature-weighted mean shift algorithm improves clustering in high-dimensional data.
problem Clustering high-dimensional data with traditional mean shift algorithms.
method Feature-weighted mean shift algorithm.
result The algorithm outperforms conventional mean shift and preserves computational simplicity.
A novel variational inference based resampling framework is proposed to evaluate the robustness and generalization capability of deep learning models with respect to distribution shift. We use Auto Encoding Variational Bayes to find a latent representation of the data, on which a Variational Gaussian Mixture Model is a…
New method calibrates models under covariate shifts.
problem Calibration of models can be lost under covariate shifts.
method Importance sampling based approach.
result Efficacy demonstrated on real-world and synthetic datasets.
A natural way to characterize the cluster structure of a dataset is by finding regions containing a high density of data. This can be done in a nonparametric way with a kernel density estimate, whose modes and hence clusters can be found using mean-shift algorithms. We describe the theory and practice behind clustering…
Algorithm constructs prediction sets with PAC guarantees in label shift settings.
problem Reliable uncertainty quantification in the face of distribution shift.
method Estimates predicted probabilities and confusion matrix, then propagates uncertainty through Gaussian elimination to compute confidence intervals and construct prediction sets.
result Satisfies PAC guarantees and produces smaller, more informative prediction sets.
We created financial benchmarks for distribution shifts in crude oil prices and volatility.
problem Scarcity of task-labeled time-series benchmarks in finance.
method Transformed asset price data into volatility proxies, generated task labels based on distribution shifts, and made datasets publicly available.
result Inclusion of task labels improves continual learning algorithms' performance on real-world data.
Undersampling often outperforms other methods in nonparametric classification.
problem Distribution shift challenges in nonparametric binary classification.
method Proved undersampling is minimax optimal in worst-case scenarios.
result Undersampling is a robustness intervention with theoretical guarantees.
In this paper, we study how the mean shift algorithm can be used to denoise a dataset. We introduce a new framework to analyze the mean shift algorithm as a denoising approach by viewing the algorithm as an operator on a distribution function. We investigate how the mean shift algorithm changes the distribution and sho…
Study evaluates methods for improving model robustness to various real-world distribution shifts.
problem Improving model robustness to real-world distribution shifts like geographic changes.
method Introduced new datasets and evaluated existing methods on four types of shifts (style, blurriness, location, camera operation).
result Data augmentations and larger models can improve robustness on real-world distribution shifts, contrary to prior claims.
Unsupervised learning representations generalize better than supervised learning under distribution shifts.
problem Robustness of unsupervised representations to distribution shift.
method Extensive evaluation on synthetic and realistic datasets, including controllable domain generalization datasets.
result Unsupervised representations learned from SSL and AE generalize better than supervised learning under various distribution shifts.
Extends FJS analysis to general label spaces, including classification and regression.
problem Distribution shift in general label spaces, including covariate and label shifts.
method Proposes a framework for analyzing FJS in general label spaces and generalizes existing results.
result Generalizes FJS analysis to general label spaces, including classification and regression.
While deep learning models become more widespread, their ability to handle unseen data and generalize for any scenario is yet to be challenged. In medical imaging, there is a high heterogeneity of distributions among images based on the equipment that generates them and their parametrization. This heterogeneity trigger…
We present a novel class of convolutional neural networks (CNNs) for set functions, i.e., data indexed with the powerset of a finite set. The convolutions are derived as linear, shift-equivariant functions for various notions of shifts on set functions. The framework is fundamentally different from graph convolutions b…
Prediction-time batch normalization improves model robustness under covariate shift.
problem Dealing with covariate shift in deep learning models.
method Prediction-time batch normalization, a simple but effective method.
result Significantly improves model accuracy and calibration under covariate shift.
Drift-Resilient TabPFN learns to adapt to changing data distributions.
problem Real-world data often shifts over time, degrading model performance.
method In-Context Learning with a Prior-Data Fitted Network, using structural causal models.
result Significant performance improvements across various datasets.
New method creates diverse neural ensembles for better uncertainty estimation and robustness.
problem Creating more robust neural networks for uncertainty estimation and dataset shift.
method Automatically constructing ensembles with varying architectures.
result Ensembles with varying architectures outperform deep ensembles in accuracy, uncertainty calibration, and robustness.
HypeGBMS clusters data in hyperbolic space, overcoming Euclidean limitations.
problem Clustering in hierarchical or tree-like datasets in curved spaces.
method Hyperbolic Gaussian Blurring Mean Shift with Möbius-weighted means.
result HypeGBMS effectively captures latent hierarchies in non-Euclidean data.
The mean-shift algorithm is a popular algorithm in computer vision and image processing. It can also be cast as a minimum gamma-divergence estimation. In this paper we focus on the "blurring" mean shift algorithm, which is one version of the mean-shift process that successively blurs the dataset. The analysis of the bl…
This paper improves STL inference reliability under covariate shift.
problem Ensuring correct STL formulas in real-world settings with distribution shift.
method Proposes a conformalized STL inference framework that addresses covariate shift.
result Significantly improves symbolic learning reliability at deployment time.
This paper investigates how machine learning APIs change over time and proposes an efficient method to monitor these changes.
problem Understanding and assessing changes in machine learning APIs over time.
method Systematic investigation of ML API shifts, proposing a principled adaptive sampling algorithm (MASA) for efficient estimation of confusion matrix shifts.
result MASA can accurately estimate confusion matrix shifts using up to 90% fewer samples compared to random sampling.
The hypothesis that computational models can be reliable enough to be adopted in prognosis and patient care is revolutionizing healthcare. Deep learning, in particular, has been a game changer in building predictive models, thus leading to community-wide data curation efforts. However, due to inherent variabilities in …