The paper offers guidelines for validating data-driven models.
problem Ensuring reliable validation of data-driven models.
method A set of general rules for model validation.
result Helps practitioners create reliable validation plans and report results transparently.
Active learning method improves local model validity estimation.
problem Ensuring local model validity in machine learning applications.
method Learning model error to estimate local validity using active learning.
result The proposed method can estimate local validity with a small amount of data.
In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for empirical risk minimizers. In the general setting, we prove sanity-check bounds in the spirit of \cite{KR99} \textquotedblleft\textit{bounds showing that the worst-case error of this estimate is not m…
AST provides a method to validate safe autonomy without unsafe simplifications.
problem Validation of safe autonomy in complex systems.
method Adaptive Stress Testing (AST) approach.
result AST can find failures without unsafe simplifications.
Paper revisits pre-validation method, improving hypothesis testing.
problem Improving hypothesis testing in pre-validated models with different feature dimensions.
method Extended problem formulation, analytical distribution, and bootstrap procedure.
result Proposed analytical distribution and bootstrap procedure for pre-validated predictors.
In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for stable predictors in the context of risk assessment. The notion of stability has been first introduced by \cite{DEWA79} and extended by \cite{KEA95}, \cite{BE01} and \cite{KUNIY02} to characterize cla…
A fast bootstrap method estimates cross-validation standard error.
problem Uncertainty quantification in cross-validation estimates.
method Random-effects model to estimate variance component.
result Valid confidence intervals for model performance.
This paper tackles deep clustering evaluation challenges in high-dimensional data.
problem Evaluation of deep clustering methods is problematic due to the curse of dimensionality and variations in embedding spaces.
method Develops a theoretical framework to highlight the ineffectiveness of internal validation measures and proposes a systematic approach to applying clustering validity indices in deep learning.
result The proposed framework reduces misguidance from improper use of clustering validity indices in deep learning.
Develops methods for valid and validated confidence sets in multiclass and multilabel prediction.
problem Challenges of typical conformal prediction methods in multiclass and multilabel problems, especially uneven coverage.
method Leverages quantile regression to build methods that always guarantee correct coverage and asymptotically optimal conditional coverage, addressing label interactions with tree-structured classifiers.
result Empirical evaluation suggests more robust coverage of confidence sets.
In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for subagged estimators, both for classification and regressor. General loss functions and class of predictors with both finite and infinite VC-dimension are considered. We slightly generalize the formali…
New CV method reduces bias in spatial prediction models.
problem Bias in standard cross-validation due to uneven sampling.
method Target-Weighted Cross-Validation (TWCV) framework.
result Weighted CV approaches reduce bias in prediction error.
New IF method improves accuracy in deep neural networks with noisy data.
problem Inaccurate influence estimates in deep neural networks, especially with noisy data.
method Established a connection between influence estimation error, validation set risk, and sharpness, introducing a novel estimation form for flat validation minima.
result Our novel Influence Function approach provides more accurate influence estimates, validated across various tasks.
New method improves spatial prediction validation accuracy.
problem Validation methods fail for spatial prediction tasks due to mismatch between validation and test locations.
method Proposes a new validation method that adapts existing covariate-shift ideas to spatial settings.
result Proves and demonstrates the new method's superiority in spatial prediction validation.
When selecting a classification algorithm to be applied to a particular problem, one has to simultaneously select the best algorithm for that dataset \emph{and} the best set of hyperparameters for the chosen model. The usual approach is to apply a nested cross-validation procedure; hyperparameter selection is performed…
Improves test set performance and reduces out-of-sample disappointment for unstable models.
problem Ensuring strong test set performance via cross-validation for unstable models.
method Nested k-fold cross-validation with hyperparameter selection based on a weighted sum of cross-validation metric and model stability measure.
result Improves out-of-sample MSE for sparse ridge regression and CART by 4% and 2% respectively, compared to k-fold cross-validation.
Echo State Networks (ESNs) are known for their fast and precise one-shot learning of time series. But they often need good hyper-parameter tuning for best performance. For this good validation is key, but usually, a single validation split is used. In this rather practical contribution we suggest several schemes for cr…
A new method improves super learner validation efficiency.
problem Improving the efficiency of super learner validation.
method Bootstrap Bias Corrected Cross Validation applied to Super Learning.
result Bootstrap Bias Corrected Cross Validation proved efficient and cost-effective.
We compare and contrast two approaches to validating a trained classifier while using all in-sample data for training. One is simultaneous validation over an organized set of hypotheses (SVOOSH), the well-known method that began with VC theory. The other is withhold and gap (WAG). WAG withholds a validation set, trains…
This paper addresses external validity bias in causal inference.
problem Estimating causal effects in a target population.
method Synthesis of approaches for generalizability and transportability, including tests for heterogeneity of treatment effects and differences between study and target populations.
result Framework for addressing external validity bias in causal inference.
Improving cancer treatment decisions requires considering causal effects, not just model accuracy.
problem Cancer outcome prediction models may cause harm when used for treatment decisions.
method Explains the importance of considering causal effects in model validation and provides guidelines.
result Building and validating models that are useful for decision making requires considering causal effects.
The paper challenges the validity of cluster validity measures in unsupervised learning.
problem The validity of cluster validity measures in selecting optimal clusterings.
method The authors investigate the use of cluster validity measures as objective functions in unsupervised learning and introduce a new variant of the Dunn index.
result Many cluster validity measures promote clusterings that do not match expert knowledge well.
Model selection on validation data is an essential step in machine learning. While the mixing of data between training and validation is considered taboo, practitioners often violate it to increase performance. Here, we offer a simple, practical method for using the validation set for training, which allows for a conti…
The paper improves confidence intervals for test error using cross-validation.
problem Improving confidence intervals for test error in machine learning.
method Develops central limit theorems and consistent estimators for cross-validation.
result Provides asymptotically-exact confidence intervals and hypothesis tests.
Prediction of future observations is an important and challenging problem. The two mainstream approaches for quantifying prediction uncertainty use prediction regions and predictive distributions, respectively, with the latter believed to be more informative because it can perform other prediction-related tasks. The st…
Machine learning detects survey validity from user behavior.
problem Detecting valid responses in web surveys.
method Uses mouse activity and machine learning models (LSTM, HMM).
result Predicts survey validity without analyzing specific answers.
Proposes a test to ensure predictive algorithms predict intended outcomes better than unintended ones.
problem Unintended model behavior leading to prediction of unintended outcomes.
method Falsification framework using nonparametric hypothesis testing to compare prediction losses across outcomes.
result Establishes discriminant validity with respect to gender but not race in an admissions setting.
Combines trial and observational data to improve policy evaluation.
problem External validity of randomized trial results in target populations.
method Uses covariate data to model trial sampling and certifies policy evaluations.
result Valid trial-based policy evaluations under model miscalibration.
New method uses weak labels to create valid confidence sets for predictions.
problem Lack of labeled data in machine learning models.
method Developed a conformal prediction framework to provide valid predictive confidence sets using weakly labeled data.
result New coverage definition allows for tighter and more informative (but valid) confidence sets.
Deep generative models have been successfully used to learn representations for high-dimensional discrete spaces by representing discrete objects as sequences and employing powerful sequence-based deep models. Unfortunately, these sequence-based models often produce invalid sequences: sequences which do not represent a…
A new cross-validation method reduces redundancy and improves model performance.
problem Redundancy in traditional k-fold cross-validation leads to biased results.
method Irredundant k-fold cross-validation, where each instance is used exactly once for training and testing.
result Consistent performance estimates with reduced variance and lower computational cost.
Unified taxonomy for ML uncertainty in physics, validated.
problem Uncertainty quantification in machine learning for physics.
method Unified taxonomy, principled validation tools.
result Illustrated validation tools with examples.
The study assesses external validity by evaluating worst-case treatment effects across subpopulations.
problem Underrepresentation of marginalized groups and limited study populations.
method Develops a semiparametrically efficient estimator for worst-case treatment effects (WTE) and uses cross-fitting to guard against brittle findings.
result The proposed framework guards against invalid findings due to unanticipated population shifts.
Study compares mutation validation and cross-validation for model selection.
problem Comparing model selection methods for generalization performance and computational efficiency.
method Empirical comparison using benchmark and real-world datasets with Bayesian tests.
result Both methods select models with equivalent generalization performance but MV selects simpler models and is computationally cheaper.
Cross-validation estimates model performance on unseen data, not training data.
problem Understanding how cross-validation estimates prediction error and its limitations.
method Analyzing linear models and popular prediction error estimates, introducing nested cross-validation.
result Cross-validation estimates the average prediction error of models fit on other unseen training sets, not the model at hand.
CPTD improves prediction intervals in time series regression with cross-sectional data.
problem Constructing valid prediction intervals in time series regression with a cross-section.
method Conformal Prediction with Temporal Dependence (CPTD) for post-hoc, light-weight approach.
result CPTD maintains cross-sectional validity while improving longitudinal coverage.
Adapts conformal prediction for missing data, ensuring valid coverage.
problem Uncertainty quantification with missing covariates.
method Proposes a reweighted conformal prediction procedure for handling missing values.
result Guaranteed Marginal Coverage and Mask-Conditional Validity for general missing data mechanisms.
Causal ML methods failed to validate their personalized treatment effects in two large trials.
problem Validating causal machine learning methods for personalized treatment effects in precision medicine.
method Assessed 17 mainstream causal heterogeneity ML methods using two large randomized controlled trials.
result None of the ML methods reliably validated their performance, internal or external, showing significant discrepancies between training and test data.
The paper develops a cross-validation method for improving signal denoising techniques.
problem Improving signal denoising methods for nonparametric regression.
method Develops a general cross-validation framework for signal denoising and applies it to Trend Filtering and Dyadic CART.
result Cross validated versions of Trend Filtering and Dyadic CART achieve nearly optimal convergence rates.
HLTF generates chemically valid 3D molecules with improved topology control.
problem Generating chemically valid 3D molecules is challenging due to bond topology errors.
method HLTF uses a latent multi-scale plan for global context and a constraint-aware sampler to suppress topology-driven failures.
result HLTF achieves high validity and uniqueness on QM9 and GEOM-DRUGS datasets.
Recent studies have shown that tuning prediction models increases prediction accuracy and that Random Forest can be used to construct prediction intervals. However, to our best knowledge, no study has investigated the need to, and the manner in which one can, tune Random Forest for optimizing prediction intervals { thi…
Optimizes Lasso hyperparameters using leave-one-out CV.
problem Finding optimal hyperparameters for Lasso regression.
method Develops an algorithm to compute exact or approximate leave-one-out CV.
result Algorithm finds optimal hyperparameters for Lasso.
We introduce an exploratory study on Mutation Validation (MV), a model validation method using mutated training labels for supervised learning. MV mutates training data labels, retrains the model against the mutated data, then uses the metamorphic relation that captures the consequent training performance changes to as…
A new model validation framework for agentic AI systems based on POMDPs.
problem Model validation of agentic AI systems.
method A POMDP-based framework for belief-state, forecast, and policy validation.
result The framework decomposes autonomous decision making into information, beliefs, forecasts, actions, and utility.
Proposes a falsification framework to test algorithmic discriminant validity.
problem Unintended model behavior in predictive algorithms.
method Falsification framework based on statistical tests comparing prediction losses across outcomes.
result Establishes discriminant validity for some outcomes but not others.
Study evaluates cross-validation methods for clinical ECG classification, finding leave-source-out more reliable.
problem Overoptimistic cross-validation estimates for new patient sources.
method Empirical evaluation of K-fold and leave-source-out cross-validation methods.
result Leave-source-out cross-validation provides more reliable performance estimates.
Simplifies machine learning validation using kNN and conditional probability algorithms.
problem Validating machine learning models in practical applications.
method Reformulated regression and classification problems using kNN and conditional probability algorithms.
result Online capability and reduced memory usage compared to kNN.
OpenAlpha validates decentralized capital strategies using game theory and market aggregation.
problem Decentralized capital management's lack of trust-minimised, adaptive deployment.
method Game-theoretic validation, adversarial auditing, market-based belief aggregation.
result Confidence scores from validation phases inform capital allocation rules.
Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that the cross-validation risk is differentiable with respect to the hyperparameters and training data for …