The paper offers guidelines for validating data-driven models.
problem Ensuring reliable validation of data-driven models.
method A set of general rules for model validation.
result Helps practitioners create reliable validation plans and report results transparently.
Active learning method improves local model validity estimation.
problem Ensuring local model validity in machine learning applications.
method Learning model error to estimate local validity using active learning.
result The proposed method can estimate local validity with a small amount of data.
A fast bootstrap method estimates cross-validation standard error.
problem Uncertainty quantification in cross-validation estimates.
method Random-effects model to estimate variance component.
result Valid confidence intervals for model performance.
Improving cancer treatment decisions requires considering causal effects, not just model accuracy.
problem Cancer outcome prediction models may cause harm when used for treatment decisions.
method Explains the importance of considering causal effects in model validation and provides guidelines.
result Building and validating models that are useful for decision making requires considering causal effects.
Deep generative models have been successfully used to learn representations for high-dimensional discrete spaces by representing discrete objects as sequences and employing powerful sequence-based deep models. Unfortunately, these sequence-based models often produce invalid sequences: sequences which do not represent a…
Study compares mutation validation and cross-validation for model selection.
problem Comparing model selection methods for generalization performance and computational efficiency.
method Empirical comparison using benchmark and real-world datasets with Bayesian tests.
result Both methods select models with equivalent generalization performance but MV selects simpler models and is computationally cheaper.
We introduce an exploratory study on Mutation Validation (MV), a model validation method using mutated training labels for supervised learning. MV mutates training data labels, retrains the model against the mutated data, then uses the metamorphic relation that captures the consequent training performance changes to as…
Model selection on validation data is an essential step in machine learning. While the mixing of data between training and validation is considered taboo, practitioners often violate it to increase performance. Here, we offer a simple, practical method for using the validation set for training, which allows for a conti…
Improves test set performance and reduces out-of-sample disappointment for unstable models.
problem Ensuring strong test set performance via cross-validation for unstable models.
method Nested k-fold cross-validation with hyperparameter selection based on a weighted sum of cross-validation metric and model stability measure.
result Improves out-of-sample MSE for sparse ridge regression and CART by 4% and 2% respectively, compared to k-fold cross-validation.
A new model validation framework for agentic AI systems based on POMDPs.
problem Model validation of agentic AI systems.
method A POMDP-based framework for belief-state, forecast, and policy validation.
result The framework decomposes autonomous decision making into information, beliefs, forecasts, actions, and utility.
Proposes a framework to explain KS deterioration in credit risk models.
problem Inconsistent and ad hoc diagnosis of KS decline in credit risk models.
method Counterfactual diagnostic framework attributing KS decline to sampling variability, portfolio composition, covariate shift, and residual deterioration.
result The proposed approach provides more interpretable and governance-relevant explanations than threshold-based review alone.
New CV method reduces bias in spatial prediction models.
problem Bias in standard cross-validation due to uneven sampling.
method Target-Weighted Cross-Validation (TWCV) framework.
result Weighted CV approaches reduce bias in prediction error.
A new cross-validation method reduces redundancy and improves model performance.
problem Redundancy in traditional k-fold cross-validation leads to biased results.
method Irredundant k-fold cross-validation, where each instance is used exactly once for training and testing.
result Consistent performance estimates with reduced variance and lower computational cost.
Cross validation residuals are well known for the ordinary least squares model. Here leave-M-out cross validation is extended to generalised least squares. The relationship between cross validation residuals and Cook's distance is demonstrated, in terms of an approximation to the difference in the generalised residual …
Study validates metrics for offline MBO using diffusion models.
problem Evaluate metrics for offline MBO without ground truth oracle.
method Propose and quantify validation metrics over datasets.
result Identify most effective validation metrics.
Paper revisits pre-validation method, improving hypothesis testing.
problem Improving hypothesis testing in pre-validated models with different feature dimensions.
method Extended problem formulation, analytical distribution, and bootstrap procedure.
result Proposed analytical distribution and bootstrap procedure for pre-validated predictors.
Deep learning techniques have been hugely successful for traditional supervised and unsupervised machine learning problems. In large part, these techniques solve continuous optimization problems. Recently however, discrete generative deep learning models have been successfully used to efficiently search high-dimensiona…
Proposes a new cross-validation method to estimate model performance.
problem The standard cross-validation method does not accurately estimate the performance of the recommended model.
method Develops a new random-effects model framework to improve naive cross-validation estimators.
result Proposed estimators outperform conventional and naive methods in estimating model performance.
Combines trial and observational data to improve policy evaluation.
problem External validity of randomized trial results in target populations.
method Uses covariate data to model trial sampling and certifies policy evaluations.
result Valid trial-based policy evaluations under model miscalibration.
Linear principal component analysis (PCA) can be extended to a nonlinear PCA by using artificial neural networks. But the benefit of curved components requires a careful control of the model complexity. Moreover, standard techniques for model selection, including cross-validation and more generally the use of an indepe…
Cross-validation estimates model performance on unseen data, not training data.
problem Understanding how cross-validation estimates prediction error and its limitations.
method Analyzing linear models and popular prediction error estimates, introducing nested cross-validation.
result Cross-validation estimates the average prediction error of models fit on other unseen training sets, not the model at hand.
Cross-regularization adapts model complexity during training.
problem Manual tuning of model complexity for overfitting prevention.
method Directly adapts regularization parameters through validation gradients during training.
result Organic emergence of architecture-specific regularization during training.
Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its apparent universality. Many results exist on the model selection performances of cross-validation procedures. This survey intends to relate these results to the most recen…
The paper assesses quality measures for machine learning models using cross-validation.
problem Evaluating the accuracy and robustness of quality measures for machine learning models.
method Cross-validation approach to estimate prediction error and quantify explained variation. Confidence bounds and local quality measures derived from residuals.
result The reliability and robustness of quality measures are assessed through numerical examples and confidence bounds.
A new method validates generative models in high-dimensional data.
problem Scalability and interpretability issues in validating generative models.
method Learning-based goodness-of-fit testing inspired by Neyman--Pearson construction.
result The NPLM can effectively validate generative models in high-dimensional data.
In this paper, we introduce a new concept of stability for cross-validation, called the (β,ϖ)-stability, and use it as a new perspective to build the general theory for cross-validation. The (β,ϖ)-stability mathematically connects the generalization ability and the stability of…
Cross-validation methods help learn dynamical systems from data.
problem Learning surrogate models for dynamical systems from limited data.
method Variants of cross-validation (Kernel Flows, MMD, Lyapunov exponents).
result Simple approaches for kernel selection in dynamical system emulators.
Researchers validate ML scenario generators by checking dependencies and detecting memorization effects.
problem Validation of machine learning-based scenario generators differs from classical methods due to data-driven dependencies.
method Two novel validation aspects: checking dependencies and detecting memorization effects. Novel memorization ratio introduced.
result Validation methods successfully detect dependencies and memorization effects in ML-based scenario generators.
Develops a prediction method based on sampling design.
problem Creating accurate individual predictions.
method Design-based approach using expected cross-validation results.
result Valid inference of unobserved prediction errors defined with respect to sampling design.
CV inference can be invalid for relatively unstable model comparisons.
problem The validity of cross-validation for model comparison is questioned when models are relatively unstable.
method The study proves that simple, individually stable models can generate relatively unstable comparisons, invalidating CV inference.
result The Lasso and soft-thresholding generate relatively unstable comparisons, invalidating CV inferences.
Paper validates ABM using stylized financial facts.
problem Validate ABM-generated financial data against real-world data.
method Compare ABM results with stylized financial facts.
result Model successfully replicates stylized financial facts.
MoFlow generates chemically valid molecular graphs from latent representations.
problem Generating chemically valid molecular graphs from latent representations is challenging.
method MoFlow uses a flow-based approach with Glow for bond generation and a novel graph conditional flow for atom generation, ensuring chemical validity and efficiency.
result MoFlow achieves state-of-the-art performance in molecular graph generation and optimization.
Cross-validation is one of the most popular model selection methods in statistics and machine learning. Despite its wide applicability, traditional cross validation methods tend to select overfitting models, due to the ignorance of the uncertainty in the testing sample. We develop a new, statistically principled infere…
We speed up Gaussian process cross-validation calculations and improve model diagnostics.
problem Efficiently calculating cross-validation residuals and their covariances in Gaussian processes.
method Generalized fast Gaussian process leave-one-out formulae to multiple-fold cross-validation, highlighting covariance structures.
result Correcting for residual covariances in cross-validation improves back to Maximum Likelihood Estimation.
The article proposes a method to make valid insurance claim predictions without relying on specific models.
problem Prediction of insurance claims using statistical models can be unreliable due to model misspecification, selection effects, and lack of finite-sample validity.
method The article employs conformal prediction, a machine learning strategy that is model-free and tuning-parameter-free, ensuring finite-sample validity.
result The proposed method guarantees valid predictions at a pre-assigned coverage probability level and performs well in insurance applications, including meeting Solvency II requirements.
A method to select validation data from a dataset using statistical criteria.
problem Selecting a validation basis from a full dataset for machine learning model validation.
method Adopting a 'design of experiments' point of view and using statistical criteria, particularly Maximum Mean Discrepancy criteria.
result The 'support points' concept is particularly relevant for selecting validation data.
Study evaluates cross-validation methods for clinical ECG classification, finding leave-source-out more reliable.
problem Overoptimistic cross-validation estimates for new patient sources.
method Empirical evaluation of K-fold and leave-source-out cross-validation methods.
result Leave-source-out cross-validation provides more reliable performance estimates.
A method for efficient CV estimates in Bayesian hierarchical models.
problem Computational infeasibility of cross-validation in Bayesian hierarchical regression models.
method Conditioning on variance-covariance parameters to transform CV into an optimization problem.
result Equivalent or improved predictive estimates compared to full cross-validation.
Machine learning detects survey validity from user behavior.
problem Detecting valid responses in web surveys.
method Uses mouse activity and machine learning models (LSTM, HMM).
result Predicts survey validity without analyzing specific answers.
This paper improves model selection with cross-validation using domain knowledge.
problem Improving model selection with cross-validation risk estimation.
method Establishes distribution-free deviation bounds using VC dimension, formalizes Learning Spaces based on domain knowledge.
result Enhanced generalization through selection of candidate models based on domain knowledge.
The paper validates statistical models for groundwater data.
problem Validating statistical models for groundwater data.
method Traditional time-series models and modern neural networks.
result Validation techniques ensure lower computational cost and robust predictions.
Stacked conformal prediction simplifies model validation.
problem Validating stacked predictive models efficiently.
method Meta-learner at the top of a stacked ensemble for approximate marginal validity.
result The method achieves approximate marginal validity without a separate calibration sample.
New method uses spherical convolutional Wasserstein distance to validate climate models.
problem Ensuring the accuracy of global climate models.
method Spherical convolutional Wasserstein distance to measure model differences.
result Phase 6 models show modest improvements in realistic climatologies.
Improved GP models for fast training and good performance.
problem Training scalable Gaussian process models efficiently.
method Cross-validation and nearest neighbor truncation for scalable GP training.
result Our method offers fast training and excellent predictive performance.
This paper describes the current taxonomy of model risk, ways for its mitigation and management and the importance of the model validation function in collaboration with other departments to design and implement them.
CPTD improves prediction intervals in time series regression with cross-sectional data.
problem Constructing valid prediction intervals in time series regression with a cross-section.
method Conformal Prediction with Temporal Dependence (CPTD) for post-hoc, light-weight approach.
result CPTD maintains cross-sectional validity while improving longitudinal coverage.
New IF method improves accuracy in deep neural networks with noisy data.
problem Inaccurate influence estimates in deep neural networks, especially with noisy data.
method Established a connection between influence estimation error, validation set risk, and sharpness, introducing a novel estimation form for flat validation minima.
result Our novel Influence Function approach provides more accurate influence estimates, validated across various tasks.
This paper tackles deep clustering evaluation challenges in high-dimensional data.
problem Evaluation of deep clustering methods is problematic due to the curse of dimensionality and variations in embedding spaces.
method Develops a theoretical framework to highlight the ineffectiveness of internal validation measures and proposes a systematic approach to applying clustering validity indices in deep learning.
result The proposed framework reduces misguidance from improper use of clustering validity indices in deep learning.