When selecting a classification algorithm to be applied to a particular problem, one has to simultaneously select the best algorithm for that dataset \emph{and} the best set of hyperparameters for the chosen model. The usual approach is to apply a nested cross-validation procedure; hyperparameter selection is performed…
Simplifies machine learning validation using kNN and conditional probability algorithms.
problem Validating machine learning models in practical applications.
method Reformulated regression and classification problems using kNN and conditional probability algorithms.
result Online capability and reduced memory usage compared to kNN.
Proposes a test to ensure predictive algorithms predict intended outcomes better than unintended ones.
problem Unintended model behavior leading to prediction of unintended outcomes.
method Falsification framework using nonparametric hypothesis testing to compare prediction losses across outcomes.
result Establishes discriminant validity with respect to gender but not race in an admissions setting.
Proposes a falsification framework to test algorithmic discriminant validity.
problem Unintended model behavior in predictive algorithms.
method Falsification framework based on statistical tests comparing prediction losses across outcomes.
result Establishes discriminant validity for some outcomes but not others.
The paper improves confidence intervals for test error using cross-validation.
problem Improving confidence intervals for test error in machine learning.
method Develops central limit theorems and consistent estimators for cross-validation.
result Provides asymptotically-exact confidence intervals and hypothesis tests.
Optimizes Lasso hyperparameters using leave-one-out CV.
problem Finding optimal hyperparameters for Lasso regression.
method Develops an algorithm to compute exact or approximate leave-one-out CV.
result Algorithm finds optimal hyperparameters for Lasso.
Model selection on validation data is an essential step in machine learning. While the mixing of data between training and validation is considered taboo, practitioners often violate it to increase performance. Here, we offer a simple, practical method for using the validation set for training, which allows for a conti…
Study benchmarks 26 clustering validity measures.
problem Determining the best clustering solution from candidates.
method Enhanced revision of previous methodology with three sub-methodologies.
result Comprehensive evaluation of 26 internal validity indexes.
A new method improves super learner validation efficiency.
problem Improving the efficiency of super learner validation.
method Bootstrap Bias Corrected Cross Validation applied to Super Learning.
result Bootstrap Bias Corrected Cross Validation proved efficient and cost-effective.
Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that the cross-validation risk is differentiable with respect to the hyperparameters and training data for …
This thesis explores how submodularity aids in optimizing non-convex functions and validating algorithms.
problem Understanding which functions can be optimized efficiently in non-convex settings.
method Introducing continuous submodularity and developing algorithms for maximizing these functions.
result Characterization and optimization of continuous submodular functions with strong guarantees.
Practical model building processes are often time-consuming because many different models must be trained and validated. In this paper, we introduce a novel algorithm that can be used for computing the lower and the upper bounds of model validation errors without actually training the model itself. A key idea behind ou…
Survey of algorithms for testing AI-driven CPS safety.
problem Testing AI-driven CPS for safety in complex environments.
method Survey of applied algorithms for safety validation.
result Survey of existing tools and techniques for safety validation.
Valid inference from data and predictions.
problem Valid statistical inference with machine learning predictions.
method Framework for valid inference using machine learning predictions.
result Valid confidence intervals without assumptions on predictions.
This work proposes validation diagnostics for SBI algorithms using Normalizing Flows.
problem Lack of appropriate validation methods for SBI algorithms with complex, high-dimensional data.
method Develops validation diagnostics based on Normalizing Flows with theoretical guarantees.
result Offers theoretical guarantees on consistency of NF-based estimators.
No free lunch theorems show all algorithms perform equally under uniform distribution.
problem Analyzing scenarios involving non-uniform distributions and comparing algorithms.
method No Free Lunch theorems applied to analyze and compare algorithms without distribution assumptions.
result Anti-cross-validation performs as well as cross-validation under non-uniform distributions.
Bootstrap aggregation, known as bagging, is one of the most popular ensemble methods used in machine learning (ML). An ensemble method is a ML method that combines multiple hypotheses to form a single hypothesis used for prediction. A bagging algorithm combines multiple classifiers modeled on different sub-samples of t…
This paper reviews and proposes a new approach for evaluating internal cluster validation indices.
problem Selecting the best-performing unsupervised classification algorithm without external information.
method Examines and proposes a new evaluation approach for internal validation indices.
result Suggests a new evaluation approach for internal validation indices.
A new measure normalizes clustering accuracy to evaluate algorithms better.
problem Evaluation of clustering algorithms is challenging due to limitations of existing measures.
method Proposes a new, normalised clustering accuracy measure.
result The new measure identifies worst-case scenarios and is more interpretable.
The correct use of model evaluation, model selection, and algorithm selection techniques is vital in academic machine learning research as well as in many industrial settings. This article reviews different techniques that can be used for each of these three subtasks and discusses the main advantages and disadvantages …
Develops a validated trading framework for market microstructure signals.
problem Overfitting and lookahead bias in algorithmic trading.
method Interpretable hypothesis-driven signal generation, reinforcement learning, strict out-of-sample testing.
result Modest annualized returns with strong downside protection and market-neutral characteristics.
New method optimizes hyperparameters for non-smooth problems efficiently.
problem Efficiently tuning hyperparameters for non-smooth cost functions.
method Combines hyperparameter search with proximal gradient updates.
result Method converges to local optimum of LOO validation error.
New validation method prevents privacy breaches and biases in federated learning.
problem Privacy breaches and data leakage in federated learning.
method Stratified cross-validation for unbiased and privacy-preserving federated learning.
result Stratified cross-validation prevents data leakage without demanding deduplication algorithms.
Image segmentation is one of the most fundamental tasks of computer vision. In many practical applications, it is essential to properly evaluate the reliability of individual segmentation results. In this study, we propose a novel framework to provide the statistical significance of segmentation results in the form of …
New framework ensures valid uncertainty estimates for any data stream changes.
problem Challenges of distribution shifts and adversarial actors in real-world data streams.
method Leveraging Blackwell approachability from game theory, the framework guarantees calibrated uncertainties for any compact space.
result Improves calibration and decision-making for energy systems.
Confounding bias, missing data, and selection bias are three common obstacles to valid causal inference in the data sciences. Covariate adjustment is the most pervasive technique for recovering casual effects from confounding bias. In this paper, we introduce a covariate adjustment formulation for controlling confoundi…
This work establishes always-valid risk bounds for online matrix completion.
problem Challenges in establishing always-valid concentration inequalities for online matrix completion.
method Combines non-asymptotic martingale concentration and regularized low-rank matrix regression.
result Establishes always-valid risk bound process for online matrix completion.
Improving cancer treatment decisions requires considering causal effects, not just model accuracy.
problem Cancer outcome prediction models may cause harm when used for treatment decisions.
method Explains the importance of considering causal effects in model validation and provides guidelines.
result Building and validating models that are useful for decision making requires considering causal effects.
New algorithm reduces online hyperparameter optimization costs.
problem High cost of evaluating validation examples in online HPO.
method Modeling online HPO as a time-varying Bayesian optimization problem, proposing a costly feedback setting.
result Cost-efficient GP-UCB algorithm reaches human expert-level performance.
New method improves online nonparametric estimators with minimal extra computation.
problem Model selection and hyperparameter tuning for online nonparametric estimators.
method Weighted rolling validation procedure for online cross-validation.
result Improves base estimators to achieve better heuristic performance and adaptive convergence rate.
We consider a covariate shift problem where one has access to several different training datasets for the same learning problem and a small validation set which possibly differs from all the individual training distributions. This covariate shift is caused, in part, due to unobserved features in the datasets. The objec…
Improved TreNet for trend prediction in time series data.
problem Validation method for TreNet did not account for time series data's sequential nature.
method Walk-forward validation method and multiple independent runs to evaluate model stability.
result TreNet still performs better than vanilla DNN models but not on all data sets.
A method to select validation data from a dataset using statistical criteria.
problem Selecting a validation basis from a full dataset for machine learning model validation.
method Adopting a 'design of experiments' point of view and using statistical criteria, particularly Maximum Mean Discrepancy criteria.
result The 'support points' concept is particularly relevant for selecting validation data.
Simulation workflow is a top-level model for the design and control of simulation process. It connects multiple simulation components with time and interaction restrictions to form a complete simulation system. Before the construction and evaluation of the component models, the validation of upper-layer simulation work…
New method uses weak labels to create valid confidence sets for predictions.
problem Lack of labeled data in machine learning models.
method Developed a conformal prediction framework to provide valid predictive confidence sets using weakly labeled data.
result New coverage definition allows for tighter and more informative (but valid) confidence sets.
We introduce an exploratory study on Mutation Validation (MV), a model validation method using mutated training labels for supervised learning. MV mutates training data labels, retrains the model against the mutated data, then uses the metamorphic relation that captures the consequent training performance changes to as…
Echo State Networks (ESNs) are known for their fast and precise one-shot learning of time series. But they often need good hyper-parameter tuning for best performance. For this good validation is key, but usually, a single validation split is used. In this rather practical contribution we suggest several schemes for cr…
In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for stable predictors in the context of risk assessment. The notion of stability has been first introduced by \cite{DEWA79} and extended by \cite{KEA95}, \cite{BE01} and \cite{KUNIY02} to characterize cla…
FastMuyGPs speeds up GP predictions for large datasets.
problem High cost of Gaussian process predictions for large data.
method Combines cross-validation, batching, nearest neighbors sparsification, and precomputation.
result Superior accuracy and competitive runtime compared to other methods.
Proposes BSI for valid statistical inference on bandit algorithms.
problem Valid statistical inference on bandit algorithms' performance.
method Fits a simulator of the bandit environment from observed data and uses it to estimate mean reward under any policy.
result Proves asymptotically valid confidence intervals and maintains nominal coverage.
Bayesian online learning algorithm for one-pass data, achieving frequentist validity and uncertainty quantification.
problem Theoretical limitations in Bayesian online learning, especially in the one-pass setting.
method Proposed a new Bayesian online learning algorithm with a warm-start phase for the one-pass regime, establishing convergence rates and valid uncertainty quantification.
result The sequentially updated posterior attains optimal convergence rates and valid uncertainty quantification without diverging mini-batch sample sizes.
Generalizes conformal prediction to multiple learnable parameters for efficient prediction sets.
problem Learning valid and efficient prediction sets with low-capacity function classes.
method Constrained empirical risk minimization (ERM) with gradient-based optimization of differentiable surrogate losses and Lagrangians.
result Achieves approximate valid population coverage and near-optimal efficiency within class.
This study validates BN structure learning algorithms under noisy data, revealing performance discrepancies.
problem Inconsistent performance claims across BN structure learning algorithms due to inconsistent evaluation methods.
method Applied 15 algorithms to noisy data in multiple studies, evaluating with various criteria.
result Traditional synthetic performance may overestimate real-world performance by 10-50%.
A fast bootstrap method estimates cross-validation standard error.
problem Uncertainty quantification in cross-validation estimates.
method Random-effects model to estimate variance component.
result Valid confidence intervals for model performance.
In the modern era, abundant information is easily accessible from various sources, however only a few of these sources are reliable as they mostly contain unverified contents. We develop a system to validate the truthfulness of a given statement together with underlying evidence. The proposed system provides supporting…
Study compares mutation validation and cross-validation for model selection.
problem Comparing model selection methods for generalization performance and computational efficiency.
method Empirical comparison using benchmark and real-world datasets with Bayesian tests.
result Both methods select models with equivalent generalization performance but MV selects simpler models and is computationally cheaper.
In this paper, we propose a one-pass algorithm on MapReduce for penalized linear regression \[f_λ(α, β) = \|Y - α\mathbf{1} - Xβ\|_2^2 + p_λ(β)\] where α is the intercept which can be omitted depending on application; β is the coefficients and pλ is the penalized function with penalizing parameter λ. $f_λ(α, β…
Adapts conformal prediction for missing data, ensuring valid coverage.
problem Uncertainty quantification with missing covariates.
method Proposes a reweighted conformal prediction procedure for handling missing values.
result Guaranteed Marginal Coverage and Mask-Conditional Validity for general missing data mechanisms.