arXiv research

Study develops a method to select penalty parameters for sparse neural networks without cross-validation.

The future predictive performance of a Bayesian model can be estimated using Bayesian cross-validation. In this article, we consider Gaussian latent variable models where the integration over the latent values is approximated using the Laplace method or expectation propagation (EP). We study the properties of several B…

Statistical machine learning models should be evaluated and validated before putting to work. Conventional k-fold Monte Carlo Cross-Validation (MCCV) procedure uses a pseudo-random sequence to partition instances into k subsets, which usually causes subsampling bias, inflates generalization errors and jeopardizes the r…

Paper analyzes holdout cross-validation for large non-Gaussian covariance estimation.

Monte Carlo (MC) techniques are often used to estimate integrals of a multivariate function using randomly generated samples of the function. In light of the increasing interest in uncertainty quantification and robust design applications in aerospace engineering, the calculation of expected values of such functions (e…

Paper compares ML models for a wall-following robot, achieving high accuracy.

A hybrid algorithm fuses significance-based splitting with honest sample-splitting for estimating heterogeneous treatment effects.

In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for empirical risk minimizers. In the general setting, we prove sanity-check bounds in the spirit of \cite{KR99} \textquotedblleft\textit{bounds showing that the worst-case error of this estimate is not m…

Adaptive importance sampling is a class of techniques for finding good proposal distributions for importance sampling. Often the proposal distributions are standard probability distributions whose parameters are adapted based on the mismatch between the current proposal and a target distribution. In this work, we prese…

In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for stable predictors in the context of risk assessment. The notion of stability has been first introduced by \cite{DEWA79} and extended by \cite{KEA95}, \cite{BE01} and \cite{KUNIY02} to characterize cla…

In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for subagged estimators, both for classification and regressor. General loss functions and class of predictors with both finite and infinite VC-dimension are considered. We slightly generalize the formali…

When selecting a classification algorithm to be applied to a particular problem, one has to simultaneously select the best algorithm for that dataset \emph{and} the best set of hyperparameters for the chosen model. The usual approach is to apply a nested cross-validation procedure; hyperparameter selection is performed…

A fast bootstrap method estimates cross-validation standard error.

New cross-validation methods for Gaussian process regression with efficient gradient computation.

Cross-validation estimates model performance on unseen data, not training data.

Improves test set performance and reduces out-of-sample disappointment for unstable models.

The paper develops a cross-validation method for improving signal denoising techniques.

Optimizes Lasso hyperparameters using leave-one-out CV.

Study evaluates cross-validation methods for clinical ECG classification, finding leave-source-out more reliable.

A new method improves super learner validation efficiency.

Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that the cross-validation risk is differentiable with respect to the hyperparameters and training data for …

Paper introduces stability in model averaging and proposes a L2-penalty method.

Cross-validation methods help learn dynamical systems from data.

This text is a survey on cross-validation. We define all classical cross-validation procedures, and we study their properties for two different goals: estimating the risk of a given estimator, and selecting the best estimator among a given family. For the risk estimation problem, we compute the bias (which can also be …

CVTMLE improves statistical inference in settings of positivity or Donsker class violations.

Bayesian Additive Regression Networks use neural networks for regression tasks.

The paper improves confidence intervals for test error using cross-validation.

Cross validation residuals are well known for the ordinary least squares model. Here leave-M-out cross validation is extended to generalised least squares. The relationship between cross validation residuals and Cook's distance is demonstrated, in terms of an approximation to the difference in the generalised residual …

A new cross-validation method reduces redundancy and improves model performance.

Proposes a new cross-validation method to estimate model performance.

With the increasing size of today's data sets, finding the right parameter configuration in model selection via cross-validation can be an extremely time-consuming task. In this paper we propose an improved cross-validation procedure which uses nonparametric testing coupled with sequential analysis to determine the bes…

Sparse APCA identifies sparse factors in financial returns over time.

In this paper, we introduce a new concept of stability for cross-validation, called the $\left( β, \varpi \right)$-stability, and use it as a new perspective to build the general theory for cross-validation. The $\left( β, \varpi \right)$-stability mathematically connects the generalization ability and the stability of…

We speed up Gaussian process cross-validation calculations and improve model diagnostics.

Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its apparent universality. Many results exist on the model selection performances of cross-validation procedures. This survey intends to relate these results to the most recen…

Cross-validation is one of the most popular model selection methods in statistics and machine learning. Despite its wide applicability, traditional cross validation methods tend to select overfitting models, due to the ignorance of the uncertainty in the testing sample. We develop a new, statistically principled infere…

K-fold Cross Validation is commonly used to evaluate classifiers and tune their hyperparameters. However, it assumes that data points are Independent and Identically Distributed (i.i.d.) so that samples used in the training and test sets can be selected randomly and uniformly. In Human Activity Recognition datasets, we…

The paper assesses quality measures for machine learning models using cross-validation.

Improved LOO cross-validation for function approximation.

A method for efficient CV estimates in Bayesian hierarchical models.

We show how to adjust the coefficient of determination ($R^2$) when used for measuring predictive accuracy via leave-one-out cross-validation.

New method for cross-validation in high-dimensional data with dependent or heavy-tailed covariates.

Decoding, ie prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didacti…

Optimizes hyperparameter tuning for models using approximate leave-one-out cross-validation.

Study improves predictive performance testing for high-dimensional data using exhaustive nested cross-validation.

In Bayesian statistics, the marginal likelihood, also known as the evidence, is used to evaluate model fit as it quantifies the joint probability of the data under the prior. In contrast, non-Bayesian models are typically compared using cross-validation on held-out data, either through $k$-fold partitioning or leave-$p…

Improved GP models for fast training and good performance.

We investigate the accuracy of the two most common estimators for the maximum expected value of a general set of random variables: a generalization of the maximum sample average, and cross validation. No unbiased estimator exists and we show that it is non-trivial to select a good estimator without knowledge about the …