Cross-validation is the de facto standard for predictive model evaluation and selection. In proper use, it provides an unbiased estimate of a model's predictive performance. However, data sets often undergo various forms of data-dependent preprocessing, such as mean-centering, rescaling, dimensionality reduction, and o…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “cross-validation reduction”
Supervised linear feature extraction can be achieved by fitting a reduced rank multivariate model. This paper studies rank penalized and rank constrained vector generalized linear models. From the perspective of thresholding rules, we build a framework for fitting singular value penalized models and use it for feature …
The strength of association between a pair of data vectors is represented by a nonnegative real number, called matching weight. For dimensionality reduction, we consider a linear transformation of data vectors, and define a matching error as the weighted sum of squared distances between transformed vectors with respect…
The paper evaluates company investment value using machine learning models.
problem Evaluating the investment value of companies based on machine learning.
method Data mining, feature selection, cross-validation, stacking model, Bayesian Ridge Regression.
result The RMSE of the final model is 3.047, indicating improved stability and generalization.
In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for empirical risk minimizers. In the general setting, we prove sanity-check bounds in the spirit of \cite{KR99} \textquotedblleft\textit{bounds showing that the worst-case error of this estimate is not m…
This paper addresses overfitting in dimension reduction methods by calibrating hyperparameters considering noise.
problem Overfitting in dimension reduction methods, especially t-SNE and UMAP, when data contains noise.
method Present a framework to calibrate hyperparameters in the presence of noise for t-SNE and UMAP.
result Recommended hyperparameter values for t-SNE and UMAP are too small and overfit the noise.
In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for stable predictors in the context of risk assessment. The notion of stability has been first introduced by \cite{DEWA79} and extended by \cite{KEA95}, \cite{BE01} and \cite{KUNIY02} to characterize cla…
In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for subagged estimators, both for classification and regressor. General loss functions and class of predictors with both finite and infinite VC-dimension are considered. We slightly generalize the formali…
Nested cross-validation when selecting classifiers is overzealous for most practical applicationscs.LG
When selecting a classification algorithm to be applied to a particular problem, one has to simultaneously select the best algorithm for that dataset \emph{and} the best set of hyperparameters for the chosen model. The usual approach is to apply a nested cross-validation procedure; hyperparameter selection is performed…
A fast bootstrap method estimates cross-validation standard error.
problem Uncertainty quantification in cross-validation estimates.
method Random-effects model to estimate variance component.
result Valid confidence intervals for model performance.
New cross-validation methods for Gaussian process regression with efficient gradient computation.
problem Estimating parameters of Gaussian process covariance functions.
method Derive new cross-validation criteria and efficient adjoint computation of gradients.
result Efficient method for evaluating cross-validation criteria and their gradients.
A hybrid model combines machine learning with a land surface model to improve soil moisture predictions.
problem Improving soil moisture predictions in climatological situations.
method Noah land-surface model integrated with Gaussian Processes, using autoregressive model for out-of-sample results.
result 3-fold reduction in RMSE using one-year leave-one-out cross-validation.
Cross-validation estimates model performance on unseen data, not training data.
problem Understanding how cross-validation estimates prediction error and its limitations.
method Analyzing linear models and popular prediction error estimates, introducing nested cross-validation.
result Cross-validation estimates the average prediction error of models fit on other unseen training sets, not the model at hand.
Improves test set performance and reduces out-of-sample disappointment for unstable models.
problem Ensuring strong test set performance via cross-validation for unstable models.
method Nested k-fold cross-validation with hyperparameter selection based on a weighted sum of cross-validation metric and model stability measure.
result Improves out-of-sample MSE for sparse ridge regression and CART by 4% and 2% respectively, compared to k-fold cross-validation.
The paper develops a cross-validation method for improving signal denoising techniques.
problem Improving signal denoising methods for nonparametric regression.
method Develops a general cross-validation framework for signal denoising and applies it to Trend Filtering and Dyadic CART.
result Cross validated versions of Trend Filtering and Dyadic CART achieve nearly optimal convergence rates.
Optimizes Lasso hyperparameters using leave-one-out CV.
problem Finding optimal hyperparameters for Lasso regression.
method Develops an algorithm to compute exact or approximate leave-one-out CV.
result Algorithm finds optimal hyperparameters for Lasso.
Study evaluates cross-validation methods for clinical ECG classification, finding leave-source-out more reliable.
problem Overoptimistic cross-validation estimates for new patient sources.
method Empirical evaluation of K-fold and leave-source-out cross-validation methods.
result Leave-source-out cross-validation provides more reliable performance estimates.
A new method improves super learner validation efficiency.
problem Improving the efficiency of super learner validation.
method Bootstrap Bias Corrected Cross Validation applied to Super Learning.
result Bootstrap Bias Corrected Cross Validation proved efficient and cost-effective.
Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that the cross-validation risk is differentiable with respect to the hyperparameters and training data for …
Cross-validation methods help learn dynamical systems from data.
problem Learning surrogate models for dynamical systems from limited data.
method Variants of cross-validation (Kernel Flows, MMD, Lyapunov exponents).
result Simple approaches for kernel selection in dynamical system emulators.
Cross-validationmath.ST
This text is a survey on cross-validation. We define all classical cross-validation procedures, and we study their properties for two different goals: estimating the risk of a given estimator, and selecting the best estimator among a given family. For the risk estimation problem, we compute the bias (which can also be …
The paper improves confidence intervals for test error using cross-validation.
problem Improving confidence intervals for test error in machine learning.
method Develops central limit theorems and consistent estimators for cross-validation.
result Provides asymptotically-exact confidence intervals and hypothesis tests.
Cross validation residuals are well known for the ordinary least squares model. Here leave-M-out cross validation is extended to generalised least squares. The relationship between cross validation residuals and Cook's distance is demonstrated, in terms of an approximation to the difference in the generalised residual …
A new cross-validation method reduces redundancy and improves model performance.
problem Redundancy in traditional k-fold cross-validation leads to biased results.
method Irredundant k-fold cross-validation, where each instance is used exactly once for training and testing.
result Consistent performance estimates with reduced variance and lower computational cost.
With the increasing size of today's data sets, finding the right parameter configuration in model selection via cross-validation can be an extremely time-consuming task. In this paper we propose an improved cross-validation procedure which uses nonparametric testing coupled with sequential analysis to determine the bes…
Proposes a new cross-validation method to estimate model performance.
problem The standard cross-validation method does not accurately estimate the performance of the recommended model.
method Develops a new random-effects model framework to improve naive cross-validation estimators.
result Proposed estimators outperform conventional and naive methods in estimating model performance.
In this paper, we introduce a new concept of stability for cross-validation, called the -stability, and use it as a new perspective to build the general theory for cross-validation. The -stability mathematically connects the generalization ability and the stability of…
Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its apparent universality. Many results exist on the model selection performances of cross-validation procedures. This survey intends to relate these results to the most recen…
Fast calculation of Gaussian Process multiple-fold cross-validation residuals and their covariancesstat.ME
We speed up Gaussian process cross-validation calculations and improve model diagnostics.
problem Efficiently calculating cross-validation residuals and their covariances in Gaussian processes.
method Generalized fast Gaussian process leave-one-out formulae to multiple-fold cross-validation, highlighting covariance structures.
result Correcting for residual covariances in cross-validation improves back to Maximum Likelihood Estimation.
Cross-validation is one of the most popular model selection methods in statistics and machine learning. Despite its wide applicability, traditional cross validation methods tend to select overfitting models, due to the ignorance of the uncertainty in the testing sample. We develop a new, statistically principled infere…
K-fold Cross Validation is commonly used to evaluate classifiers and tune their hyperparameters. However, it assumes that data points are Independent and Identically Distributed (i.i.d.) so that samples used in the training and test sets can be selected randomly and uniformly. In Human Activity Recognition datasets, we…
The paper assesses quality measures for machine learning models using cross-validation.
problem Evaluating the accuracy and robustness of quality measures for machine learning models.
method Cross-validation approach to estimate prediction error and quantify explained variation. Confidence bounds and local quality measures derived from residuals.
result The reliability and robustness of quality measures are assessed through numerical examples and confidence bounds.
New method explains computational barriers in high-dimensional statistical models.
problem Understanding detection-recovery gaps in high-dimensional inference.
method Combining algorithmic contiguity and cross-validation reduction to obtain conditional computational lower bounds.
result Mild control of low-degree advantage is sufficient to explain computational barriers for recovery.
Improved LOO cross-validation for function approximation.
problem Estimating the Integrated Squared Error (ISE) for function approximation.
method Weighted Leave-One-Out cross-validation based on Gaussian Process.
result Significantly more precise ISE estimation compared to unweighted LOO.
Efficiently estimates shrinkage coefficient for RTME using LOOCV approximation.
problem Estimating optimal shrinkage coefficient for Regularized Tyler's M-estimator.
method Proposes an approximate LOOCV method to estimate efficiently.
result Significant speedup and accuracy improvement over existing methods.
A method for efficient CV estimates in Bayesian hierarchical models.
problem Computational infeasibility of cross-validation in Bayesian hierarchical regression models.
method Conditioning on variance-covariance parameters to transform CV into an optimization problem.
result Equivalent or improved predictive estimates compared to full cross-validation.
We show how to adjust the coefficient of determination () when used for measuring predictive accuracy via leave-one-out cross-validation.
New method for cross-validation in high-dimensional data with dependent or heavy-tailed covariates.
problem Inconsistent cross-validation in high-dimensional settings with dependent or heavy-tailed covariates.
method ROTI-GCV framework for cross-validation under proportional asymptotics regime.
result Demonstrated accuracy of ROTI-GCV in synthetic and semi-synthetic settings.
Decoding, ie prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didacti…
Optimizes hyperparameter tuning for models using approximate leave-one-out cross-validation.
problem Finding optimal hyperparameters for regularized models using approximate leave-one-out cross-validation.
method Derive efficient formulas for gradient and hessian of approximate leave-one-out cross-validation, apply second-order optimization.
result Demonstrates the effectiveness of the approach on real-world data sets.
Study improves predictive performance testing for high-dimensional data using exhaustive nested cross-validation.
problem Reproducibility issues in -fold cross-validation for high-dimensional data.
method Proposes a novel predictive performance test based on exhaustive nested cross-validation, addressing computational complexity with a closed-form expression.
result Demonstrates the effectiveness of Ridge-based methods in high-dimensional predictive performance testing.
In Bayesian statistics, the marginal likelihood, also known as the evidence, is used to evaluate model fit as it quantifies the joint probability of the data under the prior. In contrast, non-Bayesian models are typically compared using cross-validation on held-out data, either through -fold partitioning or leave-$p…
Improved GP models for fast training and good performance.
problem Training scalable Gaussian process models efficiently.
method Cross-validation and nearest neighbor truncation for scalable GP training.
result Our method offers fast training and excellent predictive performance.
We investigate the accuracy of the two most common estimators for the maximum expected value of a general set of random variables: a generalization of the maximum sample average, and cross validation. No unbiased estimator exists and we show that it is non-trivial to select a good estimator without knowledge about the …
Develops a prediction method based on sampling design.
problem Creating accurate individual predictions.
method Design-based approach using expected cross-validation results.
result Valid inference of unobserved prediction errors defined with respect to sampling design.
Analysis of cross-validation for early-stopped gradient descent in high-dimensional regression.
problem Inconsistency of GCV for early-stopped GD in high-dimensional least squares regression.
method Theoretical analysis of GCV and LOOCV applied to early-stopped GD in high-dimensional least squares regression.
result LOOCV converges uniformly to the prediction risk of early-stopped GD, while GCV is generically inconsistent.
Many learning tasks, such as cross-validation, parameter search, or leave-one-out analysis, involve multiple instances of similar problems, each instance sharing a large part of learning data with the others. We introduce a robust framework for solving multiple square-root LASSO problems, based on a sketch of the learn…
This paper improves model selection with cross-validation using domain knowledge.
problem Improving model selection with cross-validation risk estimation.
method Establishes distribution-free deviation bounds using VC dimension, formalizes Learning Spaces based on domain knowledge.
result Enhanced generalization through selection of candidate models based on domain knowledge.