In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for empirical risk minimizers. In the general setting, we prove sanity-check bounds in the spirit of \cite{KR99} \textquotedblleft\textit{bounds showing that the worst-case error of this estimate is not m…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “validation procedures”
Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its apparent universality. Many results exist on the model selection performances of cross-validation procedures. This survey intends to relate these results to the most recen…
Nested cross-validation when selecting classifiers is overzealous for most practical applicationscs.LG
When selecting a classification algorithm to be applied to a particular problem, one has to simultaneously select the best algorithm for that dataset \emph{and} the best set of hyperparameters for the chosen model. The usual approach is to apply a nested cross-validation procedure; hyperparameter selection is performed…
Statistical machine learning models should be evaluated and validated before putting to work. Conventional k-fold Monte Carlo Cross-Validation (MCCV) procedure uses a pseudo-random sequence to partition instances into k subsets, which usually causes subsampling bias, inflates generalization errors and jeopardizes the r…
New research investigates why influence functions are fragile and proposes new validation procedures.
problem Understanding and mitigating the fragility of influence functions in deep learning model explanations.
method Verification of influence functions using various conditions and procedures, including convexity and non-convexity.
result Validation procedures may cause the observed fragility of influence functions.
Pre-validation Revisitedstat.ME
Paper revisits pre-validation method, improving hypothesis testing.
problem Improving hypothesis testing in pre-validated models with different feature dimensions.
method Extended problem formulation, analytical distribution, and bootstrap procedure.
result Proposed analytical distribution and bootstrap procedure for pre-validated predictors.
We apply multiple testing procedures to the validation of estimated default probabilities in credit rating systems. The goal is to identify rating classes for which the probability of default is estimated inaccurately, while still maintaining a predefined level of committing type I errors as measured by the familywise …
With the increasing size of today's data sets, finding the right parameter configuration in model selection via cross-validation can be an extremely time-consuming task. In this paper we propose an improved cross-validation procedure which uses nonparametric testing coupled with sequential analysis to determine the bes…
This paper identifies a problem with the usual procedure for L2-regularization parameter estimation in a domain adaptation setting. In such a setting, there are differences between the distributions generating the training data (source domain) and the test data (target domain). The usual cross-validation procedure requ…
Simulation workflow is a top-level model for the design and control of simulation process. It connects multiple simulation components with time and interaction restrictions to form a complete simulation system. Before the construction and evaluation of the component models, the validation of upper-layer simulation work…
Paper extends prediction-powered inference using conformal prediction for robust and valid imputation.
problem Safe use of black-box ML models for imputing missing data with strong guarantees.
method Connecting prediction-powered inference with conformal prediction for valid and additional guarantees.
result First general prediction-powered procedure for e-values operating off-line.
Framework mitigates risk non-monotonicity in high-dimensional predictions.
problem Risk non-monotonicity in high-dimensional predictions.
method Model-agnostic framework using cross-validation and data-driven methodologies (zero- and one-step).
result Modified prediction procedures achieve monotonic asymptotic risk behavior.
In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for stable predictors in the context of risk assessment. The notion of stability has been first introduced by \cite{DEWA79} and extended by \cite{KEA95}, \cite{BE01} and \cite{KUNIY02} to characterize cla…
Paper introduces a novel approach to generalize models without validation data.
problem Achieving generalization in machine learning without validation data.
method Introduces a new risk measure and a novel criterion for training models.
result Demonstrates that the new approach outperforms popular methods in generalization and estimation tasks.
Develop conformal prediction for dyadic regression under complex missingness.
problem Conformal prediction for dyadic regression under complex missingness mechanisms.
method Developing general technical tools and conformal prediction procedures for dyadic regression under complex missingness.
result Establishing asymptotic validity of weighted conformal prediction under a nonparametric graphon model for missingness mechanism.
Transforms any test into anytime-valid with sample savings.
problem Sequential data invalidates classical test guarantees.
method Predicts test outcomes to create anytime-valid stopping rules.
result Ensures Type-I error control and near-optimal power.
Improves test set performance and reduces out-of-sample disappointment for unstable models.
problem Ensuring strong test set performance via cross-validation for unstable models.
method Nested k-fold cross-validation with hyperparameter selection based on a weighted sum of cross-validation metric and model stability measure.
result Improves out-of-sample MSE for sparse ridge regression and CART by 4% and 2% respectively, compared to k-fold cross-validation.
In the regression setting, given a set of hyper-parameters, a model-estimation procedure constructs a model from training data. The optimal hyper-parameters that minimize generalization error of the model are usually unknown. In practice they are often estimated using split-sample validation. Up to now, there is an ope…
Extends PPI to sequential setting, improving inference over time.
problem Sequential data growth with unlabelled data.
method Prediction-powered confidence sequence procedures using Ville's inequality and the method of mixtures.
result Asymptotically valid uniformly over time, accommodating prior knowledge.
We provide a rigorous numerical computation method to validate periodic, homoclinic and heteroclinic orbits as the continuation of singular limit orbits for the fast-slow system with one-dimensional slow variable . Our validation procedure is based on topological tools called isolatin…
Stacked conformal predictionstat.ML
Stacked conformal prediction simplifies model validation.
problem Validating stacked predictive models efficiently.
method Meta-learner at the top of a stacked ensemble for approximate marginal validity.
result The method achieves approximate marginal validity without a separate calibration sample.
New method for estimating and testing impulse responses in high-dimensional VAR systems.
problem Statistical inference for impulse responses in sparse, high-dimensional vector autoregressions.
method Local projection equations and de-sparsified estimators combined with a non-regularized contemporaneous impact matrix.
result Valid inference procedures for structural impulse responses in high-dimensional systems.
Chemical plants are complex and dynamical systems consisting of many components for manipulation and sensing, whose state transitions depend on various factors such as time, disturbance, and operation procedures. For the purpose of supporting human operators of chemical plants, we are developing an AI system that can s…
Feature selection aims to select the smallest subset of features for a specified level of performance. The optimal achievable classification performance on a feature subset is summarized by its Receiver Operating Curve (ROC). When infinite data is available, the Neyman- Pearson (NP) design procedure provides the most e…
Multilayer networks are attracting growing attention in many fields, including finance. In this paper, we develop a new tractable procedure for multilayer aggregation based on statistical validation, which we apply to investor networks. Moreover, we propose two other improvements to their analysis: transaction bootstra…
Paper develops methods for statistical inference with SGD in nonconvex optimization.
problem Statistical inference for nonconvex optimization problems.
method Proposes two online inferential procedures combining SGD and bootstrap techniques.
result Establishes error convergence rates and asymptotically valid bootstrap confidence intervals.
New framework validates counterfactual estimations in network interference settings.
problem Challenges in causal effect estimation and validation in network interference settings.
method Introduces a distribution-preserving network bootstrap and counterfactual cross-validation procedure.
result Validates counterfactual estimations in diverse network interference settings.
In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for subagged estimators, both for classification and regressor. General loss functions and class of predictors with both finite and infinite VC-dimension are considered. We slightly generalize the formali…
Cross-validationmath.ST
This text is a survey on cross-validation. We define all classical cross-validation procedures, and we study their properties for two different goals: estimating the risk of a given estimator, and selecting the best estimator among a given family. For the risk estimation problem, we compute the bias (which can also be …
Unified framework for FDR control in knockoffs, validating Gaussian knockoffs.
problem Asymptotic FDR control in knockoffs with user-specified distributions.
method Unified theoretical framework, three conditions on approximate knockoff statistics, Gaussian knockoffs generator based on moments matching.
result Gaussian knockoffs generator achieves asymptotic FDR control.
Kernel ridge regression inference for nonstandard data.
problem Inferential theory for kernel ridge regression with nonstandard data.
method Constructs valid and sharp confidence sets using anti-symmetric multipliers.
result Develops a test for match effects in school matching mechanisms.
Validity problems in clinical machine learning by indirect data labeling using consensus definitionscs.LG
Machine learning models trained on indirect data labels can fail on real-world examples.
problem Validity issues in machine learning when target labels are indirectly defined.
method Identification of problematic datasets and models using a general procedure.
result Machine learning models trained on indirect data labels will fail on real-world examples.
Paper proposes a new dynamic pricing method with always-valid online statistical learning.
problem Designing dynamic pricing policies that adapt to online uncertainty and maintain validity.
method Regularized online statistical learning with theoretical guarantees and three major advantages.
result Proposed OORMLP pricing policy secures logarithmic regret in decision horizon.
Enhanced conformal methods improve validity of LLM outputs.
problem Lack of conditional validity and high false rejection rates in LLM validity guarantees.
method Adaptive conditional conformal procedure and improved scoring function differentiation.
result Demonstrated improved validity and utility on real datasets.
Method selects valid IVs from a large set using clustering and test of overidentifying restrictions.
problem Selecting valid instrumental variables from a large set of candidates.
method Agglomerative hierarchical clustering combined with a test of overidentifying restrictions.
result Achieves oracle properties when the largest group of IVs is valid.
Prediction-Powered E-Valuesstat.ML
Predictive e-values enhance statistical inference across various tasks.
problem Insufficient data limits traditional statistical inference.
method Apply prediction-powered inference to e-values.
result Every e-value-based inference has a prediction-powered counterpart.
We describe a simple, efficient, permutation based procedure for selecting the penalty parameter in the LASSO. The procedure, which is intended for applications where variable selection is the primary focus, can be applied in a variety of structural settings, including generalized linear models. We briefly discuss conn…
Recent advances in statistical theory, together with advances in the computational power of computers, provide alternative methods to do mass-univariate hypothesis testing in which a large number of univariate tests, can be properly used to compare MEEG data at a large number of time-frequency points and scalp location…
We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have differ…
Paper proposes a novel method to assess treatment effect estimators using cross-validation.
problem Lack of ground truth to objectively assess treatment effect estimators in RCTs.
method Cross-validation-like methodology combining noisy difference-of-means estimate and aggregation across RCTs.
result Aggressive downweighting or truncation of large values reduces variance and improves treatment effect estimation.
A new cross-validation method reduces redundancy and improves model performance.
problem Redundancy in traditional k-fold cross-validation leads to biased results.
method Irredundant k-fold cross-validation, where each instance is used exactly once for training and testing.
result Consistent performance estimates with reduced variance and lower computational cost.
The study assesses external validity by evaluating worst-case treatment effects across subpopulations.
problem Underrepresentation of marginalized groups and limited study populations.
method Develops a semiparametrically efficient estimator for worst-case treatment effects (WTE) and uses cross-fitting to guard against brittle findings.
result The proposed framework guards against invalid findings due to unanticipated population shifts.
Proposes a method to estimate treatment effects using instruments.
problem Estimating treatment effects from observational data is challenging when unconfoundedness is violated.
method Leverages instruments to estimate bounds on conditional average treatment effect (CATE) through a mapping to a discrete representation space and a two-step procedure.
result Demonstrates theoretical validity and reduced estimation variance in finite-sample settings.
Study proposes a statistical testing framework for evaluating clustering pipelines.
problem Quantifying the statistical reliability of clustering results from data analysis pipelines.
method Selective inference-based statistical testing framework for clustering pipelines.
result The proposed test controls the type I error rate and is effective in validating clustering results.
Efficient nonparametric statistical inference on population feature importance using Shapley valuesstat.ME
Efficiently estimates variable importance in prediction tasks using Shapley values.
problem Valid statistical inference on the importance of variables in prediction tasks.
method Randomly sampling feature subsets to estimate Shapley Population Variable Importance Measure (SPVIM) efficiently.
result The proposed estimator converges at an asymptotically optimal rate and can construct valid confidence intervals and hypothesis tests.
This paper tackles deep clustering evaluation challenges in high-dimensional data.
problem Evaluation of deep clustering methods is problematic due to the curse of dimensionality and variations in embedding spaces.
method Develops a theoretical framework to highlight the ineffectiveness of internal validation measures and proposes a systematic approach to applying clustering validity indices in deep learning.
result The proposed framework reduces misguidance from improper use of clustering validity indices in deep learning.
A procedure for unfolding the true distribution from experimental data is presented. Machine learning methods are applied for simultaneous identification of an apparatus function and solving of an inverse problem. A priori information about the true distribution from theory or previous experiments is used for Monte-Car…
As a consequence of the dependence experienced in loan portfolios, the standard binomial test which is based on the assumption of independence does not appear appropriate for validating probabilities of default (PDs). The model underlying the new rules for minimum capital requirements (Basle II) is taken as a point of …