Holdout set improves risk score accuracy without biasing predictions.
problem Updating risk scores can lead to biased estimates when directly applied.
method Use a holdout set of non-intervention population to update risk scores.
result Optimal holdout size reduces adverse outcomes to optimal level.
Methodology creates holdout and test/train sets for ML studies, preserving data for future research.
problem Preserving data for future research studies that are analysis-naive.
method Modification of k-fold cross-validation, randomization, and three-way split (holdout, test, training).
result Efficiently creates holdout and test/train sets without forcing.
In statistical learning, a dataset is often partitioned into two parts: the training set and the holdout (i.e., testing) set. For instance, the training set is used to learn a predictor, and then the holdout set is used for estimating the accuracy of the predictor on the true distribution. However, often in practice, t…
AI agents improve forecast combination but require transparency.
problem AI coding agents increase flexibility in empirical economics, leading to hidden degrees of freedom.
method Adapted open-source agent-loop architecture to empirical economics workflow, adding post-search holdout evaluation.
result Multiple agent runs outperform standard benchmarks in rolling evaluation but not all on post-search holdout.
Paper analyzes holdout cross-validation for large non-Gaussian covariance estimation.
problem Estimating large covariance matrices for non-Gaussian data.
method Use of Weingarten calculus and Ledoit-Péché formula for theoretical error derivation.
result Optimal train-test split ratio is proportional to square root of matrix dimension.
We compare and contrast two approaches to validating a trained classifier while using all in-sample data for training. One is simultaneous validation over an organized set of hypotheses (SVOOSH), the well-known method that began with VC theory. The other is withhold and gap (WAG). WAG withholds a validation set, trains…
AI agents improve forecast combination in empirical economics.
problem Hidden researcher degrees of freedom in AI-generated code.
method Adapted agent-loop architecture to empirical economics, added holdout evaluation.
result Independent agent searches find better forecast methods than benchmarks.
The study introduces a holdout-based framework to assess synthetic data fidelity and privacy.
problem Evaluating the quality and privacy of synthetic data solutions for mixed-type tabular data.
method Holdout-based empirical assessment framework measuring fidelity and privacy risk.
result Synthetic data samples are as close to the training as to the holdout data, indicating generalization and independence from individual records.
The study compares parametric and nonparametric models for estimating mean-variance mixtures and finds that nonparametric models perform better.
problem Estimating the distribution of a normal mean-variance mixture under uncertainty.
method Comparison of six parametric mixing laws with a grid nonparametric maximum likelihood estimator, using a paired block bootstrap for score comparison.
result Nonparametric models outperform parametric models in estimating the distribution of a normal mean-variance mixture.
Memory is an important aspect of intelligence and plays a role in many deep reinforcement learning models. However, little progress has been made in understanding when specific memory systems help more than others and how well they generalize. The field also has yet to see a prevalent consistent and rigorous approach f…
Excessive reuse of holdout data can lead to overfitting. However, there is little concrete evidence of significant overfitting due to holdout reuse in popular multiclass benchmarks today. Known results show that, in the worst-case, revealing the accuracy of k adaptively chosen classifiers on a data set of size n al…
Optimal data splitting improves covariance matrix estimation in large datasets.
problem Improving large covariance matrix estimation in high-dimensional settings.
method Focus on holdout method, derive closed-form error expression, connect to eigenvalue variance.
result Optimal train-test split scales as square root of matrix dimension.
Paper examines fairness of data augmentation methods, finding vanilla Mixup outperforms Fair Mixup.
problem Improving fairness in classification models with limited minority data.
method Uses multicalibration to rigorously evaluate and improve data augmentation methods for classification fairness.
result Vanilla Mixup outperforms Fair Mixup and baseline methods in fairness and accuracy, especially with small minority groups.
New method uses unlabeled data to improve generalization bounds for deep learning.
problem Vacuous guarantees and shrinking holdout sets for overparameterized models.
method Augmenting labeled training set with unlabeled data and training as usual.
result Proves tight upper bounds on true risk for 0-1 empirical risk minimization.
Cross-validation pitfalls in change-point regression are addressed with new approaches.
problem Cross-validation's prediction error-based criterion may lead to under- or over-estimation of change-points.
method Proposes two approaches: absolute error loss and modified holdout sets.
result Consistent estimation of the number of change-points under certain conditions.
Drug-drug interactions are preventable causes of medical injuries and often result in doctor and emergency room visits. Computational techniques can be used to predict potential drug-drug interactions. We approach the drug-drug interaction prediction problem as a link prediction problem and present two novel methods fo…
Jointly tuning ensemble models improves performance and uncertainty calibration.
problem Improving both predictive performance and uncertainty calibration in deep ensembles.
method Investigated the impact of jointly tuning weight decay, temperature scaling, and early stopping.
result Jointly tuning ensemble models generally matches or improves performance, with significant variation across tasks.
Reshuffling splits improves hyperparameter optimization's generalization performance.
problem Improving peak performance of machine learning models through better hyperparameter optimization.
method Reshuffling splits for every hyperparameter configuration improves generalization performance.
result Reshuffling leads to better generalization performance compared to fixed splits.
Recent studies have shown that tuning prediction models increases prediction accuracy and that Random Forest can be used to construct prediction intervals. However, to our best knowledge, no study has investigated the need to, and the manner in which one can, tune Random Forest for optimizing prediction intervals { thi…
A framework for private prediction sets using conformal prediction and differential privacy.
problem Jointly addressing reliability and privacy in machine learning predictions.
method Split conformal prediction with privatized quantile subroutine.
result Private prediction sets can be generated from privately-trained models.
Study evaluates how well question-answering models generalize to new data types.
problem Generalization of question-answering models to new data types.
method Constructed new test sets from different domains and evaluated models' performance.
result Models show significant performance drops when tested on new data types.
The paper offers a method to create prediction sets with uncertainty control.
problem Calibrating and communicating uncertainty in machine learning predictions.
method Distribution-free, risk-controlling prediction sets using a holdout set to calibrate set sizes.
result Explicit finite-sample guarantees for error control in various machine learning tasks.
Medical research is risky and expensive. Drug discovery, as an example, requires that researchers efficiently winnow thousands of potential targets to a small candidate set for more thorough evaluation. However, research groups spend significant time and money to perform the experiments necessary to determine this cand…
We present a new modeling technique for solving the problem of ecological inference, in which individual-level associations are inferred from labeled data available only at the aggregate level. We model aggregate count data as arising from the Poisson binomial, the distribution of the sum of independent but not identic…
PAC-Bayesian method improves deep learning generalization in medical imaging.
problem Overfitting in deep networks for medical imaging datasets.
method PAC-Bayesian framework applied to large (stochastic) networks.
result PAC-Bayesian bounds are competitive and more explainable than simpler methods.
Analyzes error sources in global feature effect estimation methods.
problem Unexplored error sources in global feature effect estimation methods.
method Systematic, estimator-level analysis of bias and variance.
result Holdout data is theoretically cleanest, but estimation variance depends on sample size and model characteristics.
Replicated and validated Rank-N-Contrast for robust regression.
problem Deep regression models struggle with continuous sample orders.
method Contrastive learning of continuous representations by ranking samples.
result Improved performance and robustness of RNC framework.
Deep neural networks predict prostate motion from MR images.
problem Predicting prostate motion during ultrasound-guided interventions.
method Biomechanically-trained deep neural networks on unstructured nodes.
result Trained networks yield near real-time inference with 0.017 mm error.
In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex lo…
Study evaluates neural networks for corporate credit rating assessment.
problem Improving machine learning algorithms for credit assessment.
method Analysis of four neural network architectures (MLP, CNN, CNN2D, LSTM) on financial data from energy, financial, and healthcare sectors.
result LSTM architecture consistently outperforms others in predicting corporate credit ratings.
Traditional statistical analysis requires that the analysis process and data are independent. By contrast, the new field of adaptive data analysis hopes to understand and provide algorithms and accuracy guarantees for research as it is commonly performed in practice, as an iterative process of interacting repeatedly wi…
A privacy-preserving synthetic data generation framework that distinguishes between true and phantom data disclosures.
problem Detecting and explaining data disclosures in synthetic datasets.
method Customizable empirical auditing framework with statistical hypothesis testing.
result Demonstrated tighter privacy leakage bounds than prior methods.
Tests for overfitting in machine learning models.
problem Overfitting in high complexity models.
method Hypothesis test using concentration bounds.
result Valid test for identifying overfitting.
New estimate reduces overfitting risk in machine learning models.
problem Error rate on test data may not reflect true population error due to adaptive data analysis practices.
method Introduces Rip van Winkle's Razor, a simple estimate of overfit to test data based on information content.
result Shows non-vacuous estimate of deviation in many modern settings.
Big spatio-temporal datasets, available through both open and administrative data sources, offer significant potential for social science research. The magnitude of the data allows for increased resolution and analysis at individual level. While there are recent advances in forecasting techniques for highly granular te…
ECGDetect uses deep learning to detect ischemia in heart ECGs.
problem Detecting early signs of acute coronary syndrome in patients.
method Developed a deep learning model using the LTST database.
result Deep neural network achieved 90.31% ROC-AUC, 89.34% sensitivity, 87.81% specificity.
The paper explores how neural networks learn logical functions and their generalization error.
problem Learning logical functions with neural networks and understanding generalization error.
method Gradient descent on neural networks, analyzing noise-stability and Boolean influence.
result Gradient descent on certain neural architectures tends to favor low-degree representations, impacting generalization error.
The correct use of model evaluation, model selection, and algorithm selection techniques is vital in academic machine learning research as well as in many industrial settings. This article reviews different techniques that can be used for each of these three subtasks and discusses the main advantages and disadvantages …
Datasets are often reused to perform multiple statistical analyses in an adaptive way, in which each analysis may depend on the outcomes of previous analyses on the same dataset. Standard statistical guarantees do not account for these dependencies and little is known about how to provably avoid overfitting and false d…
We build a deep reinforcement learning (RL) agent that can predict the likelihood of an individual testing positive for malaria by asking questions about their household. The RL agent learns to determine which survey question to ask next and when to stop to make a prediction about their likelihood of malaria based on t…
This paper improves multi-class calibration methods using mutual information maximization-based binning.
problem Calibration of deep neural network predictions, especially for small prior classes.
method I-Max concept for binning, shared class-wise calibration strategy.
result Improves multi-class ranking and calibration performance using a small calibration set.
A novel bootstrap method improves concept drift detection in predictive models.
problem Detecting changes in predictive relationships (concept drift) in data-driven applications.
method Developed a nested bootstrap procedure to calibrate control limits using the entire initial sample.
result The method yields more accurate baseline models and faster CL setup times.
Python package for ordinal regression using gradient boosting.
problem Handling ordinal variables in machine learning.
method Gradient boosting with latent variable framework.
result Performs joint optimization of latent function and threshold vector.
This paper optimizes perpetual contract liquidity by accounting for funding rates.
problem Optimal liquidity provision for perpetual contracts with stochastic funding rates.
method Formulated a control problem, solved with a HJB scheme, and calibrated on real data.
result Funding-aware market making improves performance and reduces inventory risk.
We propose the Sobolev Independence Criterion (SIC), an interpretable dependency measure between a high dimensional random variable X and a response variable Y . SIC decomposes to the sum of feature importance scores and hence can be used for nonlinear feature selection. SIC can be seen as a gradient regularized Integr…
Malaria is a female anopheles mosquito-bite inflicted life-threatening disease which is considered endemic in many parts of the world. This article focuses on improving malaria detection from patches segmented from microscopic images of red blood cell smears by introducing a deep convolutional neural network. Compared …
The paper tackles sampling bias in credit scoring models and proposes methods to improve their training and evaluation.
problem Sampling bias in credit scoring models leads to an incomplete representation of the borrower population.
method Bias-aware self-learning framework and Bayesian evaluation method to correct for bias.
result Bayesian evaluation outperforms standard accuracy measures in predicting future performance.
Paper evaluates synthetic retail data for fidelity, utility, and privacy.
problem Ensuring accurate synthetic data in retail.
method Differentiates between continuous and discrete data, measures fidelity and utility, and uses Differential Privacy for privacy.
result Validated framework for reliable and scalable synthetic data evaluation.