Two validation methods compared for classifier accuracy.
problem Validating trained classifiers using all in-sample data.
method SVOOSH vs. WAG (withhold and gap)
result WAG can be a better alternative for complex hypothesis classes and limited training data.
Methodology creates holdout and test/train sets for ML studies, preserving data for future research.
problem Preserving data for future research studies that are analysis-naive.
method Modification of k-fold cross-validation, randomization, and three-way split (holdout, test, training).
result Efficiently creates holdout and test/train sets without forcing.
Holdout set improves risk score accuracy without biasing predictions.
problem Updating risk scores can lead to biased estimates when directly applied.
method Use a holdout set of non-intervention population to update risk scores.
result Optimal holdout size reduces adverse outcomes to optimal level.
Models predict drug interactions with high accuracy.
problem Detecting drug-drug interactions to prevent medical injuries.
method Artificial neural networks and graph similarity measures.
result Models achieve high accuracy in predicting drug interactions.
AI agents improve forecast combination but require transparency.
problem AI coding agents increase flexibility in empirical economics, leading to hidden degrees of freedom.
method Adapted open-source agent-loop architecture to empirical economics workflow, adding post-search holdout evaluation.
result Multiple agent runs outperform standard benchmarks in rolling evaluation but not all on post-search holdout.
New bounds show multiclass problems reduce overfitting from test set reuse.
problem Overfitting from test set reuse in multiclass problems.
method Upper and lower bounds on bias of attacks, practical and inefficient attacks.
result Multiclass problems mitigate overfitting from test set reuse.
Paper analyzes holdout cross-validation for large non-Gaussian covariance estimation.
problem Estimating large covariance matrices for non-Gaussian data.
method Use of Weingarten calculus and Ledoit-Péché formula for theoretical error derivation.
result Optimal train-test split ratio is proportional to square root of matrix dimension.
Adaptive statistical learning improves model generalization with privacy-preserving techniques.
problem Overfitting and poor generalization in adaptive statistical learning.
method Adapting Bayesian differential privacy techniques to handle correlated data.
result The holdout dataset can be reused adaptively in statistical learning with privacy-preserving perturbations.
AI agents improve forecast combination in empirical economics.
problem Hidden researcher degrees of freedom in AI-generated code.
method Adapted agent-loop architecture to empirical economics, added holdout evaluation.
result Independent agent searches find better forecast methods than benchmarks.
The study introduces a holdout-based framework to assess synthetic data fidelity and privacy.
problem Evaluating the quality and privacy of synthetic data solutions for mixed-type tabular data.
method Holdout-based empirical assessment framework measuring fidelity and privacy risk.
result Synthetic data samples are as close to the training as to the holdout data, indicating generalization and independence from individual records.
New method uses unlabeled data to improve generalization bounds for deep learning.
problem Vacuous guarantees and shrinking holdout sets for overparameterized models.
method Augmenting labeled training set with unlabeled data and training as usual.
result Proves tight upper bounds on true risk for 0-1 empirical risk minimization.
The paper aims to develop a methodology to test memory systems in reinforcement learning agents.
problem Understanding and evaluating memory systems in reinforcement learning agents and their generalization.
method Construct diverse memory tasks, develop an agent architecture combining multiple memory systems, and perform ablations.
result The agent architecture can apply learned knowledge to holdout data that differs from the training data.
Optimal data splitting improves covariance matrix estimation in large datasets.
problem Improving large covariance matrix estimation in high-dimensional settings.
method Focus on holdout method, derive closed-form error expression, connect to eigenvalue variance.
result Optimal train-test split scales as square root of matrix dimension.
The study compares parametric and nonparametric models for estimating mean-variance mixtures and finds that nonparametric models perform better.
problem Estimating the distribution of a normal mean-variance mixture under uncertainty.
method Comparison of six parametric mixing laws with a grid nonparametric maximum likelihood estimator, using a paired block bootstrap for score comparison.
result Nonparametric models outperform parametric models in estimating the distribution of a normal mean-variance mixture.
Jointly tuning ensemble models improves performance and uncertainty calibration.
problem Improving both predictive performance and uncertainty calibration in deep ensembles.
method Investigated the impact of jointly tuning weight decay, temperature scaling, and early stopping.
result Jointly tuning ensemble models generally matches or improves performance, with significant variation across tasks.
Reshuffling splits improves hyperparameter optimization's generalization performance.
problem Improving peak performance of machine learning models through better hyperparameter optimization.
method Reshuffling splits for every hyperparameter configuration improves generalization performance.
result Reshuffling leads to better generalization performance compared to fixed splits.
Paper examines fairness of data augmentation methods, finding vanilla Mixup outperforms Fair Mixup.
problem Improving fairness in classification models with limited minority data.
method Uses multicalibration to rigorously evaluate and improve data augmentation methods for classification fairness.
result Vanilla Mixup outperforms Fair Mixup and baseline methods in fairness and accuracy, especially with small minority groups.
This paper optimizes prediction intervals by tuning Random Forest using meta-validation.
problem Optimizing prediction intervals constructed by Random Forest.
method Combines exhaustive search with meta-validation techniques to tune Random Forest parameters.
result The 75/25 holdout meta-validation technique is always beneficial for tuning Random Forest.
Cross-validation pitfalls in change-point regression are addressed with new approaches.
problem Cross-validation's prediction error-based criterion may lead to under- or over-estimation of change-points.
method Proposes two approaches: absolute error loss and modified holdout sets.
result Consistent estimation of the number of change-points under certain conditions.
Analyzes error sources in global feature effect estimation methods.
problem Unexplored error sources in global feature effect estimation methods.
method Systematic, estimator-level analysis of bias and variance.
result Holdout data is theoretically cleanest, but estimation variance depends on sample size and model characteristics.
A new method for feature selection in high-dimensional data reduces search cost while maintaining performance.
problem Feature selection in very-high-dimensional datasets is computationally expensive and challenging.
method Stochastic Sequential Search (SSS) using temperature-controlled softmax sampling and dependency-aware statistics.
result The method significantly reduces search cost while maintaining or improving performance.
Python package for ordinal regression using gradient boosting.
problem Handling ordinal variables in machine learning.
method Gradient boosting with latent variable framework.
result Performs joint optimization of latent function and threshold vector.
A new regularizer boosts long-range dependency in sequence data.
problem Improving long-range dependency in sequence data models.
method Developed a mutual information regularizer to enhance sequence learning.
result The approach increases mutual information and likelihood on holdout data.
Replicated and validated Rank-N-Contrast for robust regression.
problem Deep regression models struggle with continuous sample orders.
method Contrastive learning of continuous representations by ranking samples.
result Improved performance and robustness of RNC framework.
New model reveals voter preferences from aggregate election data.
problem Infer individual-level voter preferences from aggregate election data.
method Modeling aggregate count data as Poisson binomial, relating probabilities to covariates using logistic and neural networks.
result Model predicts voter preferences at precinct and individual levels.
PAC-Bayesian method improves deep learning generalization in medical imaging.
problem Overfitting in deep networks for medical imaging datasets.
method PAC-Bayesian framework applied to large (stochastic) networks.
result PAC-Bayesian bounds are competitive and more explainable than simpler methods.
This paper optimizes perpetual contract liquidity by accounting for funding rates.
problem Optimal liquidity provision for perpetual contracts with stochastic funding rates.
method Formulated a control problem, solved with a HJB scheme, and calibrated on real data.
result Funding-aware market making improves performance and reduces inventory risk.
Study evaluates how well question-answering models generalize to new data types.
problem Generalization of question-answering models to new data types.
method Constructed new test sets from different domains and evaluated models' performance.
result Models show significant performance drops when tested on new data types.
A framework for private prediction sets using conformal prediction and differential privacy.
problem Jointly addressing reliability and privacy in machine learning predictions.
method Split conformal prediction with privatized quantile subroutine.
result Private prediction sets can be generated from privately-trained models.
Study evaluates neural networks for corporate credit rating assessment.
problem Improving machine learning algorithms for credit assessment.
method Analysis of four neural network architectures (MLP, CNN, CNN2D, LSTM) on financial data from energy, financial, and healthcare sectors.
result LSTM architecture consistently outperforms others in predicting corporate credit ratings.
Paper evaluates synthetic retail data for fidelity, utility, and privacy.
problem Ensuring accurate synthetic data in retail.
method Differentiates between continuous and discrete data, measures fidelity and utility, and uses Differential Privacy for privacy.
result Validated framework for reliable and scalable synthetic data evaluation.
Reducing volatility proxy improves apparent market correlation dynamics.
problem Attributing apparent slow collective market dynamics to intrinsic or driver inheritance.
method Coupled Ornstein-Uhlenbeck model with VIX proxy, decomposing and controlling for autocorrelation.
result VIX-coupled model reduces effective relaxation time from 298 to 61 trading days, improving fit over bare mean reversion.
Novel framework uses synthetic data to quantify uncertainty in complex data.
problem Uncertainty quantification in complex, unstructured data.
method Perturbation-Assisted Sample Synthesis (PASS) and Perturbation-Assisted Inference (PAI) framework.
result Statistically guaranteed validity in inference, enhancing reliability of synthetic data.
Deep neural networks predict prostate motion from MR images.
problem Predicting prostate motion during ultrasound-guided interventions.
method Biomechanically-trained deep neural networks on unstructured nodes.
result Trained networks yield near real-time inference with 0.017 mm error.
Interpretable additive models outperform complex DL and hybrid pipelines for air quality forecasting.
problem Accurate forecasting of urban air pollution for public health and policy guidance.
method Investigated lightweight additive models (FBP, NP) vs. deep learning and hybrid pipelines on Beijing PM2.5 and PM10 data.
result Facebook Prophet consistently outperformed NeuralProphet and traditional models, achieving high R2 values. The paper offers a method to create prediction sets with uncertainty control.
problem Calibrating and communicating uncertainty in machine learning predictions.
method Distribution-free, risk-controlling prediction sets using a holdout set to calibrate set sizes.
result Explicit finite-sample guarantees for error control in various machine learning tasks.
Tests for overfitting in machine learning models.
problem Overfitting in high complexity models.
method Hypothesis test using concentration bounds.
result Valid test for identifying overfitting.
SIC measures dependency between variables, promoting feature selection.
problem Measuring and selecting features in high-dimensional data.
method Gradient regularized Integral Probability Metric (IPM) with sparsity inducing penalties.
result SIC can be used for reliable and interpretable feature selection.
New framework learns complex AI attitudes from heterogeneous data.
problem Heterogeneous ordinal structure in AI attitudes, poorly captured by existing methods.
method Monotone Gaussian score embedding, BNP complexity discovery, confirmatory fixed-K estimation.
result Reduced holdout MSE by 25.8% over single-graph baseline.
AGATHA predicts promising research directions from scientific data.
problem Efficiently winnowing thousands of potential targets in drug discovery.
method Deep-learning hypothesis generation system that ranks plausible term-pairs.
result Best-in-class performance on an established benchmark, high recommendation scores across subdomains.
Deep neural network improves malaria detection from red blood cells.
problem Improving malaria detection from red blood cell images.
method End-to-end deep learning approach using convolutional neural networks.
result Best model achieves 97.77% accuracy.
The paper explores how neural networks learn logical functions and their generalization error.
problem Learning logical functions with neural networks and understanding generalization error.
method Gradient descent on neural networks, analyzing noise-stability and Boolean influence.
result Gradient descent on certain neural architectures tends to favor low-degree representations, impacting generalization error.
Study shows annotation instrument design affects model performance in hate speech detection.
problem Impact of annotation instrument design on model performance in hate speech detection.
method Collected annotations from five experimental conditions of an annotation instrument, fine-tuned BERT models on each dataset, evaluated performance on holdout portion.
result Significant differences in model performance and annotations across conditions.
Identity-link IRT improves TVD-MI scores without curvature violations.
problem Preserving additivity in TVD-MI scores for efficient LLM evaluation.
method Derives clipped-linear model from Gini entropy maximization, using identity link.
result Identity-link yields lower curvature violations (median curl 0.080-0.150) compared to probit/logit.
New algorithms for model selection in off-policy evaluation of reinforcement learning.
problem Hyperparameter tuning for off-policy evaluation methods in reinforcement learning.
method Developed new model-free and model-based selectors with theoretical guarantees and a new experimental protocol.
result New model-free selector, LSTD-Tournament, demonstrates promising empirical performance.
Enhances model OOD detection by diversifying feature representation.
problem Limited diversity in model ensembles for OOD detection.
method Integrates diverse training tasks into model ensembles to generate distinct feature representations.
result Superior performance of Multi-Comprehension Ensemble in OOD detection.
A privacy-preserving synthetic data generation framework that distinguishes between true and phantom data disclosures.
problem Detecting and explaining data disclosures in synthetic datasets.
method Customizable empirical auditing framework with statistical hypothesis testing.
result Demonstrated tighter privacy leakage bounds than prior methods.
The paper proposes a method to estimate individual behavioral profiles using smart meter data.
problem Estimating individual behavioral profiles with granular temporal data.
method Gaussian Process-based models for segmenting and clustering time series data.
result The method can predict individual behavioral patterns with high accuracy.