The network jackknife provides conservative variance estimates for network statistics.
problem Estimating the variance of network statistics.
method Leave-node-out jackknife procedure for network data under the sparse graphon model.
result The network jackknife leads to conservative estimates of the variance for network functionals invariant to node permutation.
We study the variability of predictions made by bagged learners and random forests, and show how to estimate standard errors for these methods. Our work builds on variance estimates for bagging proposed by Efron (1992, 2012) that are based on the jackknife and the infinitesimal jackknife (IJ). In practice, bagged predi…
Extends Infinitesimal Jackknife for model covariance, enhancing ensemble model analysis.
problem Estimating uncertainty in combinations of models.
method Extends Infinitesimal Jackknife to estimate covariance between models.
result Theoretical consistency of Infinitesimal Jackknife covariance estimate demonstrated.
Jackknife variance estimation validated for generalized U-statistics.
problem Uncertainty quantification for subsampling-based estimators.
method Jackknife variance estimation for generalized U-statistics with row-wise Lr weak law. result Jackknife and delete-d variance estimators are ratio-consistent for generalized U-statistics. Cluster jackknife improves inference for staggered DID methods.
problem Over-rejection of CSDID in small clusters or treated clusters.
method Cluster jackknife for CSDID inference.
result Cluster jackknife greatly improves inference for CSDID.
Discriminative jackknife estimates deep learning uncertainty.
problem Quantifying uncertainty in deep learning models.
method Discriminative jackknife using influence functions of loss.
result DJ satisfies frequentist coverage and discriminative accuracy.
This article proposes a generalisation of the delete-d jackknife to solve hyperparameter selection problems for time series. I call it artificial delete-d jackknife to stress that this approach substitutes the classic removal step with a fictitious deletion, wherein observed datapoints are replaced with artificial …
Modified jackknife method improves predictive inference for time series data.
problem Lack of exchangeability and temporal dependence in time series data.
method Leave-a-window-out (LWO) method modification of the jackknife.
result LWO method achieves valid coverage in time series models with mild temporal dependence.
Enhances polynomial chaos models with uncertainty intervals.
problem Uncertainty quantification in surrogate models.
method Jackknife-based conformal prediction integrated into polynomial chaos expansions.
result Produces accurate predictive intervals for low-accuracy models.
Cross validation (CV) and the bootstrap are ubiquitous model-agnostic tools for assessing the error or variability of machine learning and statistical estimators. However, these methods require repeatedly re-fitting the model with different weighted versions of the original dataset, which can be prohibitively time-cons…
We provide additional statistical background for the methodology developed in the clinical analysis of knee osteoarthritis in "A Precision Medicine Approach to Develop and Internally Validate Optimal Exercise and Weight Loss Treatments for Overweight and Obese Adults with Knee Osteoarthritis" (Jiang et al. 2020). Jiang…
Adaptive method improves prediction intervals with global coverage guarantees and local error distribution.
problem Global coverage guarantees of conformal regression are often violated by local error distributions.
method Adaptive Conformal Regression with Jackknife+ Rescaled Scores
result Improves local coverage without sacrificing global coverage, especially in low-data regimes.
The paper proposes a method to improve fairness in machine learning models without refitting.
problem Mitigating biases in machine learning models that disadvantage certain groups.
method Infinitesimal jackknife-based approach to drop selected training data points.
result The intervention improves fairness without significantly reducing predictive performance.
Frequentist method estimates uncertainty in RNNs without altering architecture.
problem Uncertainty quantification in RNNs for decision-making.
method Jackknife resampling and influence functions to estimate variability.
result The method provides theoretical coverage guarantees on uncertainty intervals.
JAWS audits predictive uncertainty under covariate shift using jackknife+ weighted methods.
problem Auditing predictive uncertainty under data distribution shifts.
method JAW and JAWA methods for distribution-free uncertainty quantification.
result JAW relaxes the jackknife+'s assumption of data exchangeability for covariate shift.
Paper proposes diagnostics for error and variance estimation in randomized matrix computations.
problem Safe use of randomized matrix algorithms in applications.
method Leave-one-out error estimator and jackknife resampling method.
result Provides rapid diagnostics to assess quality of randomized matrix computations.
Adaptive classification methods ensure correct prediction intervals.
problem Developing methods to ensure correct prediction intervals for classification problems.
method Specialized conformal inference techniques combining cross-validation+, jackknife+, and a novel conformity score.
result The methods provide guaranteed approximate conditional coverage for complex data distributions.
New methods improve prediction intervals across multiple environments.
problem Valid confidence intervals and sets in multi-environment prediction.
method Extended jackknife and split-conformal methods, with resizing for problem difficulty.
result Distribution-free coverage achieved in non-traditional data scenarios.
The paper derives uniform stability-based coverage bounds for conformal prediction methods.
problem Establishing theoretical guarantees for conformal prediction methods.
method Uniform stability perspective applied to full-conformal, jackknife+, and CV+ prediction regions.
result Coverage bounds for finite-dimensional models derived using a concentration argument.
Paper accelerates conformal prediction by using approximate leave-one-out estimators.
problem Limited computational cost for conformal prediction.
method Incorporates approximate leave-one-out estimators to accelerate conformal prediction.
result ALO-based methods achieve comparable coverage and efficiency to exact methods but with significantly reduced runtime.
The infinitesimal jackknife (IJ) has recently been applied to the random forest to estimate its prediction variance. These theorems were verified under a traditional random forest framework which uses classification and regression trees (CART) and bootstrap resampling. However, random forests using conditional inferenc…
Random forests are stable and provide reliable prediction intervals.
problem Stability and reliability of random forest prediction intervals.
method Established stability under mild conditions and proved coverage bounds.
result Non-asymptotic lower and upper bounds for prediction interval coverage.
coverforest speeds up conformal predictions for random forests.
problem Efficient uncertainty quantification for random forest predictions.
method Optimized Python package leveraging random forest's out-of-bag scores for cross-conformal predictions.
result coverforest achieves desired coverage with faster training and prediction times.
We use statistical learning methods to construct an adaptive state estimator for nonlinear stochastic systems. Optimal state estimation, in the form of a Kalman filter, requires knowledge of the system's process and measurement uncertainty. We propose that these uncertainties can be estimated from (conditioned on) past…
MAGIC method optimally estimates model predictions changes.
problem Estimating how training data affects model predictions in large-scale settings.
method Combines classical methods and recent advances in metadifferentiation.
result MAGIC method nearly optimally estimates model predictions changes.
Study evaluates posterior covariance matrix W for frequentist evaluation of Bayesian estimators.
problem Evaluating variability of posterior estimates in Bayesian models.
method Use of Bayesian Infinitesimal Jackknife approximation and W-kernel.
result Principal space of W is central to frequentist evaluation of Bayesian models.
A new method reduces bootstrap simulation cost and improves accuracy.
problem Efficiently simulating input uncertainty with large sample sizes.
method Orthogonal Bootstrap: Decomposes into Infinitesimal Jackknife and orthogonal parts.
result Significantly reduces computational cost and maintains accuracy.
Assessing heterogeneous treatment effects has become a growing interest in advancing precision medicine. Individualized treatment effects (ITE) play a critical role in such an endeavor. Concerning experimental data collected from randomized trials, we put forward a method, termed random forests of interaction trees (RF…
Prediction intervals in supervised Machine Learning bound the region where the true outputs of new samples may fall. They are necessary in the task of separating reliable predictions of a trained model from near random guesses, minimizing the rate of False Positives, and other problem-specific tasks in applied Machine …
There has been increasing interest in modelling survival data using deep learning methods in medical research. Current approaches have focused on designing special cost functions to handle censored survival data. We propose a very different method with two steps. In the first step, we transform each subject's survival …
Random forests have proven to be reliable predictive algorithms in many application areas. Not much is known, however, about the statistical properties of random forests. Several authors have established conditions under which their predictions are consistent, but these results do not provide practical estimates of ran…
Paper proposes a new method to improve BART model predictions outside training data range.
problem Improving prediction and prediction intervals for BART models at extrapolation points.
method Gaussian processes are added to BART leaf nodes for extrapolation.
result The new method outperforms standard BART and frequentist resampling methods in simulations.
New algorithm uses control variates to improve multi-armed bandit performance.
problem Stochastic multi-armed bandits with auxiliary reward information.
method Developed UCB-CV algorithm using control variates for mean estimation.
result UCB-CV algorithm provides tighter confidence bounds and smaller variance.
Paper proposes CIV estimator for categorical instruments in small sample settings.
problem Estimation with categorical instruments in settings with few observations per category.
method CIV estimator leveraging regularization assumption for latent categorical variable.
result CIV estimator is asymptotically normal, efficient, and semiparametrically efficient under homoskedasticity.
High-dimensional regression models struggle with resampling methods.
problem Estimating uncertainty in high-dimensional supervised regression tasks.
method Investigation of bootstrap, subsampling, and jackknife methods in high-dimensional generalized linear models.
result Resampling methods exhibit double-descent behavior and are inconsistent in high dimensions.
Proposes a method to make statistical inferences robust in spatially dependent settings with missing at random labels.
problem Statistical inference challenges with missing at random labels and spatial dependence.
method Doubly robust estimator with cross-fit nuisances and jackknife spatial HAC variance correction.
result Asymptotically valid confidence intervals with improved finite-sample calibration.
There has been an increasing interest in testing the equality of large Pearson's correlation matrices. However, in many applications it is more important to test the equality of large rank-based correlation matrices since they are more robust to outliers and nonlinearity. Unlike the Pearson's case, testing the equality…
Conformal prediction is a popular tool for providing valid prediction sets for classification and regression problems, without relying on any distributional assumptions on the data. While the traditional description of conformal prediction starts with a nonconformity score, we provide an alternate (but equivalent) view…
Rectified decision trees improve machine learning interpretability and effectiveness.
problem Combining interpretability and effectiveness in machine learning models.
method Knowledge distillation and modified decision tree splitting criteria.
result Soft labels improve model performance and reduce model size.
New algorithms delete user data from machine learning models efficiently.
problem Deleting user data from machine learning models trained with empirical risk minimization.
method Developed an online unlearning algorithm using the infinitesimal jackknife, targeting non-smooth regularizers.
result Empirically improved runtime while maintaining memory requirements and test accuracy.
Two methods improve Gaussian process predictive distributions' calibration.
problem Improving the reliability of Gaussian process predictive intervals.
method Introduces two methods: cps-gp and bcr-gp, both adapting conformal predictive systems to GP interpolation.
result Both methods provide finite-sample marginal calibration and smooth predictive distributions.
A new method controls risk for set predictors using cross-validation.
problem Inefficient set predictors when data limited.
method Cross-validation conformal risk control (CV-CRC).
result CV-CRC offers theoretical guarantees and reduces set size.
We consider the performance of the bootstrap in high-dimensions for the setting of linear regression, where p<n but p/n is not close to zero. We consider ordinary least-squares as well as robust regression methods and adopt a minimalist performance requirement: can the bootstrap give us good confidence intervals fo…
SurvFM-RMST converts survival outcomes into pseudo-observation targets for tabular models.
problem Right-censored follow-up prevents direct use of survival labels in tabular patient data.
method SurvFM-RMST framework that converts survival outcomes into jackknife pseudo-observation targets for restricted mean survival time.
result SurvFM-RMST accurately recovered restricted event-free time in simulations and outperformed naive targets in static datasets.
Decentralized mechanism for collective predictions without sharing data or models.
problem Making predictions jointly among multiple parties without sharing data or models.
method Inspired by social science consensus-making, a decentralized mechanism for test-time collective predictions.
result Our mechanism converges to inverse meansquared-error weighting in the large-sample limit and achieves significant gains over classical model averaging.
We develop a general framework for distribution-free predictive inference in regression, using conformal inference. The proposed methodology allows for the construction of a prediction band for the response variable using any estimator of the regression function. The resulting prediction band preserves the consistency …
In this paper we propose using the principle of boosting to reduce the bias of a random forest prediction in the regression setting. From the original random forest fit we extract the residuals and then fit another random forest to these residuals. We call the sum of these two random forests a \textit{one-step boosted …
Deep latent variable models have become a popular model choice due to the scalable learning algorithms introduced by (Kingma & Welling, 2013; Rezende et al., 2014). These approaches maximize a variational lower bound on the intractable log likelihood of the observed data. Burda et al. (2015) introduced a multi-sample v…