A stability-based method selects the most desirable conformal prediction set.
problem Selecting the most desirable conformal prediction set from multiple valid sets invalidates coverage guarantees.
method A stability-based approach that ensures coverage for the selected prediction set.
result The stability-based approach maintains coverage guarantees for the selected prediction set.
Develops methods to select informative conformal prediction sets with FCR control.
problem Selecting informative prediction sets with FCR control in supervised learning.
method Unified framework for informative conformal prediction sets with FCR control.
result First procedures providing FCR control for informative prediction sets.
Proposes PEMI for online selective conformal prediction with asymmetric rules.
problem Challenges of handling asymmetric selection mechanisms in online selective conformal prediction.
method PEMI: permutation-based framework for selective conformal prediction with arbitrary asymmetric selection rules.
result Achieves exact selection-conditional coverage for any asymmetric selection mechanism and any prediction model.
CAP algorithm controls FCR in online selective prediction.
problem Online predictive tasks with temporal multiplicity and FCR control.
method CAP framework with adaptive pick rule and calibration set construction.
result CAP achieves exact selection-conditional coverage guarantee and FCR control.
The paper addresses selection bias in conformal prediction for focal units.
problem Selection bias in marginally valid conformal prediction intervals for focal units.
method A general framework for constructing selection-conditional coverage prediction sets.
result Efficient methods for various selection rules with exact finite-sample coverage.
Develops methods to adjust prediction set coverage based on post-selection analysis.
problem Adjusting prediction set coverage after initial analysis to better fit specific needs.
method Post-selection conformal inference to adjust miscoverage levels.
result Allows for trade-off between coverage and prediction set quality.
New causal models perform poorly when evaluated on biased training sets.
problem Sample selection bias affects the evaluation of causal models' prediction performance.
method Re-evaluated prediction performance of causal models on a genetic perturbation data set, proposing a less-biased evaluation set.
result Causal models have similar or worse performance when evaluated on a less-biased set compared to standard association-based estimators.
Framework for assessing fairness across similar predictive models.
problem Fairness in predictive models across different groups.
method Develops a framework for characterizing fairness over the set of good models under selective labels.
result Framework can replace or audit models for better fairness properties.
Proposes a new stability measure for model fitting on similar feature data sets.
problem Model fitting on data sets with similar features is challenging.
method Tuning hyperparameters in a multi-criteria fashion with predictive accuracy and feature selection stability.
result Our approach achieves similar or better predictive performance than single-criteria and stability selection approaches.
Though black-box predictors are state-of-the-art for many complex tasks, they often fail to properly quantify predictive uncertainty and may provide inappropriate predictions for unfamiliar data. Instead, we can learn more reliable models by letting them either output a prediction set or abstain when the uncertainty is…
Random small feature subsets outperform FS in diverse datasets.
problem The significance of selected features in high-dimensional datasets is questionable.
method Analysis of 28 diverse datasets (microarray, RNA-Seq, etc.).
result Any arbitrary set of features performs as well as or better than selected features across datasets.
Proposes a method to compute valid lower confidence bounds for multiple models selected based on their performance.
problem Model selection and evaluation in machine learning.
method Interprets model selection as a simultaneous inference problem, uses bootstrap tilting and maxT-type multiplicity correction.
result Yields valid lower confidence bounds that are at least as good as standard approaches and reliably reach nominal coverage probability.
Cluster stability selection improves feature selection in correlated data.
problem Feature selection stability in correlated data.
method Cluster stability selection exploiting known cluster structure.
result Better predictive performance than lasso alone and stability selection.
Paper introduces novel model selection for CRO to balance robustness and decision risk.
problem Balancing robustness and decision risk in CRO.
method Conformalized Robust Optimization with Model Selection (CROMS) framework.
result Significant improvements in decision efficiency across various applications.
Method evaluates classification uncertainty with adaptively chosen features.
problem Finding a balance between model efficiency and fairness.
method Adaptively selects features for equalized coverage in classification.
result Valid and effective method demonstrated on simulated and real data.
Background: Choosing the most performing method in terms of outcome prediction or variables selection is a recurring problem in prognosis studies, leading to many publications on methods comparison. But some aspects have received little attention. First, most comparison studies treat prediction performance and variable…
Validates conformal prediction for network data under non-uniform sampling.
problem Validity of conformal prediction for network data under non-representative sampling.
method Interprets sampling mechanisms as selection rules, studies validity conditional on selection events, uses permutation invariance and joint exchangeability.
result Finite-sample validity of conformal prediction for certain selection events and asymptotic validity for random walk sampling.
FIRES framework selects stable features from online data.
problem Efficiently selecting features in online settings with limited data.
method FIRES framework uses model parameter importance for feature selection.
result FIRES selects stable feature sets with minimal model complexity.
New method for multi-modal depth prediction challenges.
problem Challenges in applying unimodal coreset selection to multi-modal data.
method Adapted state-of-the-art coreset selection technique for multimodal data.
result Challenges in extending unimodal algorithms to multi-modal scenarios.
Stable health predictions need deconfounding test set features.
problem Stability of predictions in health machine learning is compromised by selection biases.
method Deconfounding the test set features improves prediction stability across different environments.
result Improved stability achieved by deconfounding test set features.
Variable selection for Gaussian process models is often done using automatic relevance determination, which uses the inverse length-scale parameter of each input variable as a proxy for variable relevance. This implicitly determined relevance has several drawbacks that prevent the selection of optimal input variables i…
Benchmarking 18 deep neural network selective classification models on various datasets.
problem Designing a model that can abstain from making predictions when there's high risk of error.
method Evaluation of 18 selective classification models using multiple criteria.
result No single model is the clear winner; best model depends on objectives.
COMA combines prediction sets from multiple models for online, adaptive prediction.
problem Combining multiple prediction models with uncertainty guarantees.
method Online model aggregation using weighted voting of conformal prediction sets.
result COMA retains coverage guarantees under negative correlation assumptions.
The study improves compound selection in in silico screening by focusing on model's ability to predict desirable outcomes.
problem Improving compound selection in in silico screening to reduce errors and enhance generalization.
method Extending learning theory, the study analyzes the impact of selection policies on generalization and proposes a method to mitigate challenges.
result Generalization can be enhanced by considering a model's ability to predict the fraction of desired outcomes in a batch.
Active learning selects samples for labeling to build accurate models with minimal labeled data.
problem Costly acquisition of labeled data in supervised learning.
method Adaptive selection of unlabeled data samples for labeling.
result Efficient model building with minimal labeled data.
OnlineSCI extends ACI for adaptive selective inference with improved coverage and IER control.
problem Adaptive selective inference in online settings with improved coverage and IER control.
method Adaptive selective inference with extended ACI algorithm.
result OnlineSCI controls average missed coverage and instantaneous error rate at selected times, up to a non-asymptotic remainder term.
A new method for selective classification trades off accuracy for coverage.
problem Selective classification allows a classifier to abstain from predicting some instances.
method Optimizes a collection of class-wise decoupled one-sided empirical risks.
result The method achieves near-optimal coverage in high target accuracy regimes.
A stability metric compares feature selection algorithms in machine learning.
problem Stability of feature selection algorithms in machine learning.
method Rank-based instability index to compare MDA, LIME, and SHAP algorithms.
result LIME and SHAP are more stable than MDA, with LIME being best for human interpretability.
New method predicts model performance under selection bias in healthcare.
problem Selection bias limits model generalizability in healthcare.
method Proposes a novel upper bound method for estimating model performance.
result Validates and demonstrates the practical utility of the method.
Machine learning selects the best prediction rules from noisy data.
problem Selection under uncertainty in machine learning.
method Statistical tools and inequalities to control noise in empirical estimates.
result Theoretical guarantees on selection outcomes under uncertainty.
PEAKS selects key training examples incrementally based on prediction error and kernel similarity.
problem Dynamic data selection in deep learning models.
method Prediction Error Anchored by Kernel Similarity (PEAKS) for incremental data selection.
result PEAKS outperforms existing selection strategies and yields better performance returns as training data size grows.
Paper improves feature selection for predicting outcomes from observational data.
problem Feature selection for post-intervention outcome prediction from pre-intervention variables in healthcare settings.
method Extends Markov boundary concept to treatment-outcome pairs, uses observational and experimental data.
result Combining observational and experimental data improves feature selection and effect estimation.
Given a set of possible models (e.g., Bayesian network structures) and a data sample, in the unsupervised model selection problem the task is to choose the most accurate model with respect to the domain joint probability distribution. In contrast to this, in supervised model selection it is a priori known that the chos…
EviTrack improves sequential prediction in delayed disambiguation scenarios.
problem Challenges in sequential prediction with delayed disambiguation where early observations are ambiguous.
method EviTrack operates over latent trajectories, applying evidence- and likelihood-ratio-based selection to delay commitment until supported by data.
result EviTrack outperforms sampling-based baselines in a controlled synthetic benchmark, achieving faster post-disambiguation recovery.
New algorithms reduce label collection for online prediction with expert advice.
problem Efficiently predicting binary sequences with expert advice using fewer labels.
method Adaptive selective sampling for exponentially weighted forecasters.
result Label complexity scales roughly as the square root of the number of rounds for a scenario with a strictly better expert.
Efficiently predict LLM benchmarks using feature selection and regression.
problem Predicting full benchmark scores with minimal question subsets.
method Multiple regression with feature selection, using kernel ridge regression and mRMR.
result Improved prediction accuracy and ranking correlation across various benchmarks.
ARIMA and LSTM models predict stock prices with varying accuracy.
problem Predicting stock prices accurately.
method ARIMA and LSTM models trained on historical stock data.
result LSTM model performs better in predicting stock price changes.
Knockoffs method selects financial factors, controlling false discoveries.
problem Controlling false discoveries in financial factor selection.
method Apply knockoff procedure to build fake factors.
result Shows versatility in fund replication and network inference.
New method for selective prediction under interventions learns causal structure from data.
problem Tight uncertainty sets in selective conformal prediction under unknown interventional settings.
method Partial causal structure learning for descendant indicators, contamination-robust coverage theorem, algorithms for descendant discovery and distance estimation.
result Valid selective conformal prediction under contamination up to 30% with controlled coverage.
New method selects relevant dimensions for better prediction in mixtures.
problem Learning mixtures with limited components for prediction tasks.
method Prediction-focused modeling for mixtures.
result Improves prediction performance compared to non-focused models.
In this thesis we discuss machine learning methods performing automated variable selection for learning sparse predictive models. There are multiple reasons for promoting sparsity in the predictive models. By relying on a limited set of input variables the models naturally counteract the overfitting problem ubiquitous …
Missing data are a concern in many real world data sets and imputation methods are often needed to estimate the values of missing data, but data sets with excessive missingness and high dimensionality challenge most approaches to imputation. Here we show that appropriate feature selection can be an effective preprocess…
Selective classification improves trading strategies by abstaining from predictions.
problem Designing effective trading strategies using selective classification.
method Extends binary or multi-class classifiers to allow abstaining from predictions, evaluates across different feature sets and classifiers.
result Selective classifiers can improve trading performance by avoiding poor predictions.
New method improves reliability of selecting individuals based on predicted treatment effects.
problem Reliability of selecting individuals based on predicted conditional average treatment effects (CATE) is unreliable.
method Denoised Conformal Alignment, combining proxy errors, variance estimation, and Benjamini-Hochberg selection.
result Significantly improved power in selecting individuals while maintaining false discovery rate control.
The success of machine learning on a given task dependson, among other things, which learning algorithm is selected and its associated hyperparameters. Selecting an appropriate learning algorithm and setting its hyperparameters for a given data set can be a challenging task, especially for users who are not experts in …
Heterogeneous ensembles built from the predictions of a wide variety and large number of diverse base predictors represent a potent approach to building predictive models for problems where the ideal base/individual predictor may not be obvious. Ensemble selection is an especially promising approach here, not only for …
Given a set of possible models (e.g., Bayesian network structures) and a data sample, in the unsupervised model selection problem the task is to choose the most accurate model with respect to the domain joint probability distribution. In contrast to this, in supervised model selection it is a priori known that the chos…
A new method combines multiple bounds and betting strategies for selective prediction, improving risk coverage in data-scarce settings.
problem Selective prediction with risk control in data-scarce domains.
method Combines concentration inequalities, multiple-testing corrections, and betting-based confidence sequences.
result Transfer-Informed Betting achieves tighter bounds and better coverage in data-scarce settings.