Variable importance is central to scientific studies, including the social sciences and causal inference, healthcare, and other domains. However, current notions of variable importance are often tied to a specific predictive model. This is problematic: what if there were multiple well-performing predictive models, and …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “variable importance”
Random Forest variable importance is improved by class balancing techniques.
problem Class imbalance problem in machine learning.
method Proposed a variable selection algorithm using RF variable importance and its confidence interval.
result Our algorithm efficiently selects an optimal feature set, leading to improved prediction performance.
New framework quantifies variable importance across all good models and is stable across data distribution.
problem Conflicting variable importance conclusions from different models trained on the same data.
method Proposes a new variable importance framework that considers all good models and is stable across data distribution.
result Framework accurately estimates true variable importance and recovers rankings for complex setups.
A new variable importance measure for DRFs detects broader impacts on output distributions.
problem Estimating full conditional distributions of multivariate outputs given inputs.
method Based on the drop and relearn principle and MMD distance.
result Consistent and high-performing variable importance measure for DRFs.
CPI overcomes limitations of permutation importance by providing accurate variable selection.
problem Misidentification of unimportant variables in complex models due to covariate correlations.
method Developed a model agnostic and computationally lean Conditional Permutation Importance (CPI) approach.
result CPI provides accurate type-I error control and more parsimonious variable selection.
Proposes a simple solution to Gini importance bias in random forests.
problem Gini importance measure in random forests is biased and unreliable.
method Computes loss reduction on out-of-bag samples instead of in-bag.
result Solves the misleading/untrustworthy Gini importance issue.
Causal Explainability of Machine Learning in Heart Failure Prediction from Electronic Health Recordsstat.ML
The paper investigates causal relationships in heart failure prediction using machine learning.
problem Understanding the causal relationships between clinical variables and heart failure.
method Proposes a new computational framework for causal structure discovery (CSD) of mixed-type clinical variables for binary disease outcomes.
result Feature importance from nonlinear classifiers strongly correlates with causal strength of variables, but not differentiating cause and effect.
Inference on Variable Importance for Treatment Effect Heterogeneity: Shapley Values and Beyondstat.ME
Framework assesses variable importance for heterogeneous treatment effects.
problem High-risk domains need reliable methods to assess treatment effect heterogeneity.
method Inferential framework based on Shapley values and semiparametric theory.
result Valid inference on variable importance for heterogeneous treatment effects.
Hierarchical-CPI improves variable importance measurement for medical data.
problem Limited interpretability of complex medical models.
method Hierarchical-CPI measures conditional variable importance with statistical control, handling correlated data.
result Hierarchical-CPI outperforms existing methods in medical datasets.
Study examines challenges in variable importance ranking due to feature correlation.
problem Challenges in variable importance ranking under correlation.
method Simulation study and theoretical analysis of feature knockoffs and conditional predictive impact (CPI).
result Highly correlated features increase the correlation of knockoff variables, posing a limitation for CPI.
Proposes CLIQUE for improved local variable importance in multi-class classification.
problem Lack of methods to characterize local structure in model loss space.
method CLIQUE (Conditional Local Importance by Quantile Expectations)
result CLIQUE emphasizes locally dependent information and captures interaction behavior.
Proposes a new method for variable importance using targeted learning.
problem Uncertainty quantification in variable importance metrics.
method Employing the targeted learning (TL) framework for conditional permutation variable importance.
result Improved accuracy in finite sample contexts compared to traditional methods.
Tree ensemble methods such as random forests [Breiman, 2001] are very popular to handle high-dimensional tabular data sets, notably because of their good predictive accuracy. However, when machine learning is used for decision-making problems, settling for the best predictive procedures may not be reasonable since enli…
Proposes a framework to assess feature importance without algorithm constraints.
problem Lack of a general framework for assessing feature importance across different algorithms.
method Develops a nonparametric framework for algorithm-agnostic variable importance assessment.
result Valid confidence intervals and testing strategies for variable importance.
A new method for variable importance measures without impossible data.
problem Using impossible data for variable importance measures in black box models.
method Cohort Shapley, a method grounded in economic game theory using only observed data.
result Cohort Shapley provides a more trustworthy explanation of black box models' decisions.
Efficient nonparametric statistical inference on population feature importance using Shapley valuesstat.ME
Efficiently estimates variable importance in prediction tasks using Shapley values.
problem Valid statistical inference on the importance of variables in prediction tasks.
method Randomly sampling feature subsets to estimate Shapley Population Variable Importance Measure (SPVIM) efficiently.
result The proposed estimator converges at an asymptotically optimal rate and can construct valid confidence intervals and hypothesis tests.
A new method selects important variables for clustering from dependency networks.
problem Variable selection for clustering in high-cost data scenarios.
method Create dependency networks, rank variables by centrality, select top-n variables.
result Top-n variables improve clustering performance compared to existing methods.
Fast method estimates variable importance for large neural networks.
problem Estimating variable importance in large neural networks is computationally expensive and lacks theoretical guarantees.
method Linearization initialized at full model parameters with ridge-like penalty.
result Estimates variable importance with error rate of and asymptotic normality.
In this paper, we propose multi-variable LSTM capable of accurate forecasting and variable importance interpretation for time series with exogenous variables. Current attention mechanism in recurrent neural networks mostly focuses on the temporal aspect of data and falls short of characterizing variable importance. To …
From global to local MDI variable importances for random forests and when they are Shapley valuesstat.ML
New local MDI variable importances derived from global scores match Shapley values.
problem Local feature relevance in tree-based models.
method Deriving local MDI importance measure from global scores and linking it to Shapley values.
result Local MDI importances have a natural connection with Shapley values.
XGBoost fails to accurately identify relevant features, while interpretable methods do.
problem Accurately identifying relevant features in black-box models like XGBoost.
method Comparison of variable importance methods (CART, Optimal Trees, XGBoost, SHAP) across various experiments.
result Interpretable methods outperform black-box models in feature selection accuracy.
New method quantifies variable importance in causal forests for treatment effect heterogeneity.
problem Lack of understanding how input variables affect treatment effect heterogeneity in causal forests.
method Developed a new importance variable algorithm for causal forests based on the drop and relearn principle.
result Shows how to handle forest retraining without a confounding variable and introduces a corrective term for confounders.
Defines a new metric to measure importance of predictors in complex machine learning models.
problem Measuring importance of predictors in black box machine learning models.
method Introduces a new metric, GVIM, based on true conditional expectation functions and causal interpretation.
result The GVIM can be represented as a function of Conditional Average Treatment Effect (CATE), providing a causal interpretation.
Proposes a new method to estimate variable importance in black box models, mitigating correlation effects.
problem Correlation between covariates affects the interpretation of variable importance parameters.
method Develops a modified LOCO (Leave Out COvariates) method and uses semiparametric models for estimation.
result Shows how to estimate a modified LOCO method that mitigates correlation effects.
New method estimates variable importance for large models efficiently.
problem Estimating variable importance for large, opaque models is computationally challenging.
method Combining early stopping and warm-start techniques for scalable variable importance estimation.
result The method provides theoretical guarantees and demonstrates improved accuracy and computational efficiency.
We characterize and study variable importance (VIMP) and pairwise variable associations in binary regression trees. A key component involves the node mean squared error for a quantity we refer to as a maximal subtree. The theory naturally extends from single trees to ensembles of trees and applies to methods like rando…
Develops HCQRF for estimating heterogeneous treatment effects with censored data.
problem Estimating heterogeneous treatment effects on censored responses with high-dimensional variables.
method Hybrid Censored Quantile Regression Forest (HCQRF) combining random forests and censored quantile regression.
result Demonstrates the effectiveness and stability of HCQRF through simulation studies and real-world application.
The paper derives theoretical foundations for two common machine learning variable importance measures.
problem Understanding variable importance in machine learning problems.
method The paper derives closed-form expressions for Permute-and-Predict (PaP) and Leave-One-Covariate-Out (LOCO) methods.
result Theoretical derivations explain the behavior of PaP and LOCO under collinearity, linking them to coefficients and predictor variability.
This paper explores emerging methods for estimating variable importance in machine learning.
problem Estimating the importance of variables in machine learning models.
method Least Absolute Shrinkage and Selection Operator (LASSO), Support Vector Machine (SVM), Predictive Error Function (PERF), Random Forest (RF), Extreme Gradient Boosting (XGBOOST).
result PERF and RF showed the best performance in highly correlated data, while PERF and XGBOOST performed poorly on small data sizes.
The paper introduces Shapley curves for measuring variable importance in nonparametric settings.
problem Limited statistical understanding of Shapley values as variable importance measures.
method Introduces Shapley curves based on conditional expectation and covariate distribution; derives convergence rates and normality; proposes a novel bootstrap procedure.
result Validates theoretical findings with numerical studies and analyzes vehicle prices determinants.
This work develops rigorous theoretical basis for the fact that deep Bayesian neural network (BNN) is an effective tool for high-dimensional variable selection with rigorous uncertainty quantification. We develop new Bayesian non-parametric theorems to show that a properly configured deep BNN (1) learns the variable im…
Variable selection in sparse regression models is an important task as applications ranging from biomedical research to econometrics have shown. Especially for higher dimensional regression problems, for which the link function between response and covariates cannot be directly detected, the selection of informative va…
Variable selection for optimal treatment regime in a clinical trial or an observational study is getting more attention. Most existing variable selection techniques focused on selecting variables that are important for prediction, therefore some variables that are poor in prediction but are critical for decision-making…
This paper examines from an experimental perspective random forests, the increasingly used statistical method for classification and regression problems introduced by Leo Breiman in 2001. It first aims at confirming, known but sparse, advice for using random forests and at proposing some complementary remarks for both …
New method reduces high-dimensional data to key features.
problem Challenges of high-dimensional data analysis and interpretability.
method Randomized search to produce subspaces, ensemble of models for variable selection.
result Outperforms existing methods in prediction and variable selection.
This paper introduces and develops a novel variable importance score function in the context of ensemble learning and demonstrates its appeal both theoretically and empirically. Our proposed score function is simple and more straightforward than its counterpart proposed in the context of random forest, and by avoiding …
In this paper, we propose an interpretable LSTM recurrent neural network, i.e., multi-variable LSTM for time series with exogenous variables. Currently, widely used attention mechanism in recurrent neural networks mostly focuses on the temporal aspect of data and falls short of characterizing variable importance. To th…
Regression Trees and Random forest based feature selection for malaria risk exposure predictionstat.ML
This paper deals with prediction of anopheles number, the main vector of malaria risk, using environmental and climate variables. The variables selection is based on an automatic machine learning method using regression trees, and random forests combined with stratified two levels cross validation. The minimum threshol…
SHAFF estimates Shapley effects efficiently even with dependent variables.
problem Challenges in estimating Shapley effects, especially with dependent variables.
method SHAFF uses random forests to estimate Shapley effects efficiently.
result SHAFF provides a fast and accurate estimate of Shapley effects.
Finite mixture model is an important branch of clustering methods and can be applied on data sets with mixed types of variables. However, challenges exist in its applications. First, it typically relies on the EM algorithm which could be sensitive to the choice of initial values. Second, biomarkers subject to limits of…
An Automatic Relevance Determination Prior Bayesian Neural Network for Controlled Variable Selectionstat.ML
We present an Automatic Relevance Determination prior Bayesian Neural Network(BNN-ARD) weight l2-norm measure as a feature importance statistic for the model-x knockoff filter. We show on both simulated data and the Norwegian wind farm dataset that the proposed feature importance statistic yields statistically signific…
Researchers develop a new method to assess variable importance in spatial machine learning models for air pollution exposure prediction.
problem Understanding the mechanism captured by machine learning models in air pollution studies, especially with spatial correlation.
method Leave-one-out approach for variable importance measure applicable to models with separable mean and covariance components.
result The new method highlights differences in model mechanisms even for similar prediction accuracies.
We introduce a variable importance measure to quantify the impact of individual input variables to a black box function. Our measure is based on the Shapley value from cooperative game theory. Many measures of variable importance operate by changing some predictor values with others held fixed, potentially creating unl…
Random forest hyperparameters affect variable selection in omics studies.
problem Impact of hyperparameters on variable selection in random forests.
method Two simulation studies using theoretical and empirical data.
result Hyperparameters influence variable selection more than the splitting strategy and sample fraction.
Paper defines feature impact and importance from data, not models.
problem Misinterpretation of feature importance as impact leads to flawed insights.
method Mathematical definitions of feature impact and importance derived from partial dependence curves.
result Feature rankings by these definitions are competitive with existing techniques.
A new method assesses algorithmic fairness using game theory.
problem Evaluating algorithmic fairness without proprietary data.
method Cohort Shapley value, a game-theoretic approach.
result Identifies individual impact of protected attributes.
In this paper we propose a computationally efficient algorithm for on-line variable selection in multivariate regression problems involving high dimensional data streams. The algorithm recursively extracts all the latent factors of a partial least squares solution and selects the most important variables for each facto…
Asymptotic Theory and Phase Transitions for Variable Importance in Quantile Regression Forestsstat.ML
The paper develops asymptotic theory for QRF variable importance, revealing a bias-variance trade-off.
problem Challenges in statistical inference for QRF variable importance due to non-smoothness and bias-variance trade-off.
method Developed asymptotic theory using pinball loss and Knight's identity, uncovered phase transition phenomenon, derived asymptotic bias.
result Theoretical foundation for understanding QRF inference limitations in high-dimensional settings.