Jigsaw-VAE tackles feature imbalance in VAE latent variables, improving generalization across environments.
problem Feature imbalance in VAE latent variables leads to poor generalization and biased sample generation.
method Proposes a regularization scheme to balance features in VAE latent variables and introduces a metric to measure balance.
result The regularization scheme substantially addresses feature imbalance, leading to improved generalization and diverse sample generation.
Study examines challenges in variable importance ranking due to feature correlation.
problem Challenges in variable importance ranking under correlation.
method Simulation study and theoretical analysis of feature knockoffs and conditional predictive impact (CPI).
result Highly correlated features increase the correlation of knockoff variables, posing a limitation for CPI.
Extraneous variables are variables that are irrelevant for a certain task, but heavily affect the distribution of the available data. In this work, we show that the presence of such variables can degrade the performance of deep-learning models. We study three datasets where there is a strong influence of known extraneo…
The paper investigates causal relationships in heart failure prediction using machine learning.
problem Understanding the causal relationships between clinical variables and heart failure.
method Proposes a new computational framework for causal structure discovery (CSD) of mixed-type clinical variables for binary disease outcomes.
result Feature importance from nonlinear classifiers strongly correlates with causal strength of variables, but not differentiating cause and effect.
XGBoost fails to accurately identify relevant features, while interpretable methods do.
problem Accurately identifying relevant features in black-box models like XGBoost.
method Comparison of variable importance methods (CART, Optimal Trees, XGBoost, SHAP) across various experiments.
result Interpretable methods outperform black-box models in feature selection accuracy.
Proposes a framework to assess feature importance without algorithm constraints.
problem Lack of a general framework for assessing feature importance across different algorithms.
method Develops a nonparametric framework for algorithm-agnostic variable importance assessment.
result Valid confidence intervals and testing strategies for variable importance.
Dealing with datasets of very high dimension is a major challenge in machine learning. In this paper, we consider the problem of feature selection in applications where the memory is not large enough to contain all features. In this setting, we propose a novel tree-based feature selection approach that builds a sequenc…
A new method learns outcome-aware spectral features for causal effect estimation.
problem Estimation of causal effects in the presence of hidden confounders.
method Augmented Spectral Feature Learning framework that minimizes a contrastive loss derived from an augmented operator incorporating outcome information.
result Our method remains effective even under spectral misalignment.
Develops a new method for nonlinear dimension reduction using random features.
problem Statistical challenges in generalizing Gaussian process-based latent variable models to non-Gaussian data.
method Random feature latent variable models (RFLVMs) that approximate nonlinear relationships with linear functions of random features.
result RFLVMs produce comparable results to state-of-the-art methods on various data types.
New method reconstructs missing variables in time series using autoencoders and automatic differentiation.
problem Reconstruct missing variables in time series with flexible input and output combinations.
method Train an autoencoder with all features, optimize missing variables as inputs, and use automatic differentiation.
result Flexible input and output combinations can be achieved without retraining the autoencoder.
A single pre-trained agent guides feature selection using knockoffs.
problem Feature selection challenges in AI-readiness of data.
method Generates knockoff features and uses reinforcement learning.
result Optimal feature subset identified with reduced dependency on target variable.
New causal measures improve feature selection in AI models.
problem Lack of causal interpretability in AI models.
method Introduces causal entropy and causal information gain to assess feature control.
result Demonstrates superiority of causal information gain in feature selection.
DFIV uses deep neural nets to learn nonlinear features in IV regression.
problem Learning causal relationships from observational data with nonlinear interactions.
method DFIV trains deep neural nets to define nonlinear features on instruments and treatments, alternating training to compose stages 1 and 2.
result DFIV outperforms state-of-the-art methods on IV benchmarks and off-policy policy evaluation.
FFRK automatically extracts features for spatial interpolation without external variables.
problem Spatial interpolation challenges, especially nonstationarity and lack of explanatory variables.
method Feature-Free Regression Kriging (FFRK) method that extracts geospatial features.
result FFRK outperforms classical methods in predicting heavy metal concentrations.
Two ANOVA-based algorithms boost random Fourier feature models for function approximation.
problem Approximating high-dimensional functions with low-order interactions.
method Utilizes ANOVA decomposition to learn low-order functions and index sets of important variables.
result Significantly reduces approximation error compared to existing methods.
Develops ELBD for efficient feature selection in VAE latent variables.
problem Feature selection in latent variables of VAE and its variants.
method ELBD score algorithm and weak convergence approximation for optimization.
result Effective feature selection and optimization of VAE models.
Improves speaker verification for variable-duration utterances using a feature pyramid module.
problem Improving robustness for variable-duration utterances in speaker verification.
method Integrates a feature pyramid module into multi-scale aggregation to enhance speaker-discriminative information from multiple layers.
result Improves performance for both short and long utterances compared to state-of-the-art approaches.
The amount of information in the form of features and variables avail- able to machine learning algorithms is ever increasing. This can lead to classifiers that are prone to overfitting in high dimensions, high di- mensional models do not lend themselves to interpretable results, and the CPU and memory resources necess…
ABM automates feature engineering and variable selection for loss-based models.
problem Improving model performance through better feature engineering and variable selection.
method ABM uses group and fused lasso regularization to automatically select cutting points and variables.
result ABM integrates feature engineering, variable selection, and model training.
A novel graph spectral method for mixed categorical and numerical data.
problem Feature learning for mixed data types (numerical and categorical).
method Graph spectral decomposition of the graph Laplacian to model probabilistic dependence structure.
result Increased separability and clusterability of observations in the transformed feature space.
Brain signal variability in the measurements obtained from different subjects during different sessions significantly deteriorates the accuracy of most brain-computer interface (BCI) systems. Moreover these variabilities, also known as inter-subject or inter-session variabilities, require lengthy calibration sessions b…
FIRES framework selects stable features from online data.
problem Efficiently selecting features in online settings with limited data.
method FIRES framework uses model parameter importance for feature selection.
result FIRES selects stable feature sets with minimal model complexity.
Strategic feature manipulation helps learners identify meaningful variables in online regression settings.
problem Strategic feature manipulation poses a challenge for learners in online regression settings.
method Investigates how strategic feature manipulation by individuals can help learners recover meaningful features.
result Simple learner behavior can help accurately recover meaningful features and incentivize feature improvement.
Random feature matrices' singular values concentrate near their full expectation in high dimensions.
problem Characterizing the spectra of random feature matrices for regression problems.
method Analyzing two settings of input variables (random or well-separated) with conditions on dimension, complexity ratio, and sampling variance.
result The singular values of random feature matrices concentrate near their full expectation and near one with high probability.
We propose a new method for input variable selection in nonlinear regression. The method is embedded into a kernel regression machine that can model general nonlinear functions, not being a priori limited to additive models. This is the first kernel-based variable selection method applicable to large datasets. It sides…
FREEtree improves tree-based methods for correlated longitudinal data.
problem Poor performance of Random Forests in high dimensional longitudinal data with correlated features.
method FREEtree uses a piecewise random effects model and clustering with WGCNA to select features and maintain interpretability.
result FREEtree outperforms other tree-based methods in prediction and feature selection accuracy.
Estimates and infers multi-stage stationary treatment policies with variable selection.
problem Valid inference for multi-stage stationary treatment policies with high-dimensional feature variables.
method Estimate the value function using augmented inverse probability weighted estimator, apply penalty for variable selection, construct one-step improvements for valid inference.
result Improved estimators are asymptotically normal, valid inference for policy parameters demonstrated.
In many cases, feature selection is often more complicated than identifying a single subset of input variables that would together explain the output. There may be interactions that depend on contextual information, i.e., variables that reveal to be relevant only in some specific circumstances. In this setting, the con…
A new method selects features for clustering without labels.
problem Identifying meaningful features in large datasets.
method Differentiable unsupervised feature selection using a gated Laplacian.
result The method improves clustering performance in noisy data.
Paper defines feature impact and importance from data, not models.
problem Misinterpretation of feature importance as impact leads to flawed insights.
method Mathematical definitions of feature impact and importance derived from partial dependence curves.
result Feature rankings by these definitions are competitive with existing techniques.
Extracts invariant features to predict Y without confounding by Z, using conditional independence and optimal transport.
problem Extracting invariant features to predict Y without confounding by Z, a response variable influenced by unknown confounders Z.
method Develops a methodology penalizing statistical dependence between feature and confounders conditioned on Y, using the Optimal Transport Barycenter Problem.
result The method extracts invariant features in the Gaussian case, equivalent to penalizing dependence between feature and conditional random variable Z_Y.
Bayesian non-linear latent variable modeling for complex data.
problem Inference for GPLVMs is computationally limited and often leads to overfitting or underestimates uncertainty.
method Approximate Gaussian process mappings with random Fourier features for MCMC inference.
result Generalized RFLVMs perform well on various data types and applications.
When applied to high-dimensional datasets, feature selection algorithms might still leave dozens of irrelevant variables in the dataset. Therefore, even after feature selection has been applied, classifiers must be prepared to the presence of irrelevant variables. This paper investigates a new training method called Co…
Two statistical tasks are shown to have equivalent sample complexity.
problem Determining if a function depends on only a few variables and identifying those variables.
method Proved statistical equivalence of feature selection and junta testing through sample complexity analysis.
result Brute-force algorithm is sample-optimal for both tasks with optimal sample size.
Proposes a few-shot learning method for feature selection without labeled data.
problem Feature selection in unlabeled data with limited instances.
method Uses Concrete random variables and permutation-invariant neural networks to select features from multiple source tasks.
result Outperforms existing methods in feature selection performance.
The identification of relevant features, i.e., the driving variables that determine a process or the properties of a system, is an essential part of the analysis of data sets with a large number of variables. A mathematical rigorous approach to quantifying the relevance of these features is mutual information. Mutual i…
New method selects features via tensor decomposition and submodular optimization.
problem Feature selection for high-dimensional data.
method Low-rank tensor model, submodular optimization, greedy algorithm.
result Proposed method outperforms state-of-the-art feature selection.
A new variable importance measure for DRFs detects broader impacts on output distributions.
problem Estimating full conditional distributions of multivariate outputs given inputs.
method Based on the drop and relearn principle and MMD distance.
result Consistent and high-performing variable importance measure for DRFs.
This paper explores emerging methods for estimating variable importance in machine learning.
problem Estimating the importance of variables in machine learning models.
method Least Absolute Shrinkage and Selection Operator (LASSO), Support Vector Machine (SVM), Predictive Error Function (PERF), Random Forest (RF), Extreme Gradient Boosting (XGBOOST).
result PERF and RF showed the best performance in highly correlated data, while PERF and XGBOOST performed poorly on small data sizes.
New method selects direct causal parents from large sets of variables.
problem Inferring direct causal parents from many variables, especially nonlinear and cyclic.
method One-vs.-the-rest feature selection approach with theoretical guarantees.
result Significant improvements over existing methods.
In many fields such as bioinformatics, high energy physics, power distribution, etc., it is desirable to learn non-linear models where a small number of variables are selected and the interaction between them is explicitly modeled to predict the response. In principle, neural networks (NNs) could accomplish this task s…
We present an Automatic Relevance Determination prior Bayesian Neural Network(BNN-ARD) weight l2-norm measure as a feature importance statistic for the model-x knockoff filter. We show on both simulated data and the Norwegian wind farm dataset that the proposed feature importance statistic yields statistically signific…
Paper introduces AIF for anomaly detection with variable feature sensitivity.
problem Lack of variable sensitivity in anomaly detection methods.
method Extended Isolation Forest with feature sensitivities (Anisotropic Isolation Forest).
result AIF enables anomaly detection with controllable sensitivity to different features.
Study compares single vs ensemble feature selection for cancer diagnosis.
problem Identifying relevant variables for cancer diagnosis and prognosis.
method Comparison of single feature selection algorithms and ensemble of diverse algorithms.
result Ensemble approach did not improve predictive performance over individual algorithms.
We propose the Sobolev Independence Criterion (SIC), an interpretable dependency measure between a high dimensional random variable X and a response variable Y . SIC decomposes to the sum of feature importance scores and hence can be used for nonlinear feature selection. SIC can be seen as a gradient regularized Integr…
I consider unsupervised extensions of the fast stepwise linear regression algorithm \cite{efroymson1960multiple}. These extensions allow one to efficiently identify highly-representative feature variable subsets within a given set of jointly distributed variables. This in turn allows for the efficient dimensional reduc…
Introduces RFI for assessing feature importance relative to any subset of features.
problem Lack of nuanced feature importance computation.
method Generalizes PFI and CFI to assess relative feature importance.
result Derives general interpretation rules for RFI.
Proposes a neural network framework for feature selection in high-dimensional settings.
problem Challenges in feature selection and non-linear function estimation in high-dimensional settings.
method Sparse-input neural networks using group concave regularization.
result Establishes finite-sample guarantees for variable selection consistency and prediction accuracy.