AEN-SAEs address feature starvation in sparse autoencoders by stabilizing the geometric alignment of sparse coding.
problem Feature starvation in sparse autoencoders, leading to unstable and misaligned representations.
method Adaptive Elastic Net SAEs (AEN-SAEs) combine ℓ2 and ℓ1 terms to stabilize the sparse coding map and control feature interactions. result AEN-SAEs mitigate feature starvation without heuristic resampling, maintaining competitive reconstruction abilities.
Gradient Starvation causes neural networks to focus on a subset of features, leading to imbalance.
problem Neural networks focusing on only a subset of features despite having access to more.
method Using Dynamical Systems theory, identified properties of learning dynamics leading to feature imbalance.
result Gradient descent can lead to feature imbalance in over-parameterized networks.
While a lot of progress has been made in recent years, the dynamics of learning in deep nonlinear neural networks remain to this day largely misunderstood. In this work, we study the case of binary classification and prove various properties of learning in such networks under strong assumptions such as linear separabil…
New method learns priors for Bayesian neural networks from datasets.
problem Lack of prior beliefs in Bayesian deep learning.
method Amortised variational inference to learn priors from datasets.
result Flexible Bayesian neural networks for meta-learning and within-task minibatching.
Gaussian processes (GP) are a well studied Bayesian approach for the optimization of black-box functions. Despite their effectiveness in simple problems, GP-based algorithms hardly scale to high-dimensional functions, as their per-iteration time and space cost is at least quadratic in the number of dimensions d and i…
A new method for multi-expert learning-to-defer avoids optimization issues.
problem Optimization issues in multi-expert learning-to-defer systems.
method A decoupled surrogate with a softmax classifier head and independent sigmoid heads per expert.
result First multi-expert L2D guarantee with a constant not growing with the expert pool.
We quantify forgetting in post-training models, distinguishing mass and drift.
problem Understanding and preventing forgetting in post-training generative models.
method Developed theoretical results under a two-mode mixture abstraction, formalizing mass and drift forgetting.
result Forgetting can be precisely quantified based on divergence direction, geometric overlap, and training regime.
Identifies features most relevant to concept drift in data.
problem Identifying features most relevant to concept drift.
method Distinguishing between drift inducing and faithfully drifting features; deriving minimal subsets of features to characterize drift.
result Derives a detection algorithm for concept drift.
Introduces RFI for assessing feature importance relative to any subset of features.
problem Lack of nuanced feature importance computation.
method Generalizes PFI and CFI to assess relative feature importance.
result Derives general interpretation rules for RFI.
A single pre-trained agent guides feature selection using knockoffs.
problem Feature selection challenges in AI-readiness of data.
method Generates knockoff features and uses reinforcement learning.
result Optimal feature subset identified with reduced dependency on target variable.
Feature networks link ML features via graph structure for enhanced learning.
problem Enhancing feature expressiveness and learning efficiency in machine learning.
method Graph representation of feature vectors, leveraging Fourier and functional analysis.
result Feature networks enable novel, complex feature dependencies.
Approach for selecting features by discarding nuisance and correlated ones.
problem Large datasets with correlated and nuisance features.
method Laplacian score criterion, autoencoder architecture, concrete layer.
result Outperforms similar approaches in clustering performance.
New stability measures for similar features improve feature selection accuracy.
problem Existing stability measures fail to distinguish similar features in highly correlated datasets.
method Introduce new adjusted stability measures that consider feature similarities.
result One new stability measure considers highly similar features as interchangeable.
Counterexamples show HSIC feature selection misses critical features.
problem Feature selection using HSIC misses important features.
method Feature selection via HSIC maximization.
result HSIC feature selection can miss critical features.
This paper shows feature importance remains valid even in low-performing models.
problem Feature importance validity in low-performing machine learning models for biomedical data.
method Experiments with synthetic and real biomedical datasets to compare feature rank stability under different data reductions.
result Feature importance can be maintained even at low performance levels if data size is adequate.
New algorithms select and rank features from MTS without feature extraction.
problem Feature extraction step for MTS classification.
method Directly computes similarity between time series and assesses cluster structure matching labels.
result Techniques match labels well without feature extraction.
In this paper we demonstrate predicting electroencephalograpgy (EEG) features from acoustic features using recurrent neural network (RNN) based regression model and generative adversarial network (GAN). We predict various types of EEG features from acoustic features. We compare our results with the previously studied p…
Pipeline learns topological features for protein stability prediction.
problem Predicting protein stability using topological features.
method Data-driven method to learn topological features, comparing with expert features.
result Topological features achieve 92%-99% of SME-based models' performance.
In text classification, dictionaries can be used to define human-comprehensible features. We propose an improvement to dictionary features called smoothed dictionary features. These features recognize document contexts instead of n-grams. We describe a principled methodology to solicit dictionary features from a teache…
FeAT improves OOD generalization by learning richer features.
problem Improving feature learning for out-of-distribution (OOD) generalization.
method Feature Augmented Training (FeAT) iteratively augments and retains features from different subsets of training data.
result FeAT effectively learns richer features, boosting OOD performance.
The paper defines and analyzes feature complexity in DNNs, proposing metrics for feature disentanglement and evaluation.
problem Understanding and quantifying the complexity of features learned by deep neural networks.
method Proposes a definition and disentanglement of feature complexity orders, introduces metrics for reliability and over-fitting evaluation.
result Establishes a relationship between feature complexity and DNN performance, and proposes a generic mathematical tool for network compression and knowledge distillation.
Conventional mutual information (MI) based feature selection (FS) methods are unable to handle heterogeneous feature subset selection properly because of data format differences or estimation methods of MI between feature subset and class label. A way to solve this problem is feature transformation (FT). In this study,…
A new method for measuring conditional feature importance using generative models.
problem Challenges in evaluating feature importance given other feature values.
method Adversarial Random Forest (ARF) for generating on-manifold data points.
result cARFi method yields robust importance scores adaptable for various feature importance notions.
Existing feature selection methods fail to properly account for interactions between features when evaluating feature subsets. In this paper, we attempt to remedy this issue by using orthogonal variance decomposition to evaluate features. The orthogonality of the decomposition allows us to directly calculate the total …
CAN approximates explicit feature interactions for CTR prediction.
problem Learning explicit feature interactions from sparse features.
method Co-Action Network approximates explicit pairwise feature interactions without introducing too many additional parameters.
result CAN outperforms state-of-the-art CTR models and the cartesian product method.
Proposes a new feature selection method integrating feature relationships.
problem Feature selection in machine learning models.
method Integrates feature-feature and feature-target relationships via penalized mRMR.
result Correctly identifies inactive features, reducing false discoveries.
Online feature selection has been an active research area in recent years. We propose a novel diverse online feature selection method based on Determinantal Point Processes (DPP). Our model aims to provide diverse features which can be composed in either a supervised or unsupervised framework. The framework aims to pro…
New method disentangles feature importance scores in machine learning.
problem Misinterpretation of feature importance scores due to interactions and dependencies.
method Derive DIP (Disentangled Importance) decomposition of feature importance scores.
result DIP decomposition uniquely separates standalone contributions from interactions and dependencies.
Feature selection has been proven a powerful preprocessing step for high-dimensional data analysis. However, most state-of-the-art methods tend to overlook the structural correlation information between pairwise samples, which may encapsulate useful information for refining the performance of feature selection. Moreove…
GRANITE unifies feature-based explanation methods to reduce disagreement.
problem Disagreement among feature-based explanation methods.
method GRANITE partitions feature space into regions minimizing interaction and distribution influences.
result Unified and consistent feature explanations.
Orthogonal random features approximate a Bessel kernel, offering sharper bounds than random Fourier features.
problem Approximating Gaussian kernel efficiently for large datasets.
method Use of Haar orthogonal matrices to construct orthogonal random features and analyze their bias and variance.
result Orthogonal random features approximate a Bessel kernel, not the Gaussian kernel, with sharper bounds.
New methods for assessing and visualizing feature groups in machine learning models.
problem Lack of methods for interpreting feature groups in machine learning models.
method Permutation-based, refitting, and Shapley-based techniques for grouped feature importance. Introduced a sequential procedure for identifying stable feature combinations. Developed a combined features effect plot.
result Effective methods for assessing and visualizing the importance and effect of feature groups in machine learning models.
Learning with streaming data has attracted much attention during the past few years. Though most studies consider data stream with fixed features, in real practice the features may be evolvable. For example, features of data gathered by limited-lifespan sensors will change when these sensors are substituted by new ones…
Supporting evidence for adaptive feature program across diverse models.
problem Analyzing feature learning in neural networks.
method Over-parameterized sequence models and feature error measure (FEM).
result FEM is decreasing during training of adaptive feature models.
Hermite polynomials improve private data generation by reducing feature count.
problem Infinite-dimensional features in kernel mean embedding are impractical for private data generation.
method Replace random features with Hermite polynomial features, leveraging their ordered nature.
result Hermite polynomial features yield a more accurate approximation of kernel mean embedding with fewer features.
Feature selection is a technique to screen out less important features. Many existing supervised feature selection algorithms use redundancy and relevancy as the main criteria to select features. However, feature interaction, potentially a key characteristic in real-world problems, has not received much attention. As a…
Paper defines feature impact and importance from data, not models.
problem Misinterpretation of feature importance as impact leads to flawed insights.
method Mathematical definitions of feature impact and importance derived from partial dependence curves.
result Feature rankings by these definitions are competitive with existing techniques.
Learning with feature evolution studies the scenario where the features of the data streams can evolve, i.e., old features vanish and new features emerge. Its goal is to keep the model always performing well even when the features happen to evolve. To tackle this problem, canonical methods assume that the old features …
NGMs create mirrored features to assess neural network feature importance.
problem Lack of feature relevance information in DNNs limits their applicability.
method Structured perturbation and kernel-based conditional dependence measure for feature importance evaluation.
result Controls feature selection error rate and maintains high selection power with correlated features.
Inf-FS selects features by graph paths, ranking them for infinite feature sets.
problem Feature selection in large datasets with relevance and redundancy.
method Graph-based feature selection with infinite paths, evaluating feature subsets using matrix power series and Markov chains.
result Inf-FS outperforms other methods in various feature selection scenarios.
GADGET framework decomposes global feature effects using recursive partitioning.
problem Misleading global feature effects when feature interactions are present.
method Generalized additive decomposition of global effects (GADGET) based on recursive partitioning.
result Minimizes interaction-related heterogeneity of local feature effects.
Neural networks outperform kernels by learning features better.
problem Current theories of feature learning do not adequately assess feature quality.
method Introduced feature quality metric and examined existing theories empirically.
result Current theories of feature learning do not provide a sufficient foundation for neural network generalization.
We propose a method for finding alternate features missing in the Lasso optimal solution. In ordinary Lasso problem, one global optimum is obtained and the resulting features are interpreted as task-relevant features. However, this can overlook possibly relevant features not selected by the Lasso. With the proposed met…
In this paper, a genetic algorithm-based frequency-domain feature search (GAFDS) method is proposed for the electroencephalogram (EEG) analysis of epilepsy. In this method, frequency-domain features are first searched and then combined with nonlinear features. Subsequently, these features are selected and optimized to …
theft package simplifies feature extraction for time series analysis in R.
problem Lack of a unified access point and methodological pipelines for feature-based time series analysis.
method theft package provides a unified framework for computing features from six open-source time series feature sets.
result theft enables comprehensive quantification and interpretation of time series structure.
New regularization scheme for FMs improves feature interaction selection.
problem Feature selection in FMs leads to loss of feature interactions.
method Proposes a new regularization scheme for FMs with upper bound of ℓ1 regularizer. result Improves feature interaction selection without restricting sparsity patterns.
Proposes a new method to find features affecting treatment effect distribution.
problem Existing methods fail to detect differences in treatment effect distribution parameters other than the mean.
method Formulates and estimates a feature importance measure that quantifies feature influence on potential outcome distribution discrepancies. Develops a feature selection algorithm to control type I error rate.
result Successfully discovers important features and outperforms existing mean-based methods.
FIRES framework selects stable features from online data.
problem Efficiently selecting features in online settings with limited data.
method FIRES framework uses model parameter importance for feature selection.
result FIRES selects stable feature sets with minimal model complexity.