The paper defines and analyzes feature complexity in DNNs, proposing metrics for feature disentanglement and evaluation.
problem Understanding and quantifying the complexity of features learned by deep neural networks.
method Proposes a definition and disentanglement of feature complexity orders, introduces metrics for reliability and over-fitting evaluation.
result Establishes a relationship between feature complexity and DNN performance, and proposes a generic mathematical tool for network compression and knowledge distillation.
Deep nets outperform shallow nets in complex feature realization.
problem Realizing complex data features with deep nets.
method Refined covering number estimates and analysis of approximation rates.
result Deep nets can improve performance without additional capacity costs for complex features.
Paper explores deep learning features for complex emotion recognition.
problem Improving emotion recognition accuracy in complex emotions.
method Used pretrained networks (AudioSet Net, VoxCeleb Net, Deep Speech Net) and their deep layer features for emotion recognition.
result Achieved highest F1 score of 0.85 on EmoReact dataset.
FS-EE selects necessary features for factored MDPs efficiently.
problem Efficiently selecting relevant features in complex state spaces.
method FS-EE algorithm that selects necessary features while learning Factored MDPs.
result FS-EE's sample complexity depends on necessary feature in-degree, not all features.
CRBM extracts speech features from complex spectra directly.
problem Speech coding ignores phase information in complex spectra.
method CRBM learns relationships between visible and hidden units from complex-valued spectra.
result CRBM outperforms conventional methods in speech coding.
Study analyzes feedback complexity for sparse feature retrieval in deep networks.
problem Learning sparse superposed features with feedback.
method Analysis of feedback complexity in sparse settings, including triplet comparisons.
result Establishes tight bounds and strong upper bounds for feature recovery.
Spectral simplicial theory improves feature selection for complex data.
problem Complex data sets and high-dimensional feature spaces require efficient feature selection methods.
method Extends spectral techniques to abstract simplicial complexes, incorporating topological data analysis.
result Spectral simplicial methods provide a unified approach for feature selection in multi-modal genomic data.
This paper reviews Relief-based feature selection algorithms.
problem Complex biomedical data with high feature dimensions.
method Relief-based algorithms (RBAs) for efficient feature selection.
result RBAs strike a balance between computational efficiency and sensitivity to complex patterns.
Regularization effect found in neural feature alignment.
problem Implicit regularization in deep learning models.
method Geometrical viewpoint and analysis of Rademacher complexity.
result Neural features align along task-relevant directions, leading to regularization.
Three-layer networks learn more complex features than two-layer networks.
problem Understanding feature learning in deep neural networks.
method Analysis of three-layer neural networks trained with gradient descent.
result Three-layer networks can learn functions that two-layer networks cannot.
New algorithm reduces kernel PCA to O ( n ) O(\sqrt{n}) O ( n ) features for streaming data.
problem Efficiently performing kernel PCA on large datasets.
method Random Fourier features and Oja's algorithm for streaming.
result Achieves O ( 1 / ε 2 ) O(1/ε^2) O ( 1/ ε 2 ) sample complexity with O ( n ) O(\sqrt{n}) O ( n ) features. New method quantifies model complexity for better interpretation.
problem Complex models produce misleading interpretation results.
method Functional decomposition to quantify model complexity.
result Post-hoc interpretation of complex models is more reliable and compact.
Improves efficiency of random feature approximations for dot product kernels.
problem Efficiency of random feature approximations for dot product kernels.
method Generalization of existing random feature approximations using complex-valued random features, theoretical analysis of variances, data-driven optimization approach.
result Complex-valued random features can significantly reduce the variances of approximations.
A new method simplifies feature explanation for complex models.
problem Explain predictions from complex models efficiently with many features.
method groupShapley: groups features for Shapley value computation.
result Equivalent to summing feature-wise Shapley values within groups.
This research simplifies computation of feature attribution methods under certain conditions.
problem Computational complexity of feature attribution methods, especially power indices.
method Identifying conditions for polynomial computation and introducing new indices.
result Conditions for efficient computation of feature attribution methods are identified.
Proposes MFI for non-linear learning to explain complex feature interactions.
problem Lack of straightforward interpretation for non-linear learning models.
method Introduces Measure of Feature Importance (MFI) for any learning machine.
result MFI detects and explains complex feature interactions.
Quantum SVM uses fewer features for faster training.
problem Training high-dimensional SVMs efficiently.
method Quantum linear programming for sparse SVM training.
result Quantum sparse SVM can be trained in sublinear time.
Many machine learning problems, especially multi-modal learning problems, have two sets of distinct features (e.g., image and text features in news story classification, or neuroimaging data and neurocognitive data in cognitive science research). This paper addresses the joint dimensionality reduction of two feature ve…
New neural network method hides input information in complex-valued features to protect privacy.
problem Preventing adversaries from inferring input attributes from neural network features.
method Transforming real-valued features into complex-valued ones, making input hidden in a randomized phase.
result Significantly diminishes adversary's ability to infer input while preserving high accuracy.
Defines complexity measure for neural networks and feature representations, revealing scaling patterns.
problem Understanding the nonlinearity and dimensionality of neural network computations and feature representations.
method Introduces complexity and effective dimension measures, investigates their dynamics during training, and analyzes their scaling properties.
result Power law scaling of complexity and effective dimension during training, revealing hidden structure of datasets.
Model predicts severity of traffic accidents using spatial and temporal features.
problem Estimating severity of traffic accidents in aggregated and disaggregated data.
method Gradient Boosting models and Gaussian Processes for inference and feature importance.
result Complexity of road networks and other situational features significantly impact accident severity.
Neural networks favor simple features over complex ones, even when complex features are available.
problem Neural networks exhibit a bias towards simple features over complex ones, even when complex features are present.
method Rigorously defined simplicity bias, theoretical and empirical demonstrations, ensemble approach to improve robustness.
result One hidden layer neural networks favor simple features over complex ones, even in the presence of more robust features.
As machine learning is applied to an increasing variety of complex problems, which are defined by high dimensional and complex data sets, the necessity for task oriented feature learning grows in importance. With the advancement of Deep Learning algorithms, various successful feature learning techniques have evolved. I…
Introduces top- k k k regularization for better feature selection in machine learning.
problem Limited ability of existing feature selection methods to reconcile feature representativeness and inter-correlations.
method Top- k k k regularization, which induces a sub-architecture on the model's architecture to select informative features and model complex relationships. result Uniform approximation error bound for top- k k k regularization approximating high-dimensional sparse functions. The paper proposes criteria and methods for evaluating and aggregating feature-based model explanations.
problem Lack of quantitative evaluation criteria for feature-based model explanations.
method Developed quantitative evaluation criteria (low sensitivity, high faithfulness, low complexity), devised a framework for aggregation, and derived a new aggregate Shapley value explanation function.
result A new aggregate Shapley value explanation function that minimizes sensitivity.
New algorithm ranks items with features using fewer comparisons.
problem Ranking items from comparisons with associated features.
method f-BTL model and fBTL-LS algorithm.
result Requires O ( α log α ) O(α\log α) O ( α log α ) samples, where α α α is number of independent items. Identifies important features and their resolution for complex models.
problem Characterizing complex learned models' decision-making across instance distributions.
method Model-agnostic approach using hypothesis testing and feature groups.
result Determines important features and their resolution levels for model accuracy.
Automated feature engineering improves interpretable models without manual work.
problem Lack of interpretability in complex models causes trust and stability issues.
method Use elastic black-box models to create simpler, interpretable glass-box models.
result Extracted features from complex models improve linear model performance.
Two new algorithms improve feature importance scoring for graph-structured data.
problem Efficiently scoring feature importance for structured data.
method Developed two linear complexity algorithms for instancewise feature importance scoring.
result Our methods compare favorably with other feature importance scoring methods.
Complexity helps identify sparse risk factors in asset pricing.
problem Tension between feature richness and economic parsimony in high-dimensional asset pricing.
method Expanding feature space and using basis pursuit to discover sparse risk factors.
result Nonlinear feature expansions combined with basis pursuit yield superior out-of-sample performance.
FIRES framework selects stable features from online data.
problem Efficiently selecting features in online settings with limited data.
method FIRES framework uses model parameter importance for feature selection.
result FIRES selects stable feature sets with minimal model complexity.
This study examines a single attention layer's capabilities using random features.
problem Understanding the learning and generalization of a single multi-head attention layer.
method Random feature setting with large number of heads, frozen query and key matrices, and trainable value matrices.
result Random-feature attention layer can express a broad class of permutation-invariant target functions.
DiffKnock improves feature selection in neural networks with complex dependencies and non-linear associations.
problem Selecting important features in neural networks with complex dependencies and non-linear associations.
method DiffKnock uses diffusion models to generate knockoffs and neural network statistics to measure feature importance.
result DiffKnock outperforms existing methods in detecting non-linear associations and preserving feature dependencies.
Develops a method to audit indirect feature influence in complex models.
problem Auditing indirect feature influence in complex, black-box models.
method Disentangled influence audits using disentangled representations.
result Can detect proxy features and show which ones affect model outcomes most.
Feature networks link ML features via graph structure for enhanced learning.
problem Enhancing feature expressiveness and learning efficiency in machine learning.
method Graph representation of feature vectors, leveraging Fourier and functional analysis.
result Feature networks enable novel, complex feature dependencies.
Bayesian TNKMs automatically infer model complexity and feature relevance.
problem Manual tuning of TN rank and feature dimensions is error-prone and computationally expensive.
method Bayesian approach with hierarchical priors on TN factors for automatic rank and feature selection.
result Superior performance in prediction accuracy, uncertainty quantification, interpretability, and scalability.
Improves local learning models for complex feature extraction.
problem Limited use of simple model families in local learning.
method Uses complex local model families to extract features.
result Demonstrates applications in various fields.
Nature-inspired algorithms improve data analytics efficiency.
problem Efficient data analytics with reduced dimensionality.
method Nature-inspired algorithms for feature selection optimization.
result Nature-inspired algorithms enhance data analytics efficiency.
New bounds on ReLU networks for low-regular functions.
problem Bounding approximation error for ReLU networks on low-regular functions.
method Complexity analysis of Fourier features residual networks to ReLU networks.
result Approximation error bound proportional to target function norm and inversely proportional to network width and depth.
OOMP selects features online for sparse linear regression.
problem Feature selection in high-dimensional sparse linear models.
method Online algorithm that alternates between feature selection and coefficient estimation.
result Theoretical guarantees and computational complexity analysis of OOMP.
Sparse group matrix completion reduces complexity and improves performance.
problem Matrix completion with non-informative side features.
method Group-Lasso regularization for feature selection in matrix factorization.
result Theoretical sample complexity is significantly lower than competitors.
Scaling feature values is an important step in numerous machine learning tasks. Different features can have different value ranges and some form of a feature scaling is often required in order to learn an accurate classifier. However, feature scaling is conducted as a preprocessing task prior to learning. This is probl…
Developed a Particle-Gibbs sampler for Bayesian feature allocation models.
problem Intractable exact inference in Bayesian feature allocation models.
method Particle-Gibbs sampler for feature allocation matrix updates.
result PG sampler improves performance of feature allocation models.
Improved explanation of complex models with dependent features using Kernel SHAP.
problem Accurate explanation of machine learning predictions when features are dependent.
method Extended Kernel SHAP method to handle dependent features, providing more accurate approximations to Shapley values.
result Our method gives more accurate approximations to true Shapley values in models with dependent features.
Featurization improves density ratio estimation for complex data.
problem Difficulty in estimating density ratios for high-dimensional, different distributions.
method Invertible generative model to map distributions into a common feature space.
result Improved accuracy in density ratio estimation through feature space.
ALT improves TSC by capturing complex patterns in time series data.
problem Challenges in traditional TSC methods with time series complexity and variability.
method ALT incorporates variable-length shifted time windows to enhance LLT for better feature representation.
result ALT achieves state-of-the-art performance with few hyperparameters.
DNN2LR bridges DNN power and LR interpretability.
problem Combining DNN power and LR interpretability for real-world tabular data.
method Automatic feature crossing method based on DNN interpretation inconsistencies.
result DNN2LR outperforms complex DNN models and feature crossing methods.
New algorithm for linear bandits learns from feature feedback, reducing regret.
problem Linear bandit problem with feature feedback.
method Developed new theory and algorithms for linear bandits with feature feedback.
result Achieves regret scaling like k T k\sqrt{T} k T , improving over traditional linear bandits.