New method distinguishes feature relevance in non-linear contexts.
problem Finding relevant features with preserved redundancies.
method Random forest models and statistical methods.
result Distinguishes strong from weak feature relevance in non-linear problems.
mRMR method selects relevant features for marketing models without redundancy.
problem Selecting relevant features from large feature spaces in machine learning.
method Extends mRMR framework with non-linear redundancy and model-based relevance measures.
result Implemented mRMR method in production for Uber's marketing machine learning platform.
The use of variable selection methods is particularly appealing in statistical problems with functional data. The obvious general criterion for variable selection is to choose the `most representative' or `most relevant' variables. However, it is also clear that a purely relevance-oriented criterion could lead to selec…
The paper tackles feature selection for ordinal regression, considering feature redundancies and privileged information.
problem Discovering relevant factors in ranked data with potentially redundant features and privileged information.
method Develops feature relevance bounds for linear ordinal regression, considering feature redundancies and privileged information.
result Identifies all strongly and weakly relevant features and their type of relevance.
RID framework quantifies and regularizes task-relevant knowledge in distillation.
problem Distilling irrelevant information can hinder student model performance.
method Partial Information Decomposition to quantify and regularize task-relevant knowledge.
result RID framework leads to more resilient distillation under nuisance teachers.
Proposes a new stability measure for model fitting on similar feature data sets.
problem Model fitting on data sets with similar features is challenging.
method Tuning hyperparameters in a multi-criteria fashion with predictive accuracy and feature selection stability.
result Our approach achieves similar or better predictive performance than single-criteria and stability selection approaches.
Feature selection is a technique to screen out less important features. Many existing supervised feature selection algorithms use redundancy and relevancy as the main criteria to select features. However, feature interaction, potentially a key characteristic in real-world problems, has not received much attention. As a…
Data acquisition, storage and management have been improved, while the key factors of many phenomena are not well known. Consequently, irrelevant and redundant features artificially increase the size of datasets, which complicates learning tasks, such as regression. To address this problem, feature selection methods ha…
With the advent of Big Data era, data reduction methods are highly demanded given its ability to simplify huge data, and ease complex learning processes. Concretely, algorithms that are able to filter relevant dimensions from a set of millions are of huge importance. Although effective, these techniques suffer from the…
Transformer models waste resources on long-context tasks.
problem Redundant attention computations in Transformer models for long-context tasks.
method Reformulate sequence modeling as supervised learning, analyze attention sparsity, formulate attention optimization as linear coding problem, propose Dynamic Group Attention.
result DGA reduces computational costs while maintaining performance.
A federated method for feature selection in multi-label data.
problem Feature selection in multi-label data for distributed and federated environments.
method Semi-Supervised Federated Multi-Label Feature Selection (SSFMLFS) using fuzzy information measures.
result SSFMLFS outperforms other methods in feature selection for multi-label data in federated settings.
This paper describes a distributed MapReduce implementation of the minimum Redundancy Maximum Relevance algorithm, a popular feature selection method in bioinformatics and network inference problems. The proposed approach handles both tall/narrow and wide/short datasets. We further provide an open source implementation…
We are working to develop automated intelligent agents, which can act and react as learning machines with minimal human intervention. To accomplish this, an intelligent agent is viewed as a question-asking machine, which is designed by coupling the processes of inference and inquiry to form a model-based learning unit.…
Proposes a new feature selection method integrating feature relationships.
problem Feature selection in machine learning models.
method Integrates feature-feature and feature-target relationships via penalized mRMR.
result Correctly identifies inactive features, reducing false discoveries.
In this paper, we present a framework to control a self-driving car by fusing raw information from RGB images and depth maps. A deep neural network architecture is used for mapping the vision and depth information, respectively, to steering commands. This fusion of information from two sensor sources allows to provide …
We propose a method for finding alternate features missing in the Lasso optimal solution. In ordinary Lasso problem, one global optimum is obtained and the resulting features are interpreted as task-relevant features. However, this can overlook possibly relevant features not selected by the Lasso. With the proposed met…
Most existing feature selection methods are insufficient for analytic purposes as soon as high dimensional data or redundant sensor signals are dealt with since features can be selected due to spurious effects or correlations rather than causal effects. To support the finding of causal features in biomedical experiment…
AMBER method selects features efficiently using autoencoders and model-based elimination.
problem Efficiently selecting relevant features for classification.
method Greedy backward elimination using a ranker model and autoencoders.
result AMBER outperforms other feature selection methods in classification accuracy.
A new graph neural network (NBA-GNN) avoids revisiting nodes to improve accuracy.
problem Redundancy in graph neural network updates causes over-squashing and inaccurate recognition.
method Proposes non-backtracking graph neural networks (NBA-GNN) that update messages without revisiting nodes.
result The NBA-GNN alleviates over-squashing and improves performance on graph benchmarks.
In this work we present a review of the state of the art of information theoretic feature selection methods. The concepts of feature relevance, redundance and complementarity (synergy) are clearly defined, as well as Markov blanket. The problem of optimal feature selection is defined. A unifying theoretical framework i…
Every day, hundreds of millions of new Tweets containing over 40 languages of ever-shifting vernacular flow through Twitter. Models that attempt to extract insight from this firehose of information must face the torrential covariate shift that is endemic to the Twitter platform. While regularly-retrained algorithms can…
Inf-FS selects features by graph paths, ranking them for infinite feature sets.
problem Feature selection in large datasets with relevance and redundancy.
method Graph-based feature selection with infinite paths, evaluating feature subsets using matrix power series and Markov chains.
result Inf-FS outperforms other methods in various feature selection scenarios.
We propose a feature selection method that finds non-redundant features from a large and high-dimensional data in nonlinear way. Specifically, we propose a nonlinear extension of the non-negative least-angle regression (LARS) called N3LARS, where the similarity between input and output is measured through the norm…
Feature selection has been proven a powerful preprocessing step for high-dimensional data analysis. However, most state-of-the-art methods tend to overlook the structural correlation information between pairwise samples, which may encapsulate useful information for refining the performance of feature selection. Moreove…
Williams and Beer (2010) proposed a nonnegative mutual information decomposition, based on the construction of redundancy lattices, which allows separating the information that a set of variables contains about a target variable into nonnegative components interpretable as the unique information of some variables not p…
The paper introduces a method to explain redundancy in deep CNNs using unit impulse response.
problem Redundancy in deep CNNs leads to unnecessary computations and increased cost.
method Empirical demonstration and unit impulse response analysis to identify and quantify redundancy across layers and depth.
result Identifies and quantifies redundancy in deep CNNs, providing better insights into their internal dynamics.
New method quantifies redundant information using information bottleneck.
problem Quantifying redundant information among multiple sources.
method Formulated as an information bottleneck problem, termed redundancy bottleneck.
result Extracts information that best predicts the target without revealing source identity.
Redundancy improves learning stability and generalization in structured systems.
problem Understanding redundancy in structured systems for learning and generalization.
method Developed a theoretical framework that redefines redundancy as a geometric principle unifying various measures.
result Redundancy balances structure and coupling, leading to optimal stability and generalization.
Transformers reduce redundancy by focusing on invariant relational quantities.
problem Substantial internal redundancy in Transformer models due to coordinate-dependent representations and continuous symmetries.
method Reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction.
result Architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization.
New framework improves multivariate time series forecasting by minimizing redundant information.
problem Improving multivariate time series forecasting with deep learning techniques.
method Cross-variable Decorrelation Aware feature Modeling (CDAM) and Temporal correlation Aware Modeling (TAM) to refine Channel-mixing and exploit temporal correlations.
result Significantly surpasses existing models in comprehensive tests.
Paper proposes redundancy-free features for zero-shot object recognition.
problem Redundant visual features degrade zero-shot object recognition.
method Project original features into a new, statistically independent space.
result RFF-GZSL achieves competitive results on benchmark datasets.
This work explains scaling laws as redundancy laws in deep learning.
problem The mathematical origins of scaling laws in deep learning models remain unclear.
method Kernel regression and analysis of data covariance spectra.
result Scaling laws can be explained as redundancy laws, revealing the learning curve's slope depends on data redundancy.
Enhash detects concept drift in data streams quickly and efficiently.
problem Detecting abrupt, gradual, virtual, or recurring events in data streams.
method Uses projection hash to insert incoming samples and detects concept drift.
result Enhash has competitive performance and moderate resource requirements compared to existing ensemble learners.
Develops a new feature selection method using deep-learning saliency.
problem Lack of instance-level feature selection information.
method Saliency-based Feature Selection (SFS) method.
result SFS method provides instance-level feature selection information.
Improved fault diagnosis for bearings using mRMR and transfer learning.
problem Challenges in forming large-scale annotated datasets for machine fault diagnosis.
method Combining mRMR with deep learning and transfer learning.
result Improved fault diagnostics performance in terms of accuracy and computational complexity.
Study on neural networks to identify redundancy issues in safe machine learning.
problem Identifying redundancy in neural network architectures for safe machine learning.
method Experiments with MNIST database using neural network classifiers.
result Underlines difficulties in using neural network classifiers for safe systems.
A new approach of solving the ill-conditioned inverse problem for analytical continuation is proposed. The root of the problem lies in the fact that even tiny noise of imaginary-time input data has a serious impact on the inferred real-frequency spectra. By means of a modern regularization technique, we eliminate redun…
Our interest in this paper is in the construction of symbolic explanations for predictions made by a deep neural network. We will focus attention on deep relational machines (DRMs, first proposed by H. Lodhi). A DRM is a deep network in which the input layer consists of Boolean-valued functions (features) that are defi…
We simplify SSL by approximating redundant structural components with low-rank factorization.
problem Improving self-supervised learning performance with limited labeled data.
method Low-rank approximation of structural redundancy, introducing ε_s to measure approximation quality.
result The proposed method enhances SSL performance, as shown by theoretical and experimental validations.
Redundancy in deep neural network (DNN) models has always been one of their most intriguing and important properties. DNNs have been shown to overparameterize, or extract a lot of redundant features. In this work, we explore the impact of size (both width and depth), activation function, and weight initialization on th…
This paper explores how optimizing data access and reducing redundancy can improve machine learning algorithm performance.
problem Performance issues in machine learning algorithms due to data locality and redundancy.
method Analysis of data access patterns and computational redundancy in machine learning algorithms, identifying opportunities for reuse and experimentation.
result Initial indicative results show potential for improving performance through data access optimization and reuse of computation results.
This paper introduces a new measure to identify model redundancy in compressed CNNs.
problem Identifying remaining model redundancy in compressed CNNs.
method Developed a statistical formulation of CNNs and compressed CNNs via tensor decomposition, revealing discrepancies in sample complexity and model redundancy.
result Introduced a new model redundancy measure, the K/R ratio, for compressed CNNs. Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
problem Why randomly trained neural networks generalize well despite interpolating training data.
method Examined a random neural network that interpolates training data and showed it generalizes well if there's a simpler underlying teacher model.
result Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
This paper analyzes self-supervised learning from a multi-view perspective.
problem Understanding and optimizing self-supervised learning from multi-view data.
method Information-theoretical framework to understand and design self-supervised learning objectives.
result Self-supervised representations can extract task-relevant information and discard task-irrelevant information.
Optimized GPRNN reduces model complexity and overfitting, improving performance.
problem Overfitting in neural networks and high model complexity.
method Gaussian Process Regression - Neural Network hybrid with optimized redundant coordinates.
result Optimized GPRNN achieves lower test set error with fewer terms/neurons.
One of Powell's generators is not necessary.
problem Unresolved conjecture about generating Goeritz group.
method Short argument showing redundancy of one generator.
result One of Powell's generators is a consequence of others.
The study analyzes optimization trajectories in neural networks to reveal redundancy and redundancy-reducing strategies.
problem Understanding the directional structure and redundancy in neural network optimization.
method Introducing natural notions of complexity for optimization trajectories and analyzing their directional nature.
result Training only scalar batchnorm parameters can match the performance of training the entire network, indicating potential for hybrid optimization schemes.
This work introduces RISE to explain LLMs more reliably by distinguishing essential context.
problem Identifying which context elements influence LLM outputs reliably.
method RISE (Redundancy-Insensitive Scoring of Explanation) method.
result RISE provides more robust explanations than traditional methods.