Bayesian model averaging under predictor redundancy
problem Reporting Bayesian model averaging posterior without changing the Bayesian target
method Using hard or soft regions of support space
result Region reports often give shorter and clearer summaries while preserving the main posterior information
The Partial Information Decomposition (PID) [arXiv:1004.2515] provides a theoretical framework to characterize and quantify the structure of multivariate information sharing. A new method (Idep) has recently been proposed for computing a two-predictor PID over discrete spaces. [arXiv:1709.06653] A lattice of maximum en…
Reduces deep learning model training dataset size by 90%.
problem Redundant information from large training datasets.
method Dynamic instance selection based on CNN predictions.
result Maintained predictive power with reduced dataset size.
A hybrid method combines GA and EN for feature selection in high-dimensional datasets.
problem Feature selection in high-dimensional datasets with high prediction error and computational inefficiency.
method Hybrid two-layer approach using Genetic Algorithm and Elastic Net.
result The hybrid method improves prediction accuracy and reduces computational time.
We develop an online learning method for prediction, which is important in problems with large and/or streaming data sets. We formulate the learning approach using a covariance-fitting methodology, and show that the resulting predictor has desirable computational and distribution-free properties: It is implemented onli…
Williams and Beer (2010) proposed a nonnegative mutual information decomposition, based on the construction of redundancy lattices, which allows separating the information that a set of variables contains about a target variable into nonnegative components interpretable as the unique information of some variables not p…
New method quantifies redundant information using information bottleneck.
problem Quantifying redundant information among multiple sources.
method Formulated as an information bottleneck problem, termed redundancy bottleneck.
result Extracts information that best predicts the target without revealing source identity.
Redundancy improves learning stability and generalization in structured systems.
problem Understanding redundancy in structured systems for learning and generalization.
method Developed a theoretical framework that redefines redundancy as a geometric principle unifying various measures.
result Redundancy balances structure and coupling, leading to optimal stability and generalization.
Transformers reduce redundancy by focusing on invariant relational quantities.
problem Substantial internal redundancy in Transformer models due to coordinate-dependent representations and continuous symmetries.
method Reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction.
result Architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization.
Convolutional neural networks (CNN) are generally designed with a heuristic initialization of network architecture and trained for a certain task. This often leads to overparametrization after learning and induces redundancy in the information flow paths within the network. This robustness and reliability is at the inc…
Paper proposes redundancy-free features for zero-shot object recognition.
problem Redundant visual features degrade zero-shot object recognition.
method Project original features into a new, statistically independent space.
result RFF-GZSL achieves competitive results on benchmark datasets.
This work explains scaling laws as redundancy laws in deep learning.
problem The mathematical origins of scaling laws in deep learning models remain unclear.
method Kernel regression and analysis of data covariance spectra.
result Scaling laws can be explained as redundancy laws, revealing the learning curve's slope depends on data redundancy.
Several multiscale methods account for sub-grid scale features using coarse scale basis functions. For example, in the Multiscale Finite Volume method the coarse scale basis functions are obtained by solving a set of local problems over dual-grid cells. We introduce a data-driven approach for the estimation of these co…
Study on neural networks to identify redundancy issues in safe machine learning.
problem Identifying redundancy in neural network architectures for safe machine learning.
method Experiments with MNIST database using neural network classifiers.
result Underlines difficulties in using neural network classifiers for safe systems.
Exclusive Lasso improves survival prediction in cancer datasets.
problem Enhanced survival prediction in cancer datasets with high-dimensional genomic and clinical data.
method Proposes Exclusive Lasso regularization for feature selection in Cox regression models for grouped variables.
result Demonstrates improved survival prediction performance using Exclusive Lasso compared to standard Cox regression.
Logistic regression with wavelets achieves bacterial infection detection accuracy.
problem Interpreting complex biomedical signal models for high-stakes decisions.
method Wavelet features and knockoff variables for feature selection.
result Logistic regression outperforms neural networks in bacterial infection detection.
Financial forecasting using news articles is an emerging field. In this paper, we proposed hybrid intelligent models for stock market prediction using the psycholinguistic variables (LIWC and TAALES) extracted from news articles as predictor variables. For prediction purpose, we employed various intelligent techniques …
We simplify SSL by approximating redundant structural components with low-rank factorization.
problem Improving self-supervised learning performance with limited labeled data.
method Low-rank approximation of structural redundancy, introducing ε_s to measure approximation quality.
result The proposed method enhances SSL performance, as shown by theoretical and experimental validations.
Redundancy in deep neural network (DNN) models has always been one of their most intriguing and important properties. DNNs have been shown to overparameterize, or extract a lot of redundant features. In this work, we explore the impact of size (both width and depth), activation function, and weight initialization on th…
This paper introduces a new measure to identify model redundancy in compressed CNNs.
problem Identifying remaining model redundancy in compressed CNNs.
method Developed a statistical formulation of CNNs and compressed CNNs via tensor decomposition, revealing discrepancies in sample complexity and model redundancy.
result Introduced a new model redundancy measure, the K/R ratio, for compressed CNNs. Optimized GPRNN reduces model complexity and overfitting, improving performance.
problem Overfitting in neural networks and high model complexity.
method Gaussian Process Regression - Neural Network hybrid with optimized redundant coordinates.
result Optimized GPRNN achieves lower test set error with fewer terms/neurons.
The study analyzes optimization trajectories in neural networks to reveal redundancy and redundancy-reducing strategies.
problem Understanding the directional structure and redundancy in neural network optimization.
method Introducing natural notions of complexity for optimization trajectories and analyzing their directional nature.
result Training only scalar batchnorm parameters can match the performance of training the entire network, indicating potential for hybrid optimization schemes.
Contrastive learning works well with redundant data views.
problem Improving unsupervised learning with redundant data views.
method Theoretical analysis of contrastive learning in a multi-view setting.
result Linear functions of learned representations are nearly optimal on prediction tasks when views are redundant.
This work introduces RISE to explain LLMs more reliably by distinguishing essential context.
problem Identifying which context elements influence LLM outputs reliably.
method RISE (Redundancy-Insensitive Scoring of Explanation) method.
result RISE provides more robust explanations than traditional methods.
This work identifies redundant tests in conditional-independence-based discovery that can improve graphical model accuracy.
problem Reliability and sensitivity of conditional-independence-based discovery algorithms.
method Analysis of redundant tests and their impact on error detection and correction.
result Redundant tests can improve graphical model accuracy but not all are beneficial.
New method distinguishes feature relevance in non-linear contexts.
problem Finding relevant features with preserved redundancies.
method Random forest models and statistical methods.
result Distinguishes strong from weak feature relevance in non-linear problems.
Paper compresses deep neural networks by eliminating redundant neurons.
problem Challenges in deploying deep learning models due to high parameter count.
method Exploits non-linear redundancy to compress neural networks without loss.
result Reduces network size by up to 99% with minimal performance loss.
Large datasets have been crucial to the success of deep learning models in the recent years, which keep performing better as they are trained with more labelled data. While there have been sustained efforts to make these models more data-efficient, the potential benefit of understanding the data itself, is largely unta…
FactorMiner discovers financial alpha factors with low redundancy.
problem Finding novel financial alpha factors in a vast search space.
method Modular Skill Architecture and Experience Memory to distill and guide exploration.
result FactorMiner constructs a diverse library of high-quality factors with competitive performance.
Wide neural networks' last hidden layers split into groups of redundant neurons.
problem Understanding why wide neural networks generalize well despite overfitting.
method Analyzed the last hidden layer representations of various convolutional neural networks.
result Wide hidden layers split into groups of redundant neurons, which help generalize.
Spectral dimensionality reduction algorithms are widely used in numerous domains, including for recognition, segmentation, tracking and visualization. However, despite their popularity, these algorithms suffer from a major limitation known as the "repeated Eigen-directions" phenomenon. That is, many of the embedding co…
This paper enhances ML algorithms by improving data locality and reducing redundancy.
problem Improving performance of machine learning algorithms with complex data.
method Exploiting data locality and reuse in memory hierarchies of modern processors.
result Efficient implementation of machine learning algorithms can be achieved by reusing computation results.
Machine learning (ML) is probably the first and foremost used technique to deal with the size and complexity of the new generation of data. In this paper, we analyze one of the means to increase the performances of ML algorithms which is exploiting data locality. Data locality and access patterns are often at the heart…
In 1980 J. Powell proposed that five specific elements sufficed to generate the Goeritz group of any Heegaard splitting of S3. This conjecture remains unresolved for genus g≥4. Here a short argument shows that one of his proposed generators is redundant, in fact a consequence of three of the other four.
An accurate load forecasting has always been one of the main indispensable parts in the operation and planning of power systems. Among different time horizons of forecasting, while short-term load forecasting (STLF) and long-term load forecasting (LTLF) have respectively got benefits of accurate predictors and probabil…
Motivation: Radiomics refers to the high-throughput mining of quantitative features from radiographic images. It is a promising field in that it may provide a non-invasive solution for screening and classification. Standard machine learning classification and feature selection techniques, however, tend to display infer…
Cell-based NAS search spaces are redundant and lack novelty.
problem Redundancy and lack of novelty in cell-based neural architecture search.
method Empirical analysis of architectures from popular cell-based search spaces.
result Randomly sampled architectures can match or outperform state-of-the-art results.
Redundancy in AI perception systems doesn't guarantee independent error occurrences.
problem Lack of direct statistical evidence of super-human automated driving performance.
method Investigated the effectiveness of redundancy in neural networks for independent error occurrences.
result Errors in neural networks for computer vision tasks are correlated, not independent.
Estimates interactions between modalities for multimodal data.
problem Accurately quantifying interactions between different data types.
method Developed Lightweight Sample-wise Multimodal Interaction (LSMI) estimator using pointwise information theory.
result LSMI reveals fine-grained dynamics in multimodal data.
Signed pairwise interactions conflate uniqueness, redundancy, and synergy
problem Signed pairwise interactions conflate uniqueness, redundancy, and synergy
method Stochastic Hi-Fi
result Stochastic Hi-Fi recovers structure missed by scalar baselines
Efficient algorithm removes redundant nodes and obsolete samples in machine learning.
problem Pruning redundant nodes and removing obsolete training samples in machine learning.
method Deduced decremented learning algorithms from incremental learning algorithms, using inverse Cholesterol factor and unitary transformation.
result Proposed decremented learning algorithms efficiently prune redundant nodes and remove obsolete training samples.
PAC-Bayesian bounds improve understanding of K-NN classifier performance.
problem Improving the understanding of K-NN classifier's generalization error.
method PAC-Bayesian analysis applied to K-NN classifier in kernel space.
result PAC-Bayesian bounds provide a function of the number of redundant training examples.
New CR representations are found and shown to be redundant.
problem Identifying and classifying CR representations of 3-manifolds.
method Experimental computation of limit sets and exact computations of triangle groups.
result Many CR representations are redundant and conjugate.
New method reduces overfitting in deep neural networks by measuring and regulating hidden unit diversity.
problem Overfitting in deep neural networks.
method Introduces a new redundancy measure based on mutual information to improve generalization.
result Reduction of redundancy improves generalization capacity, reducing overfitting.
PoWER-BERT speeds up BERT inference by eliminating redundant word-vectors.
problem Improving BERT inference speed without sacrificing accuracy.
method Eliminating redundant word-vectors using a self-attention-based significance measure and learning the number of vectors to eliminate.
result Up to 4.5x reduction in inference time with <1% loss in accuracy on GLUE benchmark.
Neural networks learn task-specific features, influenced by nonlinearity.
problem Understanding the nature of task-dependent feature learning in neural networks.
method Investigation of fully-connected, wide neural networks using Bayesian framework.
result The nature of internal representations depends on neuronal nonlinearity, leading to analog, redundant, or sparse coding schemes.
Optimal algorithms for mixable losses in dynamic environments with reduced redundancy.
problem Online optimization of mixable loss functions in a dynamic environment.
method Introduce online mixture schemes with polynomial and logarithmic time complexities.
result Achieves optimal redundancy up to a constant multiplicity gap.
Reducing the latency variance in machine learning inference is a key requirement in many applications. Variance is harder to control in a cloud deployment in the presence of stragglers. In spite of this challenge, inference is increasingly being done in the cloud, due to the advent of affordable machine learning as a s…