Efficient algorithm removes redundant nodes and obsolete samples in machine learning.
problem Pruning redundant nodes and removing obsolete training samples in machine learning.
method Deduced decremented learning algorithms from incremental learning algorithms, using inverse Cholesterol factor and unitary transformation.
result Proposed decremented learning algorithms efficiently prune redundant nodes and remove obsolete training samples.
BRIEF reduces CNN models by 32.3% on ImageNet, removing redundant channels.
problem Redundant neural channels in CNN models.
method Backward reduction algorithm based on information flow analysis.
result Significant model reduction (32.3%) on ResNet-34 in ImageNet scale.
Previously in 2014, we proposed the Nearest Descent (ND) method, capable of generating an efficient Graph, called the in-tree (IT). Due to some beautiful and effective features, this IT structure proves well suited for data clustering. Although there exist some redundant edges in IT, they usually have salient features …
In our previous works, we proposed a physically-inspired rule to organize the data points into an in-tree (IT) structure, in which some undesired edges are allowed to occur. By removing those undesired or redundant edges, this IT structure is divided into several separate parts, each representing one cluster. In this w…
Improved Bayesian neural network inference by selectively removing redundant modes.
problem Redundant modes in Bayesian neural network posteriors complicate approximate inference.
method Structured partial stochasticity and deterministic subset selection of weights.
result Improved performance of approximate inference schemes with simplified posterior distribution.
Centripetal SGD prunes deep CNNs by making filters collapse.
problem Pruning deep CNNs with complex structures.
method Centripetal SGD, a novel optimization method.
result Pruning deep CNNs without performance loss.
Novel channel pruning method accelerates deep CNNs by removing redundant, similar features.
problem Limitations of existing magnitude-based pruning algorithms in similar magnitude cases.
method Hierarchical clustering of channels based on probabilistic similarity metrics, without sparsity training.
result 30% reduction in FLOPs for pruned ResNet-50 on ImageNet, outperforming baseline.
Paper proposes redundancy-free features for zero-shot object recognition.
problem Redundant visual features degrade zero-shot object recognition.
method Project original features into a new, statistically independent space.
result RFF-GZSL achieves competitive results on benchmark datasets.
Gradient boosts monomial-order-free basis construction algorithms.
problem Lack of theoretical properties in monomial-order-free basis construction algorithms.
method Exploits gradient to sidestep spurious vanishing, achieve consistent output, and remove redundant bases.
result Proposes methods that equip monomial-order-free algorithms with theoretical properties.
We give a simplified proof of J. A. Wolf's classification of finite groups that can act freely and isometrically on a round sphere of some dimension. We slightly improve the classification by removing some non-obvious redundancy. The groups are the same as the Frobenius complements of finite group theory.
Network deconvolution removes redundant data to improve neural network performance.
problem Redundant data in neural networks makes training challenging.
method Network deconvolution optimally removes pixel-wise and channel-wise correlations before each layer.
result Network deconvolution improves performance across various datasets.
Prunes CNNs by removing redundant filters with provable guarantees.
problem Redundant filters in over-parameterized neural networks.
method Sampling-based approach using saliency scores and importance sampling.
result Consistently generates sparser and more efficient models.
This paper finds a new way to compress CNN weights, improving on pruning and quantization.
problem Improving performance and storage efficiency of CNNs.
method Identifying and exploiting repeated patterns in CNN weight tensors, using Huffman coding and block sparse matrix formats.
result Achieved compaction ratios of 1.4x to 3.1x in addition to pruning and quantization.
This work proposes efficient unit pruning for neural networks to reduce model size.
problem Redundant parameters in neural networks can be pruned without performance loss.
method Proposes unit-wise pruning over parameter pruning, introduces saliency scores, and defines dead units.
result 5x model size reduction on MNIST using unit-wise pruning.
Transformer-MGK replaces redundant heads with Gaussian key mixtures, improving efficiency and performance.
problem Redundant attention heads in transformers degrade performance and efficiency.
method Transformer-MGK replaces redundant heads with a mixture of Gaussian keys.
result Transformer-MGK accelerates training and inference, reduces parameters and FLOPs, and achieves comparable or better accuracy.
Stage-based hyper-parameter optimization reduces GPU-hours and training time.
problem Efficiently executing hyper-parameter optimization for deep learning models.
method Stage-based execution strategy to remove redundant computations.
result Stage-based execution outperforms trial-based method by up to 6.60 times in GPU-hours and 4.13 times in training time.
A new method reduces redundancy in multimodal data for improved inference.
problem Understanding and optimizing the contribution of each modality in multimodal tasks.
method Modality-based Tensor Factorization (MRRF) for multimodal fusion.
result Improves multimodal inference tasks by 1% to 4% compared to state-of-the-art.
This work compares data reduction criteria for online Gaussian Processes.
problem The computational complexity of Gaussian Processes limits their applicability to small datasets and streaming scenarios.
method Unified comparison of several data reduction criteria, analyzing computational complexity and reduction behavior.
result Practical guidelines for choosing a suitable data reduction criterion for online Gaussian Processes.
In this paper we introduce a new feature selection algorithm to remove the irrelevant or redundant features in the data sets. In this algorithm the importance of a feature is based on its fitting to the Catastrophe model. Akaike information crite- rion value is used for ranking the features in the data set. The propose…
Proposes NCMN to remove feature correlation in multiplicative noise.
problem High feature correlation in multiplicative noise reduces network performance.
method Exploits batch normalization to remove correlation effect.
result Significantly improves performance on image classification tasks.
Paper proposes efficient BNN inference techniques on FPGA.
problem Redundancy in BNN inference leading to high computation and data access costs.
method Analyzed image similarity and BNN kernel weights to exploit redundancy. Proposed two types of fast and energy-efficient architectures.
result 80% reduction in computation and 40% in buffer access, achieving 17% power reduction.
Prototype selection improves DS techniques' accuracy and reduces computational cost.
problem Improving the performance of dynamic selection techniques.
method Prototype selection techniques that edit validation data to remove noise and redundant instances.
result Improves DS techniques' classification accuracy and reduces computational cost.
Hippo optimizes deep learning hyper-parameters by reducing redundant trials.
problem Redundant hyper-parameter trials in hyper-parameter optimization.
method Hippo breaks down hyper-parameter sequences into stages and executes them in a tree structure.
result Hippo reduces GPU-hours and training time significantly compared to existing methods.
New method removes scalar curvature assumption in Ricci flow smoothing.
problem Uniform bounds on scalar curvature and other factors for Ricci flow.
method Quantitative short-time existence of Ricci flow without scalar curvature assumption.
result Ricci flow smoothing for measure space limits, Gromov-Hausdorff compactness, and topological rigidity results.
A federated method for feature selection in multi-label data.
problem Feature selection in multi-label data for distributed and federated environments.
method Semi-Supervised Federated Multi-Label Feature Selection (SSFMLFS) using fuzzy information measures.
result SSFMLFS outperforms other methods in feature selection for multi-label data in federated settings.
Enhances financial time series forecasting with a multi-period learning framework.
problem Accurate financial time series forecasting requires considering both short-term and long-term trends.
method Proposes a Multi-period Learning Framework (MLF) with three modules: Inter-period Redundancy Filtering, Learnable Weighted-average Integration, and Multi-period self-Adaptive Patching.
result Improves financial time series forecasting accuracy and efficiency.
NAT optimizes neural architectures to improve performance without extra cost.
problem Redundant operations in neural architectures consume memory and degrade performance.
method Transformed Markov Decision Process (MDP) and reinforcement learning to replace redundant operations with more efficient ones.
result Transformed architectures outperform original and existing methods on CIFAR-10 and ImageNet datasets.
The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.
problem Flaws in existing pruning criteria for CNNs.
method Empirical experiments and Convolutional Weight Distribution Assumption.
result The Convolutional Weight Distribution Assumption improves filter selection in pruning.
AMBER method selects features efficiently using autoencoders and model-based elimination.
problem Efficiently selecting relevant features for classification.
method Greedy backward elimination using a ranker model and autoencoders.
result AMBER outperforms other feature selection methods in classification accuracy.
Previously, we proposed a physically inspired rule to organize the data points in a sparse yet effective structure, called the in-tree (IT) graph, which is able to capture a wide class of underlying cluster structures in the datasets, especially for the density-based datasets. Although there are some redundant edges or…
Williams and Beer (2010) proposed a nonnegative mutual information decomposition, based on the construction of redundancy lattices, which allows separating the information that a set of variables contains about a target variable into nonnegative components interpretable as the unique information of some variables not p…
This paper challenges the conventional wisdom of trend-following by showing that the medium-term horizon adds little value once short- and long-term components are included.
problem The conventional wisdom that more horizons improve diversification and performance is challenged.
method A Bayesian optimization framework reallocates exposure dynamically across horizons, optimizing horizon-level weights at the asset level and applying sparsity and turnover control for dynamic allocation across assets.
result The medium-term horizon contributes little incremental performance or diversification once short- and long-term components are included.
Survey on pruning CNN models to reduce size for edge devices.
problem Reducing large CNN models for edge deployment.
method Comprehensive review of pruning strategies, criteria, and techniques.
result Pruning accelerates CNN models for edge applications.
Paper proposes a faster DNN pruning method.
problem Efficiently prune DNNs without sacrificing accuracy.
method Uses PCA to automatically optimize pruning.
result Prunes DNNs in one shot, no retraining needed.
The paper introduces a method to explain redundancy in deep CNNs using unit impulse response.
problem Redundancy in deep CNNs leads to unnecessary computations and increased cost.
method Empirical demonstration and unit impulse response analysis to identify and quantify redundancy across layers and depth.
result Identifies and quantifies redundancy in deep CNNs, providing better insights into their internal dynamics.
Redundancy improves learning stability and generalization in structured systems.
problem Understanding redundancy in structured systems for learning and generalization.
method Developed a theoretical framework that redefines redundancy as a geometric principle unifying various measures.
result Redundancy balances structure and coupling, leading to optimal stability and generalization.
New method quantifies redundant information using information bottleneck.
problem Quantifying redundant information among multiple sources.
method Formulated as an information bottleneck problem, termed redundancy bottleneck.
result Extracts information that best predicts the target without revealing source identity.
MARL algorithm uses regularization to avoid explicit structures, improving performance.
problem Lack of effective reinforcement learning methods for multi-agent systems.
method MARQ uses regularization to promote structured exploration without explicit centralized structures.
result MARQ outperforms existing methods in multi-agent environments.
Transformers reduce redundancy by focusing on invariant relational quantities.
problem Substantial internal redundancy in Transformer models due to coordinate-dependent representations and continuous symmetries.
method Reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction.
result Architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization.
Greedy selection finds smaller, more accurate subnetworks.
problem Finding smaller, accurate subnetworks in large neural networks.
method Greedy forward selection starting from an empty network.
result Theoretical guarantee of finding subnetworks with lower loss.
Simplifies GCNs to improve efficiency and interpretability.
problem Excessive complexity and redundant computation in GCNs.
method Successively removing nonlinearities and collapsing weight matrices.
result Simplified model maintains accuracy and scales to larger datasets.
Study shows DNNs often extract redundant features, influenced by network size and activation function.
problem Redundancy in deep neural network features.
method Hierarchical clustering of features based on cosine distances, varying network sizes and activation functions.
result Network size and activation function are key factors in DNN redundancy.
This work explains scaling laws as redundancy laws in deep learning.
problem The mathematical origins of scaling laws in deep learning models remain unclear.
method Kernel regression and analysis of data covariance spectra.
result Scaling laws can be explained as redundancy laws, revealing the learning curve's slope depends on data redundancy.
Latent features learned by deep learning approaches have proven to be a powerful tool for machine learning. They serve as a data abstraction that makes learning easier by capturing regularities in data explicitly. Their benefits motivated their adaptation to relational learning context. In our previous work, we introdu…
Study on neural networks to identify redundancy issues in safe machine learning.
problem Identifying redundancy in neural network architectures for safe machine learning.
method Experiments with MNIST database using neural network classifiers.
result Underlines difficulties in using neural network classifiers for safe systems.
New method explains GNNs using power iteration clustering.
problem Mysterious mechanism of message passing in GNNs.
method Subspace power iteration clustering (SPIC) models.
result Message passing in GNNs can be understood through power iteration.
This paper deals with a new filter algorithm for selecting the smallest subset of features carrying all the information content of a data set (i.e. for removing redundant features). It is an advanced version of the fractal dimension reduction technique, and it relies on the recently introduced Morisita estimator of Int…
Study finds 10% redundant images in image classification datasets.
problem Redundancy in large image classification datasets.
method Analysis of CIFAR-10 and ImageNet datasets to identify redundant images.
result 10% of images are redundant and can be removed without significant loss of performance.