This paper explores how optimizing data access and reducing redundancy can improve machine learning algorithm performance.
problem Performance issues in machine learning algorithms due to data locality and redundancy.
method Analysis of data access patterns and computational redundancy in machine learning algorithms, identifying opportunities for reuse and experimentation.
result Initial indicative results show potential for improving performance through data access optimization and reuse of computation results.
New method quantifies redundant information using information bottleneck.
problem Quantifying redundant information among multiple sources.
method Formulated as an information bottleneck problem, termed redundancy bottleneck.
result Extracts information that best predicts the target without revealing source identity.
HAGs eliminate redundant computations in GNNs, improving training efficiency.
problem Redundant computations in GNNs leading to inefficiencies.
method Hierarchically Aggregated computation Graphs (HAGs) to manage and eliminate redundant computations.
result Significant improvement in training efficiency (up to 2.8x) with HAGs.
The paper introduces a method to explain redundancy in deep CNNs using unit impulse response.
problem Redundancy in deep CNNs leads to unnecessary computations and increased cost.
method Empirical demonstration and unit impulse response analysis to identify and quantify redundancy across layers and depth.
result Identifies and quantifies redundancy in deep CNNs, providing better insights into their internal dynamics.
DETOX improves distributed training resilience with redundancy and robust aggregation.
problem Byzantine node failures in distributed training.
method Combines redundancy and robust aggregation methods.
result Substantial increase in robustness with nearly linear runtime.
New data encoding method speeds up distributed optimization.
problem Slow tasks (stragglers) slow down distributed optimization.
method Embed redundancy in data, allowing computation without waiting for stragglers.
result Deterministic linear convergence to approximate solution.
A method speeds up generation in convolutional autoregressive models.
problem Slow generation in convolutional autoregressive models.
method Cache hidden states to avoid redundant computation.
result Up to 21x and 183x speedups in generation for Wavenet and PixelCNN++ models.
This work prunes CNN filters based on their functionality, not just size.
problem Redundant filters in CNNs waste computation resources.
method Functionality-oriented filter pruning method.
result Pruning based on functionality optimizes computation and interprets filter importance.
New CR representations are found and shown to be redundant.
problem Identifying and classifying CR representations of 3-manifolds.
method Experimental computation of limit sets and exact computations of triangle groups.
result Many CR representations are redundant and conjugate.
Transformer models waste resources on long-context tasks.
problem Redundant attention computations in Transformer models for long-context tasks.
method Reformulate sequence modeling as supervised learning, analyze attention sparsity, formulate attention optimization as linear coding problem, propose Dynamic Group Attention.
result DGA reduces computational costs while maintaining performance.
Novel approach reduces DNN redundancy and enhances computational efficiency.
problem Redundant parameters in large-scale DNNs pose hardware deployment challenges.
method WGSEF regularization technique for structured sparsity.
result Reduces redundancy and enhances computational efficiency.
DRACO improves robustness in distributed training by using redundant gradients.
problem Byzantine failures and adversarial compute nodes corrupting model training.
method DRACO uses redundant gradients to eliminate adversarial updates, leveraging coding theory.
result DRACO provides problem-independent robustness guarantees and is significantly faster than median-based approaches.
This paper enhances ML algorithms by improving data locality and reducing redundancy.
problem Improving performance of machine learning algorithms with complex data.
method Exploiting data locality and reuse in memory hierarchies of modern processors.
result Efficient implementation of machine learning algorithms can be achieved by reusing computation results.
Collage inference uses redundancy to reduce cloud image classification latency variance.
problem Reducing latency variance in cloud image classification.
method Integrates collage-cnn for low-cost redundancy in multi-image classification.
result Significant reduction in 99th percentile tail latency and inference latency variation.
Redundancy helps speed up slow nodes in distributed learning.
problem Slow nodes (stragglers) bottleneck distributed optimization and learning performance.
method Encode data with redundancy, dynamically exclude stragglers, and compensate losses.
result Optimization algorithms converge to solutions even with straggling nodes.
Proposes finding missing features in Lasso solutions.
problem Lasso overlooks features not selected in its optimal solution.
method Computes alternate features efficiently without redundant computations.
result Reasonable alternate features found in 20 newsgroup data.
Paper proposes efficient BNN inference techniques on FPGA.
problem Redundancy in BNN inference leading to high computation and data access costs.
method Analyzed image similarity and BNN kernel weights to exploit redundancy. Proposed two types of fast and energy-efficient architectures.
result 80% reduction in computation and 40% in buffer access, achieving 17% power reduction.
New method quantifies multivariate redundancy using maximum entropy decompositions.
problem Elusive multivariate measures of redundancy that comply with nonnegativity and axioms.
method Maximum entropy framework, rooted tree-based decompositions of mutual information.
result Quantifies different multivariate redundancy contributions.
Paper compresses deep neural networks by eliminating redundant neurons.
problem Challenges in deploying deep learning models due to high parameter count.
method Exploits non-linear redundancy to compress neural networks without loss.
result Reduces network size by up to 99% with minimal performance loss.
Stage-based hyper-parameter optimization reduces GPU-hours and training time.
problem Efficiently executing hyper-parameter optimization for deep learning models.
method Stage-based execution strategy to remove redundant computations.
result Stage-based execution outperforms trial-based method by up to 6.60 times in GPU-hours and 4.13 times in training time.
Transformer-MGK replaces redundant heads with Gaussian key mixtures, improving efficiency and performance.
problem Redundant attention heads in transformers degrade performance and efficiency.
method Transformer-MGK replaces redundant heads with a mixture of Gaussian keys.
result Transformer-MGK accelerates training and inference, reduces parameters and FLOPs, and achieves comparable or better accuracy.
Novel framework reduces slow nodes' impact on distributed computing.
problem Mitigating slow nodes (stragglers) in distributed computing.
method Developed mathematical understanding and analyzed convergence behavior of encoded optimization algorithms.
result Characterized trade-offs between various parameters in encoded optimization.
Estimates interactions between modalities for multimodal data.
problem Accurately quantifying interactions between different data types.
method Developed Lightweight Sample-wise Multimodal Interaction (LSMI) estimator using pointwise information theory.
result LSMI reveals fine-grained dynamics in multimodal data.
Gradient boosts monomial-order-free basis construction algorithms.
problem Lack of theoretical properties in monomial-order-free basis construction algorithms.
method Exploits gradient to sidestep spurious vanishing, achieve consistent output, and remove redundant bases.
result Proposes methods that equip monomial-order-free algorithms with theoretical properties.
Redundancy improves learning stability and generalization in structured systems.
problem Understanding redundancy in structured systems for learning and generalization.
method Developed a theoretical framework that redefines redundancy as a geometric principle unifying various measures.
result Redundancy balances structure and coupling, leading to optimal stability and generalization.
Redundancy in AI perception systems doesn't guarantee independent error occurrences.
problem Lack of direct statistical evidence of super-human automated driving performance.
method Investigated the effectiveness of redundancy in neural networks for independent error occurrences.
result Errors in neural networks for computer vision tasks are correlated, not independent.
Collage-CNN reduces cloud inference latency by 1.47X with 9X reduced latency variation.
problem Reducing latency variance in cloud machine learning inference.
method Proposes a novel Collage-CNN model that combines multiple images for classification, providing redundancy and cost efficiency.
result Significant reduction in 99th percentile tail latency and variation in inference latency.
Transformers reduce redundancy by focusing on invariant relational quantities.
problem Substantial internal redundancy in Transformer models due to coordinate-dependent representations and continuous symmetries.
method Reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction.
result Architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization.
Bayesian model averaging under predictor redundancy
problem Reporting Bayesian model averaging posterior without changing the Bayesian target
method Using hard or soft regions of support space
result Region reports often give shorter and clearer summaries while preserving the main posterior information
Efficient algorithm solves best subset selection problem.
problem Sparse learning problems, especially best subset selection.
method Primal-dual method based on dual forms of ℓ0-regularized problems. result Improves solutions of best subset selection with reduced redundant computation.
A new cross-validation method reduces redundancy and improves model performance.
problem Redundancy in traditional k-fold cross-validation leads to biased results.
method Irredundant k-fold cross-validation, where each instance is used exactly once for training and testing.
result Consistent performance estimates with reduced variance and lower computational cost.
Coded Federated Learning speeds up training in edge computing networks.
problem Slow convergence in Federated Learning due to heterogeneity and stochastic fluctuations.
method Exploiting statistical properties of compute and communication delays, distributed kernel embedding, and random Fourier features.
result Significant performance gains for CodedFedL in distributed non-linear regression and classification problems.
Many objective Bayesian optimization tackles redundant objectives in expensive black-box functions.
problem Efficiently optimizing multiple expensive and noisy black-box functions with redundant objectives.
method Proposes a metric to identify redundant objectives and a Bayesian optimization algorithm to stop evaluating them.
result Reduces computational cost by stopping evaluation of redundant objectives, improving efficiency.
Funnel-Transformer reduces computation by compressing sequence data.
problem Redundant token-level representations in language processing.
method Gradually compresses sequence of hidden states to a shorter one.
result Funnel-Transformer outperforms standard Transformer with fewer FLOPs.
Study shows DNNs often extract redundant features, influenced by network size and activation function.
problem Redundancy in deep neural network features.
method Hierarchical clustering of features based on cosine distances, varying network sizes and activation functions.
result Network size and activation function are key factors in DNN redundancy.
Paper proposes redundancy-free features for zero-shot object recognition.
problem Redundant visual features degrade zero-shot object recognition.
method Project original features into a new, statistically independent space.
result RFF-GZSL achieves competitive results on benchmark datasets.
We introduce new definitions of universal and superuniversal computable codes, which are based on a code's ability to approximate Kolmogorov complexity within the prescribed margin for all individual sequences from a given set. Such sets of sequences may be singled out almost surely with respect to certain probability …
This work explains scaling laws as redundancy laws in deep learning.
problem The mathematical origins of scaling laws in deep learning models remain unclear.
method Kernel regression and analysis of data covariance spectra.
result Scaling laws can be explained as redundancy laws, revealing the learning curve's slope depends on data redundancy.
A2 Learning reduces redundant examples in AL for NLP tasks.
problem Redundant examples in AL strategies waste annotation effort.
method A2 Learning actively adapts to deep learning models to eliminate redundant examples.
result A2 Learning reduces data requirements by 3-25% on NLP tasks.
Previously in 2014, we proposed the Nearest Descent (ND) method, capable of generating an efficient Graph, called the in-tree (IT). Due to some beautiful and effective features, this IT structure proves well suited for data clustering. Although there exist some redundant edges in IT, they usually have salient features …
Study on neural networks to identify redundancy issues in safe machine learning.
problem Identifying redundancy in neural network architectures for safe machine learning.
method Experiments with MNIST database using neural network classifiers.
result Underlines difficulties in using neural network classifiers for safe systems.
This work compares data reduction criteria for online Gaussian Processes.
problem The computational complexity of Gaussian Processes limits their applicability to small datasets and streaming scenarios.
method Unified comparison of several data reduction criteria, analyzing computational complexity and reduction behavior.
result Practical guidelines for choosing a suitable data reduction criterion for online Gaussian Processes.
A new method prunes CNN channels recursively considering inter-layer dependency for significant acceleration.
problem NP-hard channel pruning due to inter-layer dependency in CNNs.
method Recursive Bayesian Pruning (RBP) using a Markov chain model and Dirac-like prior.
result Up to 5.0× FLOPs reduction with minimal accuracy loss on large datasets.
Reduces redundant words in neural summarization models.
problem Redundant repeating generation in RNN-based models.
method Jointly estimates vocabulary frequency and controls output based on estimation.
result Significant improvement over RNN baseline, best results on benchmark.
Study finds 10% redundant images in image classification datasets.
problem Redundancy in large image classification datasets.
method Analysis of CIFAR-10 and ImageNet datasets to identify redundant images.
result 10% of images are redundant and can be removed without significant loss of performance.
We introduce a new parameterization method for deep learning layers using spectral tensor train decomposition.
problem Efficiency and stability in deep learning models with weight matrix compression.
method Spectral Tensor Train Parameterization (STTP) of weight matrices.
result Improved compression and training stability in neural networks.
New method avoids redundancy in spectral embeddings.
problem Redundant coordinates in spectral dimensionality reduction.
method Introduces unpredictability constraints to avoid redundancy.
result Significantly more informative and compact representations.
Hippo optimizes deep learning hyper-parameters by reducing redundant trials.
problem Redundant hyper-parameter trials in hyper-parameter optimization.
method Hippo breaks down hyper-parameter sequences into stages and executes them in a tree structure.
result Hippo reduces GPU-hours and training time significantly compared to existing methods.