Ensemble pruning, selecting a subset of individual learners from an original ensemble, alleviates the deficiencies of ensemble learning on the cost of time and space. Accuracy and diversity serve as two crucial factors while they usually conflict with each other. To balance both of them, we formalize the ensemble pruni…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,878 papers · 148 categories
Trend · papers per month
48 results for “ensemble pruning”
SAEP prunes sub-architectures to reduce search cost while maintaining performance.
problem Redundancy in ensemble sub-architectures leads to high computational cost.
method SAEP leverages diversity to prune sub-architectures, reducing ensemble size.
result SAEP reduces the number of sub-architectures without degrading performance.
We consider the problem of learning decision rules for prediction with feature budget constraint. In particular, we are interested in pruning an ensemble of decision trees to reduce expected feature cost while maintaining high prediction accuracy for any test example. We propose a novel 0-1 integer program formulation …
ForestPrune optimizes tree ensemble pruning for compactness and speed.
problem Large tree ensembles in predictive models consume excessive memory and reduce interpretability.
method Developed a specialized optimization algorithm to efficiently prune tree ensembles by depth layers.
result ForestPrune produces compact, high-performing models that outperform existing post-processing methods.
Proposes a method to reduce ensemble size while maintaining accuracy.
problem Complexity and computational burden of ensemble models in large-scale data.
method Optimizes margin distribution to reduce ensemble size while increasing diversity.
result Pruned ensemble uses only a fraction of original classifiers with improved or similar generalization performance.
Deep learning ensembles improve COVID-19 detection from chest X-rays.
problem Detecting COVID-19 from chest X-rays using machine learning.
method Custom CNN and ImageNet models, transfer learning, iterative pruning, ensemble learning.
result 99.01% accuracy in detecting COVID-19 from chest X-rays.
Research reveals how diversity impacts ensemble generalization in classification tasks.
problem Understanding the relationship between diversity and generalization in classification ensembles.
method Investigated diversity measurement, its relationship with generalization error, and pruning methods.
result Generalization error is reduced effectively only when diversity is increased in specific ranges, not in others.
An ensemble of neural networks is known to be more robust and accurate than an individual network, however usually with linearly-increased cost in both training and testing. In this work, we propose a two-stage method to learn Sparse Structured Ensembles (SSEs) for neural networks. In the first stage, we run SG-MCMC wi…
Unsupervised ensemble learning has long been an interesting yet challenging problem that comes to prominence in recent years with the increasing demand of crowdsourcing in various applications. In this paper, we propose a novel method-- unsupervised ensemble learning via Ising model approximation (unElisa) that combine…
Optimizes neural networks for solving problems with pruning and ensembles of minimal structures.
problem Improving neural network performance and interpretability.
method Pruning neural networks based on the principle of controlling training and pruning, using sensitivity indicators and logically transparent NN.
result Ensemble of minimal neural networks provides diverse forecasting algorithms and identifies areas for further data collection.
Paper proposes a multi-phase pruning pipeline for deep ensemble learning on IIoT devices.
problem Computational limitations of IoT devices for deep learning models.
method Generates diverse pruned models, applies integer quantization, and uses clustering-based pruning.
result Significant reduction in model size (up to 90%) and improved performance (up to 7%) on IIoT devices.
The objective of this paper is to define an effective strategy for building an ensemble of Genetic Programming (GP) models. Ensemble methods are widely used in machine learning due to their features: they average out biases, they reduce the variance and they usually generalize better than single models. Despite these a…
New test improves tree ensemble pruning for better model performance.
problem Lack of robust theoretical justification for penalty terms in tree ensembles.
method Developed a novel hypothesis test for tree ensemble split quality.
result Significant reduction in out-of-sample loss using the new test.
Machine learning algorithms have been effectively applied into various real world tasks. However, it is difficult to provide high-quality machine learning solutions to accommodate an unknown distribution of input datasets; this difficulty is called the uncertainty prediction problems. In this paper, a margin-based Pare…
Partial Fusion of Neural Networks: Efficient Tradeoffs Between Ensembles and Weight Aggregationcs.LG
Partial fusion combines neural networks to balance accuracy and efficiency.
problem Balancing accuracy and computational cost in neural networks.
method Extending weight aggregation methods based on neuron-level similarity, using partial optimal transport to match similar neurons.
result Achieves a flexible tradeoff between computational cost and performance.
To Bag is to Prunestat.ML
Random Forests automatically prune a latent 'true' tree, explaining their overfitting without tuning.
problem Difficulty in building bad Random Forests and overfitting without apparent consequences.
method Bootstrap aggregation and model perturbation in Random Forests.
result Randomized ensembles implicitly perform optimal early stopping out-of-sample, explaining overfitting.
In the context of variable selection, ensemble learning has gained increasing interest due to its great potential to improve selection accuracy and to reduce false discovery rate. A novel ordering-based selective ensemble learning strategy is designed in this paper to obtain smaller but more accurate ensembles. In part…
A new method creates simpler, more interpretable decision trees from complex ensembles.
problem Complex tree ensembles reduce interpretability and control over machine learning models.
method Dynamic-programming based algorithm for finding a minimum-size decision tree.
result Optimal born-again trees are simpler and more interpretable than original ensembles.
RocketStack integrates predictions from multiple base learners using a recursive stacking architecture up to ten levels.
problem Feature redundancy, complexity, and computational burden in deep stacking.
method Level-aware recursive stacking with pruning and compression techniques.
result Increasing accuracy with depth and outperforming standalone ensembles at later levels.
EnSyth enhances deep learning models' predictability through ensemble synthesis.
problem Compressing deep learning models for resource-limited environments while maintaining accuracy.
method EnSyth generates diverse compressed models, synthesizes their outputs, and eliminates the least performing combinations.
result EnSyth outperforms baseline models on CIFAR-10 and CIFAR-5 datasets with LeNet-5.
Pruning CNNs by removing less important filters based on empirical loss changes.
problem Reducing memory and computation requirements for CNNs on resource-limited devices.
method Developed a novel filter importance norm based on empirical loss changes, and used sampling and ranking to prune filters.
result Reduced 60% of parameters and 64% of FLOPs with less than 0.6% accuracy drop.
Tree ensembles such as random forests and boosted trees are accurate but difficult to understand, debug and deploy. In this work, we provide the inTrees (interpretable trees) framework that extracts, measures, prunes and selects rules from a tree ensemble, and calculates frequent variable interactions. An rule-based le…
SigD2 reduces noisy rules in rule-based classifiers for better accuracy and readability.
problem Redundant and noisy rules in rule-based classifiers reduce model accuracy and readability.
method Two-stage pruning strategy and ensemble methods (bagging and boosting) to reduce noise and improve model performance.
result SigD2 and ACboost ensemble models outperform state-of-the-art classifiers in terms of accuracy and rule count.
Random forests perform bootstrap-aggregation by sampling the training samples with replacement. This enables the evaluation of out-of-bag error which serves as a internal cross-validation mechanism. Our motivation lies in using the unsampled training samples to improve each decision tree in the ensemble. We study the e…
Paper presents a self-adaptive learning model for robust classification and regression.
problem Dealing with various datasets of different complexity.
method Combines DNDN and DSP, an end-to-end training approach with multiple randomly initialized softmax layers and adaptive soft pruning.
result The model demonstrates no performance loss compared with unpruned models and higher robustness over different data and feature distributions.
This paper improves forest pruning to balance accuracy and interpretability.
problem Limited interpretability of regression forests.
method Lasso-pruning and theoretical analysis of regression forests.
result Pruned regression forests can achieve equal or better accuracy than unpruned ones, with significant size reduction.
Shallow trees in ensemble models make models more interpretable and sometimes better.
problem Lack of transparency in high-performing tree ensemble models.
method Developed an interpretation algorithm to convert tree ensembles into functional ANOVA representations. Proposed strategies to enhance interpretability.
result Shallow trees in ensemble models can lead to better generalization performance and improved interpretability.
New methods improve tree ensemble models by compressing them while maintaining accuracy.
problem Theoretical understanding and practical compression of tree ensembles like random forests and gradient boosting machines.
method Spectral perspective on tree ensembles, deriving minimax rates and developing compression schemes.
result Leading eigenfunctions/singular vectors capture dominant predictive directions, leading to smaller, competitive models.
Proposes a new method for ensembling neural subnetworks.
problem Computational expense and limited flexibility of traditional deep ensembles.
method Sequential Bayesian neural subnetwork ensembling.
result Outperforms traditional ensembles in various metrics.
This study argues for pruning trees in random forests to improve performance in low signal-to-noise scenarios.
problem Improving random forest performance in scenarios with low signal-to-noise ratio.
method Using regularization theory, the study re-examines the depth of trees in random forests and provides evidence that shallow trees are advantageous.
result Random forests with shallow trees are advantageous when the signal-to-noise ratio is low.
Engine SixtyFour uses neural networks to play Crazyhouse chess.
problem Developing a neural network-based evaluation function for Crazyhouse chess.
method Created an ensemble model for Crazyhouse chess using a neural network.
result Early versions of the network have a playing level comparable to a strong amateur.
MBExplainer provides explanations for models combining graph embeddings and tabular features.
problem Explaining models using a mix of graph embeddings and tabular features.
method Model-agnostic approach using Shapley values and Monte Carlo Tree Search.
result MBExplainer efficiently finds human-readable explanations for model predictions.
The paper proposes a method to verify tree ensembles by reasoning about potential instances.
problem Understanding how tree ensembles extrapolate to unseen data.
method Prunes input space and uses divide and conquer approach for incremental answers.
result Shows the approach's usefulness on various use cases.
Lookahead pruning extends single-layer optimization to multi-layer, outperforming magnitude-based pruning.
problem Pruning neural networks to reduce computational cost and memory usage.
method Developed a multi-layer optimization approach extending the single-layer optimization of magnitude-based pruning.
result Consistently outperforms magnitude-based pruning on various networks, especially in high sparsity.
This work presents an approach to automatically induction for non-greedy decision trees constructed from neural network architecture. This construction can be used to transfer weights when growing or pruning a decision tree, allowing non-greedy decision tree algorithms to automatically learn and adapt to the ideal arch…
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
EASIER-net uses sparse networks to improve prediction accuracy for high-dimensional data.
problem Limited use of neural networks in high-dimensional data with small samples.
method Ensemble by Averaging Sparse-Input Hierarchical networks (EASIER-net) with small modifications to neural network architecture and training procedure.
result EASIER-net achieves higher prediction accuracy than off-the-shelf methods on average.
ICE-Pruning accelerates deep neural network pruning by 9.61x.
problem Efficiently pruning deep neural networks while maintaining accuracy.
method Iterative pruning with automatic fine-tuning steps, freezing strategy, and custom learning rate scheduler.
result Significantly reduces pruning time by up to 9.61x.
DSA efficiently allocates sparsity across layers for budgeted pruning.
problem Efficiently distributing resources (sparsity) across layers in pruning under resource constraints.
method DSA uses differentiable pruning to find continuous layer-wise pruning ratios via gradient-based optimization.
result DSA achieves superior performance and significantly reduces the time cost of pruning.
Study examines effects of pruning techniques on deep learning models.
problem Understanding the impact of pruning methods on deep learning model structure and dynamics.
method Investigated differences in connectivity and learning dynamics of pruned models using various iterative pruning techniques.
result Emergence of structure in pruned models through magnitude-based unstructured pruning and weight rewinding.
Pruning neural networks can improve test accuracy even with significant parameter reduction.
problem The tradeoff between generalization and stability in neural network pruning.
method Analysis of pruning behavior over training, focusing on instability and its relation to generalization.
result Pruning's benefit to generalization increases with its instability.
Dynamic pruning during training reduces deep network complexity without significant accuracy loss.
problem High memory and computational requirements of deep networks during training and inference.
method Dynamic pruning of convolutional filters during training, using L1 normalization for optimization.
result L1 normalization-based pruning yields up to 50% reduction in filters with minimal accuracy loss.
The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.
problem Flaws in existing pruning criteria for CNNs.
method Empirical experiments and Convolutional Weight Distribution Assumption.
result The Convolutional Weight Distribution Assumption improves filter selection in pruning.
Improved ROCKET algorithm for brain activity classification.
problem Classifying multivariate time series data from brain activity.
method Detach-Rocket Ensemble, leveraging pruning and ensemble methods.
result Competitive classification accuracy and interpretable channel relevance.
Deep neural networks have dramatically achieved great success on a variety of challenging tasks. However, most successful DNNs have an extremely complex structure, leading to extensive research on model compression.As a significant area of progress in model compression, traditional gradual pruning approaches involve an…
Gibbs pruning optimizes neural networks by combining physics and regularization.
problem Large neural networks are impractical for many applications.
method Combines statistical physics and stochastic regularization to train and prune networks simultaneously.
result Gibbs pruning achieves state-of-the-art performance on ResNet-56.
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
New method prunes neural networks at initialization, improving performance.
problem Improving neural network compression at initialization.
method Formally characterizes initialization conditions for reliable pruning based on connection sensitivity.
result Improved neural network performance on image classification tasks.