Heterogeneous ensembles built from the predictions of a wide variety and large number of diverse base predictors represent a potent approach to building predictive models for problems where the ideal base/individual predictor may not be obvious. Ensemble selection is an especially promising approach here, not only for …
The motivation of this work is to improve the performance of standard stacking approaches or ensembles, which are composed of simple, heterogeneous base models, through the integration of the generation and selection stages for regression problems. We propose two extensions to the standard stacking approach. In the fir…
Embedded ensembles improve neural network performance efficiently.
problem Improving neural network performance with fewer resources.
method Analyzing the wide network limit of gradient descent dynamics using Neural-Tangent-Kernel.
result Embedded ensembles exhibit two regimes: independent and collective, affecting performance.
Q-Ensembles are a model-free approach where input images are fed into different Q-networks and exploration is driven by the assumption that uncertainty is proportional to the variance of the output Q-values obtained. They have been shown to perform relatively well compared to other exploration strategies. Further, mode…
Ensembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different \emph{forms} of uncertainty. However, ensembles come at a computational and memory cost which may be…
This paper examines how to calibrate ensemble members for better prediction accuracy.
problem Improper calibration of deep neural networks leads to unreliable probability estimates.
method Theoretical analysis and empirical evaluation on CIFAR-100 dataset.
result Well-calibrated ensemble members do not guarantee a well-calibrated ensemble prediction, but a well-calibrated ensemble prediction cannot exceed the average performance of its members.
VGE provides a practical approach to uncertainty estimation in ensemble models.
problem Uncertainty estimation in ensemble models using additive decomposition breaks down.
method Variance-Gated Ensembles (VGE) introduces a differentiable framework with a signal-to-noise gate.
result VGE provides a Variance-Gated Margin Uncertainty (VGMU) score and Variance-Gated Normalization (VGN) layer.
A new stochastic method handles ensemble creation with cost constraints.
problem Creating ensembles under cost limitations in decision-making.
method Introducing a novel stochastic approach to solve the knapsack problem.
result The approach efficiently incorporates ensemble accuracy and cost constraints.
Efficiently builds diverse sub-model ensembles for robust self-supervised learning.
problem Challenges in diversity and efficiency of deep ensembles for self-supervised representation learning.
method Ensemble of independent sub-networks with a new loss function for diversity.
result Significantly improves prediction reliability and model calibration.
The paper explores how prior functions and bootstrapping improve ensemble uncertainty estimation.
problem Improving uncertainty estimation in machine learning models.
method Investigates the benefits of prior functions and bootstrapping in ensemble models.
result Prior functions and bootstrapping enhance ensemble agents' uncertainty estimation across different inputs.
Improved wind speed forecasts for power generation using machine learning.
problem Improving the accuracy and reliability of wind speed predictions for power generation.
method A novel machine learning approach for calibrating wind speed ensemble forecasts.
result The proposed method improves the calibration and accuracy of probabilistic and point forecasts.
Proposes a new method to enhance neural learning by maximizing information gain.
problem Improving neural learning by selecting key variables to maximize information gain.
method Adaptive Ensemble Kalman Filter to quantify uncertainty and maximize information gain.
result The proposed method enables the neural network to learn more effectively from stochastic systems.
Regression Prior Networks improve ensemble performance on regression tasks.
problem Improving ensemble performance on regression tasks.
method Extending Prior Networks and Ensemble Distribution Distillation (EnD2) to regression tasks using the Normal-Wishart distribution. result Regression Prior Networks yield performance competitive with ensemble approaches on regression tasks.
An ensemble of randomized NNs improves time series forecasting accuracy.
problem Forecasting time series with multiple seasonality and nonstationarity.
method Randomized neural networks with pattern-based time series representation and diversity control strategies.
result Outperforms statistical and machine learning models in forecasting accuracy.
S2D efficiently trains models to estimate uncertainty without increasing resource costs.
problem Efficiently estimating uncertainty in deep learning models for safety-critical applications.
method Self-distribution distillation (S2D) approach to train a single model for uncertainty estimation.
result S2D models outperform standard models and Monte-Carlo dropout in uncertainty estimation.
New method tests tree models without causing computational pressure.
problem Incompatibility of adversarial robustness testing with tree ensembles.
method Smooths tree ensembles with sigmoid functions and uses gradient descent.
result Successfully reveals adversarial vulnerability of tree ensemble models.
Proposes a new method for ensembling neural subnetworks.
problem Computational expense and limited flexibility of traditional deep ensembles.
method Sequential Bayesian neural subnetwork ensembling.
result Outperforms traditional ensembles in various metrics.
Enhanced ensemble filters use machine learning to improve accuracy in filtering models.
problem Accuracy limitations of traditional ensemble Kalman filters.
method Introduces a measure neural mapping (MNM) to map joint predicted state and observation to updated state estimates.
result Superior root-mean-square-error performance compared to leading methods in filtering models.
Autoencoders, as a dimensionality reduction technique, have been recently applied to outlier detection. However, neural networks are known to be vulnerable to overfitting, and therefore have limited potential in the unsupervised outlier detection setting. Current approaches to ensemble-based autoencoders do not generat…
SPQR improves Q-ensemble diversity in reinforcement learning.
problem Overestimation bias in Q-learning for complex tasks.
method Introduces SPQR for Q-ensemble independence regularization.
result SPQR outperforms baseline algorithms in online and offline RL benchmarks.
SharpBalance improves deep ensemble performance by balancing sharpness and diversity.
problem Improving deep ensemble performance in both in-distribution and out-of-distribution scenarios.
method Introducing SharpBalance, a novel training approach that balances sharpness and diversity within ensembles.
result SharpBalance effectively improves the sharpness-diversity trade-off and ensemble performance in ID and OOD scenarios.
FED distills ensemble predictions efficiently, improving accuracy and uncertainty.
problem Efficiently distilling ensemble predictions from limited data.
method Proposes Functional Ensemble Distillation (FED) using mixup augmentation.
result Significantly boosts performance in accuracy and uncertainty estimation.
New approach verifies tree ensembles efficiently, making them more robust.
problem Verifying robustness of decision tree ensembles against evasion attacks is NP-hard.
method Identified large-spread ensembles, proposed verifiable learning, designed training algorithm.
result Large-spread ensembles can be verified in polynomial time and are more robust.
Ensembles of deep neural networks significantly improve generalization accuracy. However, training neural network ensembles requires a large amount of computational resources and time. State-of-the-art approaches either train all networks from scratch leading to prohibitive training cost that allows only very small ens…
Proposes a new method for deep ensembles that improves accuracy and calibration.
problem Improving accuracy and calibration of deep ensembles.
method Estimates confusion matrices of ensemble members and weighs them according to their inferred performance.
result Empirically shows superiority of soft Dawid Skene over ensemble averaging.
Optimal ensemble construction improves prediction accuracy for multi-study tasks, especially in pandemic scenarios.
problem Poor out-of-study prediction performance due to heterogeneous datasets.
method Optimal ensemble construction using a two-stage stacking strategy that jointly estimates ensemble weights and study-specific model parameters.
result Our method outperforms multi-study stacking and other standard methods in predicting excess mortality during the pandemic.
Method detects neural network equivalence via matrix ensembles and spectral analysis.
problem Detecting equivalence among different deep learning architectures.
method Generating Mixed Matrix Ensembles (MMEs) and matching to conjugate circular ensembles.
result Empirical evidence shows vanishing differences in spectral densities with long tail decay rates.
CE improves climate uncertainty quantification using GCM ensembles and observational data.
problem Uncertainty in climate projections due to model inadequacies and variability.
method Conformal ensembles integrating GCM ensembles and observational data.
result CE generates statistically rigorous, easy-to-interpret uncertainty estimates.
In this paper we examine the effect of applying ensemble learning to the performance of collaborative filtering methods. We present several systematic approaches for generating an ensemble of collaborative filtering models based on a single collaborative filtering algorithm (single-model or homogeneous ensemble). We pr…
In this article supervised learning problems are solved using soft rule ensembles. We first review the importance sampling learning ensembles (ISLE) approach that is useful for generating hard rules. The soft rules are then obtained with logistic regression from the corresponding hard rules. In order to deal with the p…
BoostForest combines multiple BoostTree models for improved accuracy.
problem Improving ensemble learning performance.
method BoostTree uses gradient boosting and random cut-points. BoostForest bootstraps training data and randomly samples parameters.
result BoostForest outperforms classical ensemble methods on 35 datasets.
Ensemble learning is a very prevalent method employed in machine learning. The relative success of ensemble methods is attributed to their ability to tackle a wide range of instances and complex problems that require different low-level approaches. However, ensemble methods are relatively less popular in reinforcement …
GBEST model improves survival analysis for small datasets.
problem Challenges in survival analysis, especially with small data.
method Bayesian bootstrap and Beta Stacy bootstrap methods integrated into bagging tree models.
result GBEST model outperforms classical survival models in predictive performance and stability.
Researchers improve deep ensemble forecast aggregation methods.
problem Aggregating forecast distributions from deep ensembles for better predictive performance.
method Comprehensive analysis of twelve benchmark data sets, comparing probability- and quantile-based aggregation methods for three neural network-based approaches.
result A general quantile aggregation framework for deep ensembles improves predictive performance in various settings.
The paper explores dynamic ensembles for multi-step forecasting.
problem Lack of research on dynamic ensembles for multi-step forecasting.
method Extensive experiments with 3568 time series and an ensemble of 30 multi-output models.
result Dynamic ensembles based on arbitrating and windowing perform best.
A new method selects models for ensemble learning to maximize mutual information, outperforming existing approaches.
problem Selecting models for ensemble learning to improve performance and reduce correlation issues.
method Formulate budgeted ensemble selection as maximizing mutual information, use Gaussian-copula to model correlated errors, propose a greedy mutual-information selection algorithm.
result Our method consistently outperforms strong baselines across multiple datasets.
This paper explores methods for combining predictions in multilabel classification.
problem Lack of formal framework for aggregation in multilabel ensembles.
method Introduces two approaches: 'predict then combine' (PTC) and 'combine then predict' (CTP).
result Standard voting techniques are outperformed by tailored instantiations of CTP and PTC.
In machine learning ensemble methods have demonstrated high accuracy for the variety of problems in different areas. Two notable ensemble methods widely used in practice are gradient boosting and random forests. In this paper we present InfiniteBoost - a novel algorithm, which combines important properties of these two…
IENs reduce neural network variance without increasing complexity.
problem Reducing variance in neural networks without increasing model complexity.
method IENs use ensemble parameters during training to reduce variance, removing them during testing.
result IENs reduce network variance by a factor of 1/mL−1, leading to significant error rate decreases. Ensemble learning has had many successes in supervised learning, but it has been rare in unsupervised learning and dimensionality reduction. This study explores dimensionality reduction ensembles, using principal component analysis and manifold learning techniques to capture linear, nonlinear, local, and global feature…
New method combines ensembling and regularization for genomic disease prediction.
problem Genomic diseases require accurate prediction and biomarker identification.
method Integrates regularization with ensembling techniques for high-dimensional binary classification.
result Identifies critical biomarkers overlooked by competing methods.
Bayesian Quadrature improves ensembling for neural networks with dispersed likelihood peaks.
problem Ensembling neural networks struggles with dispersed, narrow peaks in likelihood surfaces.
method Uses Bayesian Quadrature to construct weighted ensembles of architectures.
result Empirically outperforms state-of-the-art baselines in test likelihood, accuracy, and expected calibration error.
Clustering ensemble, or consensus clustering, has emerged as a powerful tool for improving both the robustness and the stability of results from individual clustering methods. Weighted clustering ensemble arises naturally from clustering ensemble. One of the arguments for weighted clustering ensemble is that elements (…
This paper improves prediction rule ensembles using model-based data generation.
problem Improving the sparsity and predictive accuracy of prediction rule ensembles.
method The authors use surrogate models to train Lasso regression with data generated by a boosted decision tree ensemble, improving PRE performance.
result The use of surrogacy models can substantially improve the sparsity of PRE while retaining predictive accuracy.
In this article, we advocate the ensemble approach for variable selection. We point out that the stochastic mechanism used to generate the variable-selection ensemble (VSE) must be picked with care. We construct a VSE using a stochastic stepwise algorithm, and compare its performance with numerous state-of-the-art algo…
Ensembles of decision trees perform well on many problems, but are not interpretable. In contrast to existing approaches in interpretability that focus on explaining relationships between features and predictions, we propose an alternative approach to interpret tree ensemble classifiers by surfacing representative poin…
This study improves uncertainty quantification in seismic inversion.
problem Uncertainty in seismic inversion due to limited data and model diversity.
method Integrates ensemble methods with importance sampling.
result More accurate uncertainty quantification in velocity models.
Paper analyzes uncertainty metrics in ensemble learning for healthcare AI.
problem Selecting appropriate uncertainty metrics for ensemble learners in healthcare AI.
method Rigorous analysis of two uncertainty metrics: ensemble mean and variance.
result Ensemble mean is preferable to ensemble variance for decision making in healthcare AI.