Study pitfalls of deep learning ensembles in uncertainty estimation.
problem Pitfalls in in-domain uncertainty estimation and ensembling in deep learning.
method Exploration of standards for uncertainty quantification and broad study of ensembling techniques.
result Many sophisticated ensembling techniques are equivalent to a simple ensemble of few networks.
Transforms ensemble predictions to maintain interpretability.
problem Loss of interpretability in deep ensembles.
method Proposes transformation ensembles that aggregate predictions while preserving interpretability.
result Transformation ensembles yield better predictions than individual models and maintain interpretability.
Auto-Ensemble automates deep learning model ensembling with adaptive learning rate scheduling.
problem Difficulty in collecting diverse and accurate deep learning models through single training.
method Auto-Ensemble collects model checkpoints and uses adaptive learning rate scheduling to ensemble them.
result Ensembled models converge to various local optima, improving performance on few-shot learning.
In this paper, we propose to provide a general ensemble learning framework based on deep learning models. Given a group of unit models, the proposed deep ensemble learning framework will effectively combine their learning results via a multilayered ensemble model. In the case when the unit model mathematical mappings a…
Proposes a new method for deep ensembles that improves accuracy and calibration.
problem Improving accuracy and calibration of deep ensembles.
method Estimates confusion matrices of ensemble members and weighs them according to their inferred performance.
result Empirically shows superiority of soft Dawid Skene over ensemble averaging.
Unified theory linking Bayesian and ensemble methods in deep learning.
problem Uncertainty quantification in deep learning.
method Reformulating optimisation as convex optimisation in probability measures, studying Wasserstein gradient flows.
result Unified theory explaining success of deep ensembles over variational inference.
Efficiently builds diverse sub-model ensembles for robust self-supervised learning.
problem Challenges in diversity and efficiency of deep ensembles for self-supervised representation learning.
method Ensemble of independent sub-networks with a new loss function for diversity.
result Significantly improves prediction reliability and model calibration.
Deep ensembles don't necessarily improve calibration in low data regimes.
problem Calibration issues in deep learning models, especially in low data regimes.
method Examination of data-augmentation, ensembling, and post-processing calibration methods.
result Standard ensembling techniques can lead to less calibrated models in low data regimes.
DGMEs use Gaussian mixtures to quantify uncertainty in deep learning.
problem Quantifying uncertainty in complex predictive densities.
method DGMEs use a Gaussian mixture model with an EM algorithm for parameter learning.
result DGMEs outperform state-of-the-art models in uncertainty quantification.
Deep RL ensemble strategy outperforms individual algorithms in stock trading.
problem Designing profitable stock trading strategies in a complex market.
method Ensemble of three deep reinforcement learning algorithms (PPO, A2C, DDPG) for stock trading.
result Deep ensemble strategy outperforms individual algorithms and traditional min-variance portfolio.
Deep ensembles mimic Bayesian averaging with learned priors.
problem Quantifying uncertainty in neural networks.
method Showed deep ensembles perform exact Bayesian averaging with an implicitly learned data-dependent prior.
result Deep ensembles are Bayesian and provide an explanation for their strong empirical performance.
Bayesian interpretation of deep ensembles improves uncertainty quantification.
problem Improving uncertainty estimation in deep learning models.
method Viewing deep ensembles as an approximate Bayesian method and specifying corresponding assumptions.
result Improved approximation leads to larger epistemic uncertainty, potentially more reliable predictions.
Study on ensemble, distillation, and self-distillation in deep learning models.
problem Improving test accuracy in deep learning models using ensemble and distillation methods.
method Formal study of ensemble and distillation, considering multi-view data structure.
result Proven that ensemble and distillation can improve test accuracy in deep learning models, and the superior performance can be distilled into a single model.
An ensemble method enhances cryptocurrency trading strategies using deep reinforcement learning.
problem Improving generalization performance in stochastic cryptocurrency trading environments.
method Model selection and mixture distribution policy to ensemble deep reinforcement learning models.
result Improved out-of-sample performance compared to benchmarks.
Researchers improve deep ensemble forecast aggregation methods.
problem Aggregating forecast distributions from deep ensembles for better predictive performance.
method Comprehensive analysis of twelve benchmark data sets, comparing probability- and quantile-based aggregation methods for three neural network-based approaches.
result A general quantile aggregation framework for deep ensembles improves predictive performance in various settings.
Jointly tuning ensemble models improves performance and uncertainty calibration.
problem Improving both predictive performance and uncertainty calibration in deep ensembles.
method Investigated the impact of jointly tuning weight decay, temperature scaling, and early stopping.
result Jointly tuning ensemble models generally matches or improves performance, with significant variation across tasks.
Ensemble learning use multiple algorithms to obtain better predictive performance than any single one of its constituent algorithms could. With growing popularity of deep learning, researchers have started to ensemble them for various purposes. Few if any, however, has used the deep learning approach as a means to ense…
GNCL algorithm controls diversity in deep ensembles.
problem Managing bias and variance in deep ensembles.
method Generalized bias-variance decomposition for arbitrary loss functions, leading to GNCL algorithm.
result Explicit control over ensemble diversity and smooth interpolation between independent and joint training.
Single model estimates ensemble uncertainty efficiently.
problem Efficient uncertainty quantification in deep learning models.
method Contextual similarity distillation, approximating ensemble variance with a single model.
result Single model estimates predictive variance with a single forward pass.
Method detects neural network equivalence via matrix ensembles and spectral analysis.
problem Detecting equivalence among different deep learning architectures.
method Generating Mixed Matrix Ensembles (MMEs) and matching to conjugate circular ensembles.
result Empirical evidence shows vanishing differences in spectral densities with long tail decay rates.
Deep ensembles effectively capture epistemic uncertainty through training stochasticity, providing a frequentist perspective.
problem Understanding and quantifying epistemic uncertainty in machine learning models.
method Bootstrap-based estimator and decomposition of deep ensembles into data variability and training stochasticity.
result Deep ensembles primarily capture training stochasticity, explaining their effectiveness in quantifying epistemic uncertainty.
Deep neural networks have revolutionized many fields such as computer vision and natural language processing. Inspired by this recent success, deep learning started to show promising results for Time Series Classification (TSC). However, neural networks are still behind the state-of-the-art TSC algorithms, that are cur…
This paper improves deep learning model consistency through ensemble methods.
problem Consistency and correct-consistency issues in deep learning models.
method Formal definition of consistency and correct-consistency, proving ensemble improvement, proposing dynamic snapshot ensemble method.
result Ensemble methods can improve correct-consistency of deep learning models.
Deep ensembles improve model accuracy and robustness, but their theoretical underpinnings are not fully understood.
problem Understanding why deep ensembles work well in practice despite theoretical limitations.
method Investigating the loss landscape of neural networks and exploring the diversity of functions in function space.
result Random initializations explore diverse modes in function space, while ensembles along an optimization trajectory cluster within a single mode.
In this paper, we introduce Deep Probabilistic Ensembles (DPEs), a scalable technique that uses a regularized ensemble to approximate a deep Bayesian Neural Network (BNN). We do so by incorporating a KL divergence penalty term into the training objective of an ensemble, derived from the evidence lower bound used in var…
Ensemble learning is a methodology that integrates multiple DNN learners for improving prediction performance of individual learners. Diversity is greater when the errors of the ensemble prediction is more uniformly distributed. Greater diversity is highly correlated with the increase in ensemble accuracy. Another attr…
Paper proposes a deep learning model to predict stock prices using sentiment analysis.
problem Predicting future stock movement using financial textual and numerical data.
method A blending ensemble deep learning model with two levels of RNNs, LSTM, and GRU followed by a fully connected neural network.
result The model improves prediction accuracy compared to traditional methods.
DASH improves ensemble generalizability by encouraging diverse, flat loss landscapes.
problem Improving generalization and robustness of deep ensembles.
method DASH promotes diversity and flatness in deep ensembles by encouraging base learners to move towards low-loss regions of minimal sharpness.
result DASH improves ensemble generalizability, as demonstrated by extensive empirical evidence.
This paper examines how to calibrate ensemble members for better prediction accuracy.
problem Improper calibration of deep neural networks leads to unreliable probability estimates.
method Theoretical analysis and empirical evaluation on CIFAR-100 dataset.
result Well-calibrated ensemble members do not guarantee a well-calibrated ensemble prediction, but a well-calibrated ensemble prediction cannot exceed the average performance of its members.
Paper improves gender detection on social media using deep learning.
problem Traditional classifiers struggle with social media data volume.
method Ensemble deep learning with multi-model architectures.
result Improved gender detection accuracy on social media posts.
SDF adapts Deep Forest for evolving data streams with active learning.
problem Adapting Deep Forest for evolving data streams.
method Streaming Deep Forest (SDF) with Augmented Variable Uncertainty (AVU) active learning.
result SDF with AVU outperforms other methods trained with all instances by 70% labeling budget.
Deep learning improves chaotic dynamics filtering without ensemble.
problem Discovering efficient DA schemes for chaotic dynamics.
method Residual Convolutional Neural Network for the analysis step.
result Deep learning achieves ensemble filtering accuracy without an ensemble.
Repulsive deep ensembles improve diversity and Bayesian inference.
problem Challenges in maintaining diversity among ensemble members trained independently.
method Introducing a repulsive term in the update rule of deep ensembles.
result Training dynamics of repulsive ensembles follow a Wasserstein gradient flow of KL divergence with the true posterior.
In this work a novel method to quantify spectral ergodicity for random matrices is presented. The new methodology combines approaches rooted in the metrics of Thirumalai-Mountain (TM) and Kullbach-Leibler (KL) divergence. The method is applied to a general study of deep and recurrent neural networks via the analysis of…
This paper focuses on a comparative evaluation of the most common and modern methods for text classification, including the recent deep learning strategies and ensemble methods. The study is motivated by a challenging real data problem, characterized by high-dimensional and extremely sparse data, deriving from incoming…
This paper introduces Deep Incremental Boosting, a new technique derived from AdaBoost, specifically adapted to work with Deep Learning methods, that reduces the required training time and improves generalisation. We draw inspiration from Transfer of Learning approaches to reduce the start-up time to training each incr…
A new stopping rule based on E-values helps efficiently use sampling in Bayesian Deep Ensembles.
problem How long should sampling continue in Bayesian Deep Ensembles to yield significant improvements?
method Formulated as a sequential anytime-valid hypothesis test, using E-values to decide when to stop sampling.
result Only a fraction of the full-chain budget is often required for significant improvements.
Improved deep hedging with ensemble uncertainty quantification.
problem Uncertainty in deep hedging models hinders their deployment.
method Trained an ensemble of LSTM networks to quantify uncertainty in deep hedging under Heston volatility and proportional transaction costs.
result The ensemble's disagreement provides a strong predictive confidence measure for hedge performance.
FoRDE uses input gradients to improve neural network ensembles.
problem Improving neural network ensembles for robustness and accuracy.
method Proposes FoRDE, an ensemble learning method based on ParVI, which repels function space by input gradients.
result FoRDE significantly outperforms DEs and other ensemble methods in accuracy and calibration.
Bayesian deep ensembles improve prediction accuracy in various settings.
problem Improving prediction accuracy of deep ensembles in out-of-distribution settings.
method Introducing a randomised, untrainable function to each ensemble member, enabling a posterior predictive distribution interpretation.
result Bayesian deep ensembles make more conservative predictions and outperform standard ensembles in various tasks.
Deep learning has become very popular for tasks such as predictive modeling and pattern recognition in handling big data. Deep learning is a powerful machine learning method that extracts lower level features and feeds them forward for the next layer to identify higher level features that improve performance. However, …
Deep neural networks improve ensemble weather forecasts.
problem Improving accuracy and efficiency of ensemble weather forecasts.
method Mixed model combining subset of trajectories with deep neural networks for post-processing.
result Achieved over 14% relative improvement in ensemble forecast skill.
New hyperparameter ensembles boost neural network performance and uncertainty.
problem Improving neural network robustness and uncertainty quantification.
method Designing ensembles over both weights and hyperparameters, stratified across random initializations.
result Hyper-deep and hyper-batch ensembles outperform deep and batch ensembles on various architectures.
S2D efficiently trains models to estimate uncertainty without increasing resource costs.
problem Efficiently estimating uncertainty in deep learning models for safety-critical applications.
method Self-distribution distillation (S2D) approach to train a single model for uncertainty estimation.
result S2D models outperform standard models and Monte-Carlo dropout in uncertainty estimation.
Deep learning ensemble detects social media rumors faster.
problem Timely detection of rumors on social media platforms.
method Ensemble model using deep neural networks and time-series tweet data.
result Improved rumor detection performance by 7.9%.
High-capacity neural network ensembles often benefit more from high-capacity models than from increased diversity.
problem The performance of high-capacity neural network ensembles is often harmed by interventions that promote predictive diversity.
method A large-scale study of nearly 600 neural network classification ensembles, examining various interventions and architectures.
result Discouraging predictive diversity can be benign in large-network ensembles, and higher-capacity models often yield better performance than diverse architectures.
Deep learning models predict mutual funds' performance better than traditional methods.
problem Predicting mutual funds' performance accurately.
method Deep learning models (LSTM, GRUs) trained with Bayesian optimization and ensemble methods.
result Ensemble method of LSTM and GRUs achieves the highest accuracy in forecasting mutual funds' Sharpe ratios.
Efficient facial feature learning with shared representations reduces redundancy and improves accuracy.
problem Redundancy and high computational load in training deep ensemble models.
method Wide Ensemble-based Convolutional Neural Networks (ESRs) with varying branching levels.
result ESRs reduce residual generalization error and outperform state-of-the-art methods on facial expression recognition.