Embedded ensembles improve neural network performance efficiently.
problem Improving neural network performance with fewer resources.
method Analyzing the wide network limit of gradient descent dynamics using Neural-Tangent-Kernel.
result Embedded ensembles exhibit two regimes: independent and collective, affecting performance.
Ensembles of neural networks learn better by sharing information.
problem Improving performance of neural networks through collective learning.
method Modeling neural networks as socially interacting agents aiming to maximize their own performance and functional relations to others.
result Optimal collective performance emerges from local interactions between networks, leading to specialization and higher confidence.
Ensembles of deep neural networks significantly improve generalization accuracy. However, training neural network ensembles requires a large amount of computational resources and time. State-of-the-art approaches either train all networks from scratch leading to prohibitive training cost that allows only very small ens…
This work investigates power laws in deep neural network ensembles and predicts their performance.
problem Understanding the performance of deep neural network ensembles and their optimal structure.
method Investigated the behavior of negative log-likelihood (CNLL) of a deep ensemble as a function of ensemble size and member network size, identifying power law dependencies.
result One large network may perform worse than an ensemble of several medium-size networks, known as a memory split.
Paper proposes ECOC for deep neural network ensembles to improve performance.
problem Designing an ensemble of deep networks is time-consuming and often not beneficial.
method ECOC framework applied to deep networks, with design strategies to balance accuracy and complexity.
result Proposed combinatory technique achieves highest classification performance.
Regression Prior Networks improve ensemble performance on regression tasks.
problem Improving ensemble performance on regression tasks.
method Extending Prior Networks and Ensemble Distribution Distillation (EnD2) to regression tasks using the Normal-Wishart distribution. result Regression Prior Networks yield performance competitive with ensemble approaches on regression tasks.
High-capacity neural network ensembles often benefit more from high-capacity models than from increased diversity.
problem The performance of high-capacity neural network ensembles is often harmed by interventions that promote predictive diversity.
method A large-scale study of nearly 600 neural network classification ensembles, examining various interventions and architectures.
result Discouraging predictive diversity can be benign in large-network ensembles, and higher-capacity models often yield better performance than diverse architectures.
New hyperparameter ensembles boost neural network performance and uncertainty.
problem Improving neural network robustness and uncertainty quantification.
method Designing ensembles over both weights and hyperparameters, stratified across random initializations.
result Hyper-deep and hyper-batch ensembles outperform deep and batch ensembles on various architectures.
A framework preserves uncertainty in ensemble distillation.
problem Preserving uncertainty decomposition in ensemble distillation.
method General framework for distilling both regression and classification ensembles, preserving natural uncertainty decomposition.
result Framework maintains decomposition of predictive uncertainty.
Neural networks learn from ensemble forecasts without considering their order.
problem Improving reliability of probabilistic weather forecasts.
method Permutation-invariant neural networks for postprocessing ensemble forecasts.
result Models achieve state-of-the-art prediction quality in surface temperature and wind gust forecasts.
Bayesian Neural Networks improve geophysical model ensembles with reduced uncertainty.
problem Improving geophysical model projections and uncertainty quantification.
method Developed a Bayesian Neural Network ensemble strategy for geophysical models.
result Bayesian Neural Network ensemble outperforms existing methods in ozone prediction.
Improved neural network regression uncertainty estimation.
problem Neural networks lack classical uncertainty due to finite data.
method Bootstrapped Deep Ensembles, incorporating parametric bootstrap.
result Significantly improved uncertainty estimation compared to standard Deep Ensembles.
Single neural networks can match deep ensembles' benefits without the complexity.
problem The effectiveness and necessity of deep ensembles in neural network models.
method Demonstrated limitations of ensemble diversity and OOD performance in deep ensembles compared to a single larger model.
result A single neural network can replicate deep ensembles' benefits in uncertainty quantification and robustness.
Overparameterized ensembles don't offer generalization benefits over single large models.
problem Theoretical limitations of ensembles in overparameterized settings.
method Using ensembles of random feature (RF) regressors, the paper clarifies how modern ensembles differ from underparameterized counterparts.
result Infinite ensembles of overparameterized RF regressors become pointwise equivalent to single infinite-width RF regressors, and finite width ensembles converge to single models with the same parameter budget.
FoRDE uses input gradients to improve neural network ensembles.
problem Improving neural network ensembles for robustness and accuracy.
method Proposes FoRDE, an ensemble learning method based on ParVI, which repels function space by input gradients.
result FoRDE significantly outperforms DEs and other ensemble methods in accuracy and calibration.
Bayesian inference using stochastic neural networks ensembles.
problem Approximating Bayesian posterior distributions.
method Formulate stochastic ensembles of neural networks, train with variational inference, and evaluate using Monte Carlo dropout.
result Stochastic ensembles provide more accurate posterior estimates than other methods.
Efficient neural network ensembles improve image classification reliability and uncertainty quantification.
problem Uncertainty in neural network predictions for industrial image classification.
method Investigated efficient neural network ensembles (snapshot, batch, multi-input multi-output) for image classification reliability and uncertainty quantification.
result Batch ensemble is a cost-effective and competitive alternative to deep ensembles, offering savings in training and test time.
Method detects neural network equivalence via matrix ensembles and spectral analysis.
problem Detecting equivalence among different deep learning architectures.
method Generating Mixed Matrix Ensembles (MMEs) and matching to conjugate circular ensembles.
result Empirical evidence shows vanishing differences in spectral densities with long tail decay rates.
IENs reduce neural network variance without increasing complexity.
problem Reducing variance in neural networks without increasing model complexity.
method IENs use ensemble parameters during training to reduce variance, removing them during testing.
result IENs reduce network variance by a factor of 1/mL−1, leading to significant error rate decreases. Deep neural networks have revolutionized many fields such as computer vision and natural language processing. Inspired by this recent success, deep learning started to show promising results for Time Series Classification (TSC). However, neural networks are still behind the state-of-the-art TSC algorithms, that are cur…
Ensemble of GANs improves performance on disconnected data.
problem Disconnected datasets in computer vision cannot be represented by continuous GANs.
method Construct an optimization problem to relate single GAN, ensemble of GANs, conditional GANs, and Gaussian Mixture GANs.
result Ensemble of GANs outperforms single GANs with fewer parameters.
Improved neural network ensembles using Stein Variational Newton updates.
problem Lack of efficient second-order information in current ensemble methods.
method Proposes a novel approximate Bayesian inference method integrating Stein Variational Newton updates with scalable Hessian approximations.
result Significantly faster convergence and more accurate posterior distribution approximations.
Dropout and similar stochastic neural network regularization methods are often interpreted as implicitly averaging over a large ensemble of models. We propose STE (stochastically trained ensemble) layers, which enhance the averaging properties of such methods by training an ensemble of weight matrices with stochastic r…
The paper connects neural network ensembles to Bayesian inference using variational methods.
problem Explaining the behavior of ensemble methods in neural networks.
method Deriving conditions for ensemble optimization to reduce divergence to the posterior distribution.
result Ensemble methods can be a valid alternative to approximate Bayesian inference.
Proposes a new method for deep ensembles that improves accuracy and calibration.
problem Improving accuracy and calibration of deep ensembles.
method Estimates confusion matrices of ensemble members and weighs them according to their inferred performance.
result Empirically shows superiority of soft Dawid Skene over ensemble averaging.
Proposes optimizing neural network ensemble diversity in feature space.
problem Improving diversity in neural network ensembles while maintaining performance.
method Optimizes particles in the feature space of a specific intermediate layer.
result Significantly outperforms Deep Ensembles on various metrics.
Artificial neural networks have been successfully applied to a variety of machine learning tasks, including image recognition, semantic segmentation, and machine translation. However, few studies fully investigated ensembles of artificial neural networks. In this work, we investigated multiple widely used ensemble meth…
Repulsive ensembles improve uncertainty estimates in PINNs for differential equations.
problem Improving uncertainty estimates in PINNs for differential equations.
method Employing repulsive ensembles (RE-PINN) with a repulsive term in the loss function.
result Repulsive ensembles produce more accurate uncertainty estimates and higher sample diversity.
Transforms ensemble predictions to maintain interpretability.
problem Loss of interpretability in deep ensembles.
method Proposes transformation ensembles that aggregate predictions while preserving interpretability.
result Transformation ensembles yield better predictions than individual models and maintain interpretability.
A new method reduces memory usage for deep neural networks ensembles.
problem Reducing memory footprint of ensemble methods for deep learning.
method Extract multiple sub-networks from a single neural network through end-to-end optimization.
result Achieves higher or comparable accuracy with significantly less memory usage.
Efficiently builds diverse sub-model ensembles for robust self-supervised learning.
problem Challenges in diversity and efficiency of deep ensembles for self-supervised representation learning.
method Ensemble of independent sub-networks with a new loss function for diversity.
result Significantly improves prediction reliability and model calibration.
In this work we perform outlier detection using ensembles of neural networks obtained by variational approximation of the posterior in a Bayesian neural network setting. The variational parameters are obtained by sampling from the true posterior by gradient descent. We show our outlier detection results are comparable …
New method creates diverse neural ensembles for better uncertainty estimation and robustness.
problem Creating more robust neural networks for uncertainty estimation and dataset shift.
method Automatically constructing ensembles with varying architectures.
result Ensembles with varying architectures outperform deep ensembles in accuracy, uncertainty calibration, and robustness.
Researchers improve deep ensemble forecast aggregation methods.
problem Aggregating forecast distributions from deep ensembles for better predictive performance.
method Comprehensive analysis of twelve benchmark data sets, comparing probability- and quantile-based aggregation methods for three neural network-based approaches.
result A general quantile aggregation framework for deep ensembles improves predictive performance in various settings.
Ensembles, where multiple neural networks are trained individually and their predictions are averaged, have been shown to be widely successful for improving both the accuracy and predictive uncertainty of single neural networks. However, an ensemble's cost for both training and testing increases linearly with the numbe…
RAFs Ensemble improves neural network uncertainty quantification.
problem Inaccurate predictions in out-of-distribution data.
method Proposes RAFs Ensemble, using random activation functions in neural networks.
result RAFs Ensemble outperforms state-of-the-art methods in uncertainty quantification.
Ensemble of MFESNs improves macroeconomic forecasting.
problem Improving accuracy in macroeconomic forecasting.
method Hedge and Follow-the-Leader schemes applied to MFESNs.
result Ensemble models outperform individual models in forecasting.
Existing information-theoretic frameworks based on maximum entropy network ensembles are not able to explain the emergence of heterogeneity in complex networks. Here, we fill this gap of knowledge by developing a classical framework for networks based on finding an optimal trade-off between the information content of a…
Deep ensembles outperform deep ensembles of Bayesian neural networks on in-distribution data.
problem Improving model calibration and uncertainty quantification in Bayesian Neural Networks.
method Systematic investigation of deep ensembles of Bayesian Neural Networks across various datasets and architectures.
result Deep ensembles consistently outperform deep ensembles of Bayesian neural networks on in-distribution data.
Though deep neural networks have achieved significant progress on various tasks, often enhanced by model ensemble, existing high-performance models can be vulnerable to adversarial attacks. Many efforts have been devoted to enhancing the robustness of individual networks and then constructing a straightforward ensemble…
Recognizing apparel attributes has recently drawn great interest in the computer vision community. Methods based on various deep neural networks have been proposed for image classification, which could be applied to apparel attributes recognition. An interesting problem raised is how to ensemble these methods to furthe…
SuperNet speeds up neural network ensembling by training a single DNN for various phases.
problem High computational demand of ensembling large neural networks.
method Train a single DNN for multiple phases of training to represent various sub-models.
result SuperNet execution time comparable to single DNN training time plus coupling factors.
In this work a novel method to quantify spectral ergodicity for random matrices is presented. The new methodology combines approaches rooted in the metrics of Thirumalai-Mountain (TM) and Kullbach-Leibler (KL) divergence. The method is applied to a general study of deep and recurrent neural networks via the analysis of…
Memory split advantage: thinner networks outperform a single wide network.
problem Optimizing deep learning models with limited memory.
method Investigated training a single wide network vs. an ensemble of thinner networks with the same total number of parameters.
result An ensemble of several thinner networks outperforms a single wide network for large memory budgets.
An ensemble of neural networks is known to be more robust and accurate than an individual network, however usually with linearly-increased cost in both training and testing. In this work, we propose a two-stage method to learn Sparse Structured Ensembles (SSEs) for neural networks. In the first stage, we run SG-MCMC wi…
Bayesian Quadrature improves ensembling for neural networks with dispersed likelihood peaks.
problem Ensembling neural networks struggles with dispersed, narrow peaks in likelihood surfaces.
method Uses Bayesian Quadrature to construct weighted ensembles of architectures.
result Empirically outperforms state-of-the-art baselines in test likelihood, accuracy, and expected calibration error.
End-to-End training (E2E) is becoming more and more popular to train complex Deep Network architectures. An interesting question is whether this trend will continue-are there any clear failure cases for E2E training? We study this question in depth, for the specific case of E2E training an ensemble of networks. Our str…
The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we …