Method detects neural network equivalence via matrix ensembles and spectral analysis.
problem Detecting equivalence among different deep learning architectures.
method Generating Mixed Matrix Ensembles (MMEs) and matching to conjugate circular ensembles.
result Empirical evidence shows vanishing differences in spectral densities with long tail decay rates.
Multi-headed ensembles boost model performance with faster training.
problem Limited computational resources hinder ensemble search performance.
method Extend NES to multi-headed ensembles, leveraging end-to-end training and one-shot NAS methods.
result Multi-headed ensemble search finds robust ensembles 3 times faster with comparable performance.
New method creates diverse neural ensembles for better uncertainty estimation and robustness.
problem Creating more robust neural networks for uncertainty estimation and dataset shift.
method Automatically constructing ensembles with varying architectures.
result Ensembles with varying architectures outperform deep ensembles in accuracy, uncertainty calibration, and robustness.
High-capacity neural network ensembles often benefit more from high-capacity models than from increased diversity.
problem The performance of high-capacity neural network ensembles is often harmed by interventions that promote predictive diversity.
method A large-scale study of nearly 600 neural network classification ensembles, examining various interventions and architectures.
result Discouraging predictive diversity can be benign in large-network ensembles, and higher-capacity models often yield better performance than diverse architectures.
Embedded ensembles improve neural network performance efficiently.
problem Improving neural network performance with fewer resources.
method Analyzing the wide network limit of gradient descent dynamics using Neural-Tangent-Kernel.
result Embedded ensembles exhibit two regimes: independent and collective, affecting performance.
Bayesian Neural Networks improve geophysical model ensembles with reduced uncertainty.
problem Improving geophysical model projections and uncertainty quantification.
method Developed a Bayesian Neural Network ensemble strategy for geophysical models.
result Bayesian Neural Network ensemble outperforms existing methods in ozone prediction.
Bayesian Quadrature improves ensembling for neural networks with dispersed likelihood peaks.
problem Ensembling neural networks struggles with dispersed, narrow peaks in likelihood surfaces.
method Uses Bayesian Quadrature to construct weighted ensembles of architectures.
result Empirically outperforms state-of-the-art baselines in test likelihood, accuracy, and expected calibration error.
Bayesian deep ensembles improve prediction accuracy in various settings.
problem Improving prediction accuracy of deep ensembles in out-of-distribution settings.
method Introducing a randomised, untrainable function to each ensemble member, enabling a posterior predictive distribution interpretation.
result Bayesian deep ensembles make more conservative predictions and outperform standard ensembles in various tasks.
Neural networks learn from ensemble forecasts without considering their order.
problem Improving reliability of probabilistic weather forecasts.
method Permutation-invariant neural networks for postprocessing ensemble forecasts.
result Models achieve state-of-the-art prediction quality in surface temperature and wind gust forecasts.
Ensembles of neural networks learn better by sharing information.
problem Improving performance of neural networks through collective learning.
method Modeling neural networks as socially interacting agents aiming to maximize their own performance and functional relations to others.
result Optimal collective performance emerges from local interactions between networks, leading to specialization and higher confidence.
Paper proposes ECOC for deep neural network ensembles to improve performance.
problem Designing an ensemble of deep networks is time-consuming and often not beneficial.
method ECOC framework applied to deep networks, with design strategies to balance accuracy and complexity.
result Proposed combinatory technique achieves highest classification performance.
Ensembles of deep neural networks significantly improve generalization accuracy. However, training neural network ensembles requires a large amount of computational resources and time. State-of-the-art approaches either train all networks from scratch leading to prohibitive training cost that allows only very small ens…
Single neural networks can match deep ensembles' benefits without the complexity.
problem The effectiveness and necessity of deep ensembles in neural network models.
method Demonstrated limitations of ensemble diversity and OOD performance in deep ensembles compared to a single larger model.
result A single neural network can replicate deep ensembles' benefits in uncertainty quantification and robustness.
Deep neural networks have revolutionized many fields such as computer vision and natural language processing. Inspired by this recent success, deep learning started to show promising results for Time Series Classification (TSC). However, neural networks are still behind the state-of-the-art TSC algorithms, that are cur…
Bayesian inference using stochastic neural networks ensembles.
problem Approximating Bayesian posterior distributions.
method Formulate stochastic ensembles of neural networks, train with variational inference, and evaluate using Monte Carlo dropout.
result Stochastic ensembles provide more accurate posterior estimates than other methods.
Proposes a new method to enhance neural learning by maximizing information gain.
problem Improving neural learning by selecting key variables to maximize information gain.
method Adaptive Ensemble Kalman Filter to quantify uncertainty and maximize information gain.
result The proposed method enables the neural network to learn more effectively from stochastic systems.
Improved neural network regression uncertainty estimation.
problem Neural networks lack classical uncertainty due to finite data.
method Bootstrapped Deep Ensembles, incorporating parametric bootstrap.
result Significantly improved uncertainty estimation compared to standard Deep Ensembles.
New hyperparameter ensembles boost neural network performance and uncertainty.
problem Improving neural network robustness and uncertainty quantification.
method Designing ensembles over both weights and hyperparameters, stratified across random initializations.
result Hyper-deep and hyper-batch ensembles outperform deep and batch ensembles on various architectures.
Efficient neural network ensembles improve image classification reliability and uncertainty quantification.
problem Uncertainty in neural network predictions for industrial image classification.
method Investigated efficient neural network ensembles (snapshot, batch, multi-input multi-output) for image classification reliability and uncertainty quantification.
result Batch ensemble is a cost-effective and competitive alternative to deep ensembles, offering savings in training and test time.
Improved neural network ensembles using Stein Variational Newton updates.
problem Lack of efficient second-order information in current ensemble methods.
method Proposes a novel approximate Bayesian inference method integrating Stein Variational Newton updates with scalable Hessian approximations.
result Significantly faster convergence and more accurate posterior distribution approximations.
Overparameterized ensembles don't offer generalization benefits over single large models.
problem Theoretical limitations of ensembles in overparameterized settings.
method Using ensembles of random feature (RF) regressors, the paper clarifies how modern ensembles differ from underparameterized counterparts.
result Infinite ensembles of overparameterized RF regressors become pointwise equivalent to single infinite-width RF regressors, and finite width ensembles converge to single models with the same parameter budget.
Proposes optimizing neural network ensemble diversity in feature space.
problem Improving diversity in neural network ensembles while maintaining performance.
method Optimizes particles in the feature space of a specific intermediate layer.
result Significantly outperforms Deep Ensembles on various metrics.
The paper connects neural network ensembles to Bayesian inference using variational methods.
problem Explaining the behavior of ensemble methods in neural networks.
method Deriving conditions for ensemble optimization to reduce divergence to the posterior distribution.
result Ensemble methods can be a valid alternative to approximate Bayesian inference.
Improved neural population modeling using shared features and ensemble detection.
problem Missing shared coding properties in neural latent variable models.
method Feature sharing across tuning curves and soft clustering of neurons.
result More interpretable and better-performing neural population models.
In this work we perform outlier detection using ensembles of neural networks obtained by variational approximation of the posterior in a Bayesian neural network setting. The variational parameters are obtained by sampling from the true posterior by gradient descent. We show our outlier detection results are comparable …
Paper proposes GP-NAS-ensemble for fast neural architecture performance prediction.
problem Estimating neural network performance without training time-consuming evaluations.
method GP-NAS-ensemble framework using ensemble learning improvements.
result Ranked second in a NAS performance prediction challenge.
Ensemble++ uses shared-factor ensembles to scale Thompson Sampling for linear and nonlinear bandits.
problem Computational challenges in Thompson Sampling for large-scale or non-conjugate settings.
method Ensemble++ with shared-factor architecture and random linear combinations.
result Ensemble++ achieves comparable regret to exact Thompson Sampling with significantly smaller ensemble sizes.
SuperNet speeds up neural network ensembling by training a single DNN for various phases.
problem High computational demand of ensembling large neural networks.
method Train a single DNN for multiple phases of training to represent various sub-models.
result SuperNet execution time comparable to single DNN training time plus coupling factors.
Repulsive ensembles improve uncertainty estimates in PINNs for differential equations.
problem Improving uncertainty estimates in PINNs for differential equations.
method Employing repulsive ensembles (RE-PINN) with a repulsive term in the loss function.
result Repulsive ensembles produce more accurate uncertainty estimates and higher sample diversity.
RAFs Ensemble improves neural network uncertainty quantification.
problem Inaccurate predictions in out-of-distribution data.
method Proposes RAFs Ensemble, using random activation functions in neural networks.
result RAFs Ensemble outperforms state-of-the-art methods in uncertainty quantification.
Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and remarkable capability to reduce the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computations that might not be affordable for researcher…
Deep ensembles outperform deep ensembles of Bayesian neural networks on in-distribution data.
problem Improving model calibration and uncertainty quantification in Bayesian Neural Networks.
method Systematic investigation of deep ensembles of Bayesian Neural Networks across various datasets and architectures.
result Deep ensembles consistently outperform deep ensembles of Bayesian neural networks on in-distribution data.
Researchers improve deep ensemble forecast aggregation methods.
problem Aggregating forecast distributions from deep ensembles for better predictive performance.
method Comprehensive analysis of twelve benchmark data sets, comparing probability- and quantile-based aggregation methods for three neural network-based approaches.
result A general quantile aggregation framework for deep ensembles improves predictive performance in various settings.
IENs reduce neural network variance without increasing complexity.
problem Reducing variance in neural networks without increasing model complexity.
method IENs use ensemble parameters during training to reduce variance, removing them during testing.
result IENs reduce network variance by a factor of 1/mL−1, leading to significant error rate decreases. Transforms ensemble predictions to maintain interpretability.
problem Loss of interpretability in deep ensembles.
method Proposes transformation ensembles that aggregate predictions while preserving interpretability.
result Transformation ensembles yield better predictions than individual models and maintain interpretability.
Artificial neural networks have been successfully applied to a variety of machine learning tasks, including image recognition, semantic segmentation, and machine translation. However, few studies fully investigated ensembles of artificial neural networks. In this work, we investigated multiple widely used ensemble meth…
Proposes a new method for ensembling neural subnetworks.
problem Computational expense and limited flexibility of traditional deep ensembles.
method Sequential Bayesian neural subnetwork ensembling.
result Outperforms traditional ensembles in various metrics.
A new method reduces memory usage for deep neural networks ensembles.
problem Reducing memory footprint of ensemble methods for deep learning.
method Extract multiple sub-networks from a single neural network through end-to-end optimization.
result Achieves higher or comparable accuracy with significantly less memory usage.
Ensembles of neural networks improve training dynamics and performance.
problem Improving neural network performance through model size increase.
method Defining collegial ensembles (CE) as multiple independent models trained as a single model, and using theoretical results on NTK to optimize architecture search.
result CE dynamics simplify and scale favorably, resembling wide models, and can be efficiently implemented using group convolutions and block diagonal layers.
This work investigates power laws in deep neural network ensembles and predicts their performance.
problem Understanding the performance of deep neural network ensembles and their optimal structure.
method Investigated the behavior of negative log-likelihood (CNLL) of a deep ensemble as a function of ensemble size and member network size, identifying power law dependencies.
result One large network may perform worse than an ensemble of several medium-size networks, known as a memory split.
Dropout and similar stochastic neural network regularization methods are often interpreted as implicitly averaging over a large ensemble of models. We propose STE (stochastically trained ensemble) layers, which enhance the averaging properties of such methods by training an ensemble of weight matrices with stochastic r…
EPINET improves neural networks with less computation.
problem Efficiently estimating uncertainty in neural networks.
method Introduces EPINET architecture that augments any neural network with minimal additional training.
result EPINET outperforms large ensembles with significantly less computation.
FoRDE uses input gradients to improve neural network ensembles.
problem Improving neural network ensembles for robustness and accuracy.
method Proposes FoRDE, an ensemble learning method based on ParVI, which repels function space by input gradients.
result FoRDE significantly outperforms DEs and other ensemble methods in accuracy and calibration.
Unified framework for ensemble sampling in nonlinear contextual bandits with provable regret bounds.
problem Efficient exploration in nonlinear contextual bandits with unknown feature dimensions.
method Developed GLM-ES and Neural-ES for generalized linear and neural contextual bandits, respectively, using maximum likelihood estimation on randomly perturbed data.
result Unified high-probability frequentist regret bounds for GLM-ES and Neural-ES, matching state-of-the-art results.
CREIMBO models diverse brain activity by identifying hidden neural sub-circuits and their non-stationary interactions.
problem Lack of alignment in neural recordings limits analysis of brain-wide dynamics.
method CREIMBO learns a unified model of neural dynamics by assuming multiple hidden global sub-circuits representing ensemble interactions.
result CREIMBO discovers session-specific neural ensembles and their non-stationary interactions, revealing cross-subject neural mechanisms.
Stein variational neural network ensembles improve diversity and uncertainty estimation.
problem Lack of proper Bayesian justification and diversity guarantees in deep neural network ensembles.
method Particle-based inference methods, specifically Stein variational gradient descent (SVGD), operating in weight space, function space, and hybrid settings.
result SVGD methods improve diversity and uncertainty estimation, approaching the true Bayesian posterior more closely.
Twin neural network regression predicts differences between two data points.
problem Traditional regression methods are inaccurate for certain data sets.
method TNN regression predicts differences between two data points and averages predictions from an ensemble of all training data points.
result TNN regression yields more accurate predictions compared to other methods.
The use of ensembles of neural networks (NNs) for the quantification of predictive uncertainty is widespread. However, the current justification is intuitive rather than analytical. This work proposes one minor modification to the normal ensembling methodology, which we prove allows the ensemble to perform Bayesian inf…