Analog methods improve forecast accuracy in complex models.
problem Improving forecast accuracy in complex models like Lorenz-96.
method Constructing analogs using variational autoencoders for ensemble data assimilation.
result Constructed analogs perform as well as a full ensemble square root filter.
Rainfall ensemble forecasts have to be skillful for both low precipitation and extreme events. We present statistical post-processing methods based on Quantile Regression Forests (QRF) and Gradient Forests (GF) with a parametric extension for heavy-tailed distributions. Our goal is to improve ensemble quality for all t…
AnEn uses past analogs for weather forecasting, but this work replaces the dataset with deep generative models.
problem Memory and computational costs for storing and searching historical data.
method Deep generative models to replace historical data and analogs search.
result Generative models reduce memory and computational costs significantly.
Extends spectral number variance convergence to random matrix ensembles for twisted Laplacians.
problem Spectral number variance convergence for twisted Laplacians and Dirac operators.
method Extends Rudnick's approach to Gaussian ensembles for twisted Laplacians and Dirac operators.
result Convergence to Gaussian ensembles for twisted Laplacians and Dirac operators.
Investigates financial portfolios using quantum system analogies and clustering properties.
problem Understanding the behavior and clustering of correlated financial assets.
method Analogy with quantum systems, development of eigenportfolios, and use of metrics for participation matrix.
result Shows localized states in the correlation matrix of digital currencies, indicating clustering behavior.
New methods improve tree ensemble models by compressing them while maintaining accuracy.
problem Theoretical understanding and practical compression of tree ensembles like random forests and gradient boosting machines.
method Spectral perspective on tree ensembles, deriving minimax rates and developing compression schemes.
result Leading eigenfunctions/singular vectors capture dominant predictive directions, leading to smaller, competitive models.
New insights into how randomization affects greedy model selection.
problem Understanding the impact of feature subsampling on greedy model selection.
method Investigated greedy forward selection with feature subsampling, proving effects on bias and variance.
result Ensembling with feature subsampling reduces both bias and variance, unlike convex base learners.
Residual networks' depth is mathematically equivalent to expanding an implicit ensemble size.
problem Understanding why deep residual networks are effective.
method Formal analysis of residual networks as ensembles of shallow models.
result Increasing network depth is equivalent to expanding the size of an implicit ensemble, revealing a hierarchical structure.
GER learns particle dynamics from unpaired snapshots using physics-informed GANs.
problem Learning particle dynamics from unpaired snapshots with physics constraints.
method Physics-informed generative model to fit particle ensemble distributions.
result Inferred dynamics of particle ensembles governed by SODEs up to 100 dimensions.
Ant colonies and boosting algorithms both reduce bias and variance through adaptive mechanisms.
problem Understanding the mathematical principles behind ensemble learning and ant colony behavior.
method Developed a formal mapping between AdaBoost's adaptive reweighting and ant recruitment dynamics.
result Proved that the fundamental theorem of weak learnability has a direct analog in colony decision-making.
Study on learning to defer to multiple experts with consistent surrogates and confidence calibration.
problem Addressing the open problems of consistent surrogates, confidence calibration, and ensembling of experts.
method Derive two consistent surrogates (softmax and OvA) and propose a conformal inference technique for choosing experts.
result The OvA-based loss does not cause mis-calibration propagation, while the softmax-based loss does.
We study a new class of codes for lossy compression with the squared-error distortion criterion, designed using the statistical framework of high-dimensional linear regression. Codewords are linear combinations of subsets of columns of a design matrix. Called a Sparse Superposition or Sparse Regression codebook, this s…
New ACE cost function encourages diversity in neural networks.
problem Training multiple classifiers with controlled diversity.
method Mathematical derivation and gradient control.
result ACE yields better ensemble results than vanilla.
Unified meta-learning framework from supervised learning.
problem Difficulty in comparing and evaluating meta-learning approaches.
method Treating meta-learning as supervised learning, reducing algorithms to supervised learning instances.
result Unified framework and improved model performance on few-shot learning.
Stacked regressions improve predictive accuracy by combining estimators.
problem Improve predictive accuracy in regression models.
method Analogous to least-squares, learn combination weights by minimizing regularized empirical risk with nonnegativity constraint.
result The stacked estimator has strictly smaller population risk than the best single estimator, especially when signal-to-noise ratio is small.
Multi-headed ensembles boost model performance with faster training.
problem Limited computational resources hinder ensemble search performance.
method Extend NES to multi-headed ensembles, leveraging end-to-end training and one-shot NAS methods.
result Multi-headed ensemble search finds robust ensembles 3 times faster with comparable performance.
Empirical study on trends reversion in financial markets.
problem Understanding when trends in financial markets revert.
method Polynomial regression and bootstrapping on 30 years of daily futures prices.
result Trends revert when they reach a critical level of statistical significance.
Ensemble learning is a methodology that integrates multiple DNN learners for improving prediction performance of individual learners. Diversity is greater when the errors of the ensemble prediction is more uniformly distributed. Greater diversity is highly correlated with the increase in ensemble accuracy. Another attr…
We develop a family of techniques to align word embeddings which are derived from different source datasets or created using different mechanisms (e.g., GloVe or word2vec). Our methods are simple and have a closed form to optimally rotate, translate, and scale to minimize root mean squared errors or maximize the averag…
Transforms ensemble predictions to maintain interpretability.
problem Loss of interpretability in deep ensembles.
method Proposes transformation ensembles that aggregate predictions while preserving interpretability.
result Transformation ensembles yield better predictions than individual models and maintain interpretability.
Overparameterized ensembles don't offer generalization benefits over single large models.
problem Theoretical limitations of ensembles in overparameterized settings.
method Using ensembles of random feature (RF) regressors, the paper clarifies how modern ensembles differ from underparameterized counterparts.
result Infinite ensembles of overparameterized RF regressors become pointwise equivalent to single infinite-width RF regressors, and finite width ensembles converge to single models with the same parameter budget.
New method creates diverse neural ensembles for better uncertainty estimation and robustness.
problem Creating more robust neural networks for uncertainty estimation and dataset shift.
method Automatically constructing ensembles with varying architectures.
result Ensembles with varying architectures outperform deep ensembles in accuracy, uncertainty calibration, and robustness.
New method certifies joint adversarial robustness of model ensembles.
problem Ensuring robustness of model ensembles against adversarial attacks.
method Proposes a novel technique to certify joint robustness, building on prior work on single-model robustness certification.
result Demonstrates the effectiveness of certifying joint robustness of ensembles, improving understanding of ensemble defenses.
Natural and social multivariate systems are commonly studied through sets of simultaneous and time-spaced measurements of the observables that drive their dynamics, i.e., through sets of time series. Typically, this is done via hypothesis testing: the statistical properties of the empirical time series are tested again…
Embedded ensembles improve neural network performance efficiently.
problem Improving neural network performance with fewer resources.
method Analyzing the wide network limit of gradient descent dynamics using Neural-Tangent-Kernel.
result Embedded ensembles exhibit two regimes: independent and collective, affecting performance.
A framework preserves uncertainty in ensemble distillation.
problem Preserving uncertainty decomposition in ensemble distillation.
method General framework for distilling both regression and classification ensembles, preserving natural uncertainty decomposition.
result Framework maintains decomposition of predictive uncertainty.
Study pitfalls of deep learning ensembles in uncertainty estimation.
problem Pitfalls in in-domain uncertainty estimation and ensembling in deep learning.
method Exploration of standards for uncertainty quantification and broad study of ensembling techniques.
result Many sophisticated ensembling techniques are equivalent to a simple ensemble of few networks.
New hyperparameter ensembles boost neural network performance and uncertainty.
problem Improving neural network robustness and uncertainty quantification.
method Designing ensembles over both weights and hyperparameters, stratified across random initializations.
result Hyper-deep and hyper-batch ensembles outperform deep and batch ensembles on various architectures.
ECV method optimizes ensemble parameters for randomized ensembles.
problem Efficient tuning of ensemble parameters in randomized ensembles.
method ECV (Extrapolated Cross-Validation) method for tuning ensemble and subsample sizes.
result ECV yields δ-optimal ensembles for squared prediction risk.
The paper explores dynamic ensembles for multi-step forecasting.
problem Lack of research on dynamic ensembles for multi-step forecasting.
method Extensive experiments with 3568 time series and an ensemble of 30 multi-output models.
result Dynamic ensembles based on arbitrating and windowing perform best.
Bayesian deep ensembles improve prediction accuracy in various settings.
problem Improving prediction accuracy of deep ensembles in out-of-distribution settings.
method Introducing a randomised, untrainable function to each ensemble member, enabling a posterior predictive distribution interpretation.
result Bayesian deep ensembles make more conservative predictions and outperform standard ensembles in various tasks.
RCAM-based ensemble combines binary classifiers using similarity and vote scheme.
problem Improving binary classification accuracy through ensemble methods.
method RCAM-based ensemble combining classifiers using similarity and recurrent consult-vote scheme.
result RCAM-based ensemble outperforms individual classifiers and majority voting.
Heterogeneous ensembles built from the predictions of a wide variety and large number of diverse base predictors represent a potent approach to building predictive models for problems where the ideal base/individual predictor may not be obvious. Ensemble selection is an especially promising approach here, not only for …
High-capacity neural network ensembles often benefit more from high-capacity models than from increased diversity.
problem The performance of high-capacity neural network ensembles is often harmed by interventions that promote predictive diversity.
method A large-scale study of nearly 600 neural network classification ensembles, examining various interventions and architectures.
result Discouraging predictive diversity can be benign in large-network ensembles, and higher-capacity models often yield better performance than diverse architectures.
New method reduces ensemble size for linear bandits, achieving near optimal regret.
problem Achieving near optimal regret in linear bandits with limited ensemble size.
method Ensemble sampling with a size of order d log T d \log T d log T for a d d d -dimensional stochastic linear bandit. result Regret is at most ( d log T ) 5 / 2 T (d \log T)^{5/2} \sqrt{T} ( d log T ) 5/2 T , improving over linear scaling with T T T . Jointly tuning ensemble models improves performance and uncertainty calibration.
problem Improving both predictive performance and uncertainty calibration in deep ensembles.
method Investigated the impact of jointly tuning weight decay, temperature scaling, and early stopping.
result Jointly tuning ensemble models generally matches or improves performance, with significant variation across tasks.
This paper reviews weighted clustering ensemble methods.
problem Improving clustering results from individual methods.
method Different types of weights and approaches to determining weight values.
result Unified framework for selecting appropriate weighting mechanisms.
Proposes a new method for deep ensembles that improves accuracy and calibration.
problem Improving accuracy and calibration of deep ensembles.
method Estimates confusion matrices of ensemble members and weighs them according to their inferred performance.
result Empirically shows superiority of soft Dawid Skene over ensemble averaging.
Paper analyzes uncertainty metrics in ensemble learning for healthcare AI.
problem Selecting appropriate uncertainty metrics for ensemble learners in healthcare AI.
method Rigorous analysis of two uncertainty metrics: ensemble mean and variance.
result Ensemble mean is preferable to ensemble variance for decision making in healthcare AI.
Self-paced ensemble learning improves audio classification models.
problem Improving performance of individual models in speech and audio classification.
method A self-paced ensemble learning scheme where models learn from each other over several iterations.
result SPEL significantly outperforms baseline ensemble models.
We propose and evaluate alternative ensemble schemes for a new instance based learning classifier, the Randomised Sphere Cover (RSC) classifier. RSC fuses instances into spheres, then bases classification on distance to spheres rather than distance to instances. The randomised nature of RSC makes it ideal for use in en…
Hydra distills ensemble models into a single model while preserving diversity and uncertainty.
problem Loss of ensemble diversity and uncertainty in distilled models.
method Single multi-headed neural network with shared body network.
result Hydra improves distillation performance and preserves ensemble diversity and uncertainty.
A fast method estimates stability of ensemble feature selectors.
problem Improving stability of ensemble feature selectors for better prediction.
method Simulator of a feature selector to estimate stability.
result Reduces computation time for estimating stability.
Ensemble++ uses shared-factor ensembles to scale Thompson Sampling for linear and nonlinear bandits.
problem Computational challenges in Thompson Sampling for large-scale or non-conjugate settings.
method Ensemble++ with shared-factor architecture and random linear combinations.
result Ensemble++ achieves comparable regret to exact Thompson Sampling with significantly smaller ensemble sizes.
Ensemble unsupervised anomaly detection using IRT for hidden ground truth.
problem Challenges in constructing an ensemble from unsupervised anomaly detection methods.
method Use Item Response Theory to compute latent traits and construct an ensemble that downplays noisy methods.
result Demonstrated effectiveness of IRT ensemble on extensive data repository.
New method improves ensemble quality by exploring the pre-train basin more effectively.
problem Limited diversity in ensembles trained from a single pre-trained checkpoint.
method Proposed StarSSE modification of Snapshot Ensembles for transfer learning.
result Stronger ensembles and uniform model soups achieved.
RobBoost optimizes a deep model ensemble for certified robustness.
problem Improving the robustness of deep models.
method Optimizes a neural network's robustness certificate through model selection and weighting.
result RobBoost forms a more robust ensemble with better certified robustness and clean accuracy.
Analogator learns to make analogies by example.
problem Developing a computer program to learn analogies.
method Connectionist approach using a recurrent network architecture trained to divide scenes into figure and ground.
result Analogator can make new analogies between novel situations.