IEA improves CNN models by averaging multiple convolutional layers.
problem Improving CNN model accuracy through ensemble learning.
method Replacing single convolutional layers with Inner Average Ensembles (IEA) of multiple convolutional layers.
result CNN models using IEA outperform those with regular convolutional layers.
IENs reduce neural network variance without increasing complexity.
problem Reducing variance in neural networks without increasing model complexity.
method IENs use ensemble parameters during training to reduce variance, removing them during testing.
result IENs reduce network variance by a factor of 1/mL−1, leading to significant error rate decreases. New SVRG and SARAH schemes reduce tuning effort for variance reduction.
problem Optimal performance of SVRG and SARAH requires tuning of parameters.
method Introduces Barzilai-Borwein step sizes, averaging, and adaptive inner loop length.
result Improves SVRG, SARAH, and BB variants' convergence rates and performance.
WASH trains ensembles with shuffled weights to improve accuracy and reduce communication.
problem Training ensembles for weight averaging leads to models converging to different loss basins.
method WASH randomly shuffles a small percentage of weights during training to keep models within the same basin.
result WASH achieves state-of-the-art image classification accuracy with lower communication costs.
Proposes a new method for deep ensembles that improves accuracy and calibration.
problem Improving accuracy and calibration of deep ensembles.
method Estimates confusion matrices of ensemble members and weighs them according to their inferred performance.
result Empirically shows superiority of soft Dawid Skene over ensemble averaging.
Enhances neural network regularization with no extra cost.
problem Improving neural network robustness and generalization.
method Train an ensemble of weight matrices with stochastic regularization and explicitly average outputs.
result Consistent improvement on various image classification tasks.
New findings suggest no ensemble averaging for certain black hole observables.
problem Mystery in AdS/CFT correspondence regarding ensemble averaging of black hole amplitudes.
method Exploring sub-threshold observables in D=3 and proving novel results about hyperbolic geometry. result Connected solutions of Einstein's equations with disconnected boundary never contribute to sub-threshold observables.
We study the price dynamics of stocks traded in the NASDAQ market by considering the statistical properties of an ensemble of stocks traded simultaneously. For each trading day of our database, we study the ensemble return distribution by extracting its first two central moments. According to previous results obtained …
Ensembling improves performance when classifiers disagree more than average.
problem When do ensembles provide significant performance improvements in classification tasks?
method Theoretical and empirical analysis of ensemble improvement rate and disagreement-error ratio.
result Ensembling improves performance significantly when the disagreement rate is large relative to the average error rate.
Fed-ensemble improves FL by averaging predictions from multiple models.
problem Improving generalization in federated learning.
method Random permutations to update K models, averaging predictions.
result Predictions from all K models have the same predictive posterior distribution.
Revises Bayesian model averaging for foundation models.
problem Ensemble pre-trained and lightly-finetuned foundation models for improved classification performance.
method Introduces trainable linear classifiers and computationally cheaper model averaging scheme (OMA).
result Ensembled models can better predict on various datasets.
Geometric Brownian motion (GBM) is a model for systems as varied as financial instruments and populations. The statistical properties of GBM are complicated by non-ergodicity, which can lead to ensemble averages exhibiting exponential growth while any individual trajectory collapses according to its time-average. A com…
Improved gas demand forecasting using ensemble methods.
problem Short-term prediction of gas demand components.
method Nine base forecasters (Ridge Regression, GP, NN, ANN, Torus, LASSO, Elastic Net, RF, SVR) and four ensemble predictors (simple, weighted, subset, SVR aggregation) were evaluated.
result Ensemble predictors outperformed individual base forecasters and TSO predictions.
Deep ensembles mimic Bayesian averaging with learned priors.
problem Quantifying uncertainty in neural networks.
method Showed deep ensembles perform exact Bayesian averaging with an implicitly learned data-dependent prior.
result Deep ensembles are Bayesian and provide an explanation for their strong empirical performance.
Proposes SWA for adversarial training to improve model robustness.
problem Overfitting in adversarial training due to insufficient sample complexity.
method Adversarial training with Stochastic Weight Average (SWA).
result Improves model robustness on CIFAR-10, CIFAR-100, and SVHN datasets.
This paper develops a new theory for ensemble learning beyond variance reduction.
problem Ensemble learning's effectiveness for stable estimators is not fully explained by variance reduction.
method Develops a general weighting theory for ensemble learning, formalizing ensembles as linear operators and introducing geometric and spectral constraints.
result Structured weights can outperform uniform averaging by reshaping approximation geometry and redistributing spectral complexity.
Researchers prove inner product recovery is impossible in latent space models.
problem Recovering inner products in latent space models with random geometric graphs.
method Rate-distortion theory applied to Gaussian or spherical latent locations.
result Impossible to recover inner products if dimensionality exceeds nh(p), matching positive results' conditions. Ensemble CNNs improve mode classification in smartphone travel surveys.
problem Classifying transportation modes from smartphone travel survey data.
method Developed an ensemble of CNN models with different architectures and hyper-parameters, combined using average voting, majority voting, optimal weights, and a Random Forest meta-learner.
result The ensemble method with Random Forest as meta-learner achieved 91.8% accuracy, surpassing other methods.
In modern portfolio theory, the balancing of expected returns on investments against uncertainties in those returns is aided by the use of utility functions. The Kelly criterion offers another approach, rooted in information theory, that always implies logarithmic utility. The two approaches seem incompatible, too loos…
Two new ensemble methods improve CATE estimation across various scenarios.
problem Estimating CATE in clinical trials to understand treatment effects heterogeneity.
method Proposed two ensemble methods: Stacked X-Learner and Consensus Based Averaging (CBA).
result Ensemble methods achieve good performance across diverse scenarios.
Optimizes ensemble weights and hyperparameters for better machine learning model predictions.
problem Improving ensemble model performance through optimal weights and hyperparameters tuning.
method Designing a nested optimization algorithm that tunes hyperparameters and finds optimal ensemble weights, using Bayesian search and a heuristic for diverse base learners.
result The algorithm (GEM-ITH) produces better ensemble model performance across various datasets.
We consider random vectors drawn from a multivariate normal distribution and compute the sample statistics in the presence of non-stationary correlations. For this purpose, we construct an ensemble of random correlation matrices and average the normal distribution over this ensemble. The resulting distribution contains…
clusterBMA combines clustering results from multiple models using Bayesian model averaging.
problem Uncertainty in model selection for clustering.
method Bayesian model averaging to combine results from multiple clustering algorithms.
result ClusterBMA offers probabilistic cluster allocations and quantifies model-based uncertainty.
The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change …
Ensemble methods improve model performance by averaging over subsampled predictors.
problem Understanding the effect of feature subsampling in ensemble methods.
method Fit linear predictors using ordinary least squares on random submatrices of the data matrix.
result The asymptotic risk of an ensemble is equal to the ridge regression risk, optimal for linear predictors.
Researchers combined linear classifiers using score functions and found simple and trimmed averages to be the best combination strategies.
problem Combining linear classifiers using their score functions.
method Two score functions tested; four combination strategies investigated; comparison with majority voting and model averaging.
result Simple and trimmed average combination strategies were the best.
Study on Gaussian ensemble of matrix products with mixed moments computed.
problem Understanding the statistical properties of matrix products of Gaussian matrices.
method Analysis of a multi-Wishart ensemble and enumeration of non-crossing pairings.
result Mixed moments of the product matrix are computed and found to be weighted by Fuss-Catalan numbers at large N. PDA method optimizes neural networks with global convergence rate analysis.
problem Quantitative convergence rate for neural network optimization in mean field regime.
method Particle dual averaging (PDA) method, combining Langevin algorithm and outer loop optimization.
result Established quantitative global convergence for two-layer mean field neural networks.
Vision impairment due to pathological damage of the retina can largely be prevented through periodic screening using fundus color imaging. However the challenge with large scale screening is the inability to exhaustively detect fine blood vessels crucial to disease diagnosis. In this work we present a computational ima…
This paper improves deep learning model consistency through ensemble methods.
problem Consistency and correct-consistency issues in deep learning models.
method Formal definition of consistency and correct-consistency, proving ensemble improvement, proposing dynamic snapshot ensemble method.
result Ensemble methods can improve correct-consistency of deep learning models.
TabSurv adapts tabular neural networks for survival analysis.
problem Survival analysis on tabular data using deep learning methods.
method Adapts modern tabular architectures to survival analysis using Weibull distribution or non-parametric prediction. Optimizes SurvHL histogram loss function.
result TabSurv consistently outperforms classical and deep learning baselines on 10 real-world survival datasets.
New analysis shows Bayesian model averaging is suboptimal under misspecification.
problem Generalization performance of Bayesian model averaging under model misspecification.
method Novel second-order PAC-Bayes bounds to analyze generalization performance.
result New Bayesian-like algorithms with better generalization performance.
We study the price dynamics of stocks traded in a financial market by considering the statistical properties both of a single time series and of an ensemble of stocks traded simultaneously. We use the n stocks traded in the New York Stock Exchange to form a statistical ensemble of daily stock returns. For each tradin…
This work studies a unified approach to ensemble aggregation using likelihood perspective.
problem Density aggregation in machine learning, focusing on improving ensemble predictions.
method Normalized generalized mean of order r in the log-likelihood framework.
result The optimal range for r is [0,1], providing a principled justification for linear and geometric pooling.
Simple mode exploration methods do not improve performance in neural networks.
problem Improving predictive probabilities in neural networks.
method Exploring local regions around diverse solutions using simple methods.
result Simple mode exploration methods do not improve performance.
The paper addresses model averaging and ensembling, providing theoretical and practical insights.
problem Combining least squares estimators from multiple candidate models for improved predictive accuracy.
method Establishes oracle inequalities for Mallows' Cp criterion, proposes a novel Mallows-type MA procedure. result Demonstrates the effectiveness of the proposed Mallows-type MA estimator through numerical experiments.
Logifold improves ensemble machine learning by identifying fuzzy domains.
problem Improving ensemble machine learning accuracy.
method Formulating logifold structure and interpreting local charts of datasets.
result Logifold improves accuracy compared to averaging model outputs.
CNN improves medium-range temperature forecasts with limited resources.
problem Limited computational resources for high-resolution temperature forecasts.
method CNN post-processing with ensemble NWP models for bias correction and spatial downscaling.
result High-resolution (5-km) surface temperature forecasts with lead times up to 5.5 days.
Transforms ensemble predictions to maintain interpretability.
problem Loss of interpretability in deep ensembles.
method Proposes transformation ensembles that aggregate predictions while preserving interpretability.
result Transformation ensembles yield better predictions than individual models and maintain interpretability.
A fast method for Lasso and Logistic Lasso problems.
problem Solving Lasso and Logistic Lasso regression problems efficiently.
method Iterative active set approach using solver updates.
result 31.41 times faster on average for compressed sensing.
RobBoost optimizes a deep model ensemble for certified robustness.
problem Improving the robustness of deep models.
method Optimizes a neural network's robustness certificate through model selection and weighting.
result RobBoost forms a more robust ensemble with better certified robustness and clean accuracy.
Ensemble models improve prediction calibration for mismatched distributions.
problem Calibration issues in deep neural networks with mismatched train and test distributions.
method Simple data augmentation and mixing techniques for ensemble models.
result Improves calibration and accuracy on CIFAR10 and CIFAR100 benchmarks.
Paper proposes dense average network for improved power load forecasting.
problem Improving power load forecasting accuracy to save millions for the power industry.
method Introduces dense average connection and constructs dense average network for power load forecasting.
result Proposed model outperforms existing methods on public datasets.
New method creates diverse neural ensembles for better uncertainty estimation and robustness.
problem Creating more robust neural networks for uncertainty estimation and dataset shift.
method Automatically constructing ensembles with varying architectures.
result Ensembles with varying architectures outperform deep ensembles in accuracy, uncertainty calibration, and robustness.
The paper explores dynamic ensembles for multi-step forecasting.
problem Lack of research on dynamic ensembles for multi-step forecasting.
method Extensive experiments with 3568 time series and an ensemble of 30 multi-output models.
result Dynamic ensembles based on arbitrating and windowing perform best.
CDST improves ensemble prediction by adjusting model weights based on covariates.
problem Improving ensemble prediction accuracy in complex scenarios.
method Covariate-dependent stacking (CDST) with flexible model weights estimated via cross-validation.
result CDST consistently outperforms conventional model averaging methods in complex datasets.
Suppose some classifiers are selected from a set of hypothesis classifiers to form an equally-weighted ensemble that selects a member classifier at random for each input example. Then the ensemble has an error bound consisting of the average error bound for the member classifiers, a term for selectivity that varies fro…
Estimates latent inner products from an anisotropic Gaussian graph with improved spectral method.
problem Recovering latent inner products from an anisotropic Gaussian random geometric graph.
method Doubly centered adjacency matrix, rank-d spectral approximation, Hermite expansion, decoupling argument.
result Estimator achieves mean squared error rate matching state of the art for isotropic case and ill-conditioned covariance matrices.