Fair MP-Boost improves fairness and interpretability in boosting methods.
problem Improving fairness and interpretability in boosting methods.
method Fair MP-Boost uses adaptive sampling of minipatches to balance accuracy and fairness.
result Fair MP-Boost enhances fairness and accuracy while providing interpretable feature importance.
Develops a fast method to learn graph structures from large datasets.
problem Learning graph structures from huge datasets with computational intractability and high complexity.
method Minipatch Graph (MPGraph) estimator: breaks up the problem into minipatches, uses hard thresholding, and integrates hyperparameter tuning.
result Proves graph selection consistency and empirically shows superior accuracy and speed compared to state-of-the-art methods.
A new method for feature importance inference without data splitting.
problem Feature importance inference for machine learning models.
method Minipatch ensembles for model-agnostic, distribution-free inference.
result Asymptotic validity of confidence intervals without data splitting.
IMPACC improves consensus clustering for bioinformatics data.
problem Consensus clustering's inefficiency and lack of interpretability for large-scale data.
method Ensemble minipatch co-occurrences, adaptive sampling of observations and features.
result Significantly improved accuracy and interpretability with substantial computational savings.
New method selects features for big data efficiently.
problem Feature selection challenges in huge data.
method Minipatch learning with STAMPS and AdaSTAMPS.
result AdaSTAMPS outperforms other methods in accuracy and speed.
MP-Boost boosts accuracy faster and more interpretable than AdaBoost.
problem Developing a faster, more interpretable boosting method.
method Adaptive selection of minipatches (small subsets of instances and features) at each iteration.
result Achieves comparable accuracy to AdaBoost and gradient boosting but faster and more interpretable.
RAMPART ranks top-k features more accurately than existing methods.
problem Accurate ranking of important features in machine learning.
method Adaptive sequential halving strategy combined with ensembling techniques.
result RAMPART achieves the correct top-k ranking with high probability.
Two new methods assess feature importance for fairness in machine learning models.
problem Understanding how features influence fairness in machine learning models.
method Two model-agnostic approaches: permutation and occlusion.
result Simple, scalable, and interpretable methods to quantify feature importance for fairness.
Cluster LOCO: A model-agnostic feature importance score for interpreting cluster outputs
problem Interpreting and auditing cluster outputs
method Cluster LOCO (Leave-One-Covariate-Out)
result More reliably recovers informative features than existing methods
Multi-headed ensembles boost model performance with faster training.
problem Limited computational resources hinder ensemble search performance.
method Extend NES to multi-headed ensembles, leveraging end-to-end training and one-shot NAS methods.
result Multi-headed ensemble search finds robust ensembles 3 times faster with comparable performance.
Ensemble learning is a methodology that integrates multiple DNN learners for improving prediction performance of individual learners. Diversity is greater when the errors of the ensemble prediction is more uniformly distributed. Greater diversity is highly correlated with the increase in ensemble accuracy. Another attr…
Transforms ensemble predictions to maintain interpretability.
problem Loss of interpretability in deep ensembles.
method Proposes transformation ensembles that aggregate predictions while preserving interpretability.
result Transformation ensembles yield better predictions than individual models and maintain interpretability.
Overparameterized ensembles don't offer generalization benefits over single large models.
problem Theoretical limitations of ensembles in overparameterized settings.
method Using ensembles of random feature (RF) regressors, the paper clarifies how modern ensembles differ from underparameterized counterparts.
result Infinite ensembles of overparameterized RF regressors become pointwise equivalent to single infinite-width RF regressors, and finite width ensembles converge to single models with the same parameter budget.
New method creates diverse neural ensembles for better uncertainty estimation and robustness.
problem Creating more robust neural networks for uncertainty estimation and dataset shift.
method Automatically constructing ensembles with varying architectures.
result Ensembles with varying architectures outperform deep ensembles in accuracy, uncertainty calibration, and robustness.
New method certifies joint adversarial robustness of model ensembles.
problem Ensuring robustness of model ensembles against adversarial attacks.
method Proposes a novel technique to certify joint robustness, building on prior work on single-model robustness certification.
result Demonstrates the effectiveness of certifying joint robustness of ensembles, improving understanding of ensemble defenses.
Embedded ensembles improve neural network performance efficiently.
problem Improving neural network performance with fewer resources.
method Analyzing the wide network limit of gradient descent dynamics using Neural-Tangent-Kernel.
result Embedded ensembles exhibit two regimes: independent and collective, affecting performance.
A framework preserves uncertainty in ensemble distillation.
problem Preserving uncertainty decomposition in ensemble distillation.
method General framework for distilling both regression and classification ensembles, preserving natural uncertainty decomposition.
result Framework maintains decomposition of predictive uncertainty.
New hyperparameter ensembles boost neural network performance and uncertainty.
problem Improving neural network robustness and uncertainty quantification.
method Designing ensembles over both weights and hyperparameters, stratified across random initializations.
result Hyper-deep and hyper-batch ensembles outperform deep and batch ensembles on various architectures.
ECV method optimizes ensemble parameters for randomized ensembles.
problem Efficient tuning of ensemble parameters in randomized ensembles.
method ECV (Extrapolated Cross-Validation) method for tuning ensemble and subsample sizes.
result ECV yields δ-optimal ensembles for squared prediction risk.
The paper explores dynamic ensembles for multi-step forecasting.
problem Lack of research on dynamic ensembles for multi-step forecasting.
method Extensive experiments with 3568 time series and an ensemble of 30 multi-output models.
result Dynamic ensembles based on arbitrating and windowing perform best.
Bayesian deep ensembles improve prediction accuracy in various settings.
problem Improving prediction accuracy of deep ensembles in out-of-distribution settings.
method Introducing a randomised, untrainable function to each ensemble member, enabling a posterior predictive distribution interpretation.
result Bayesian deep ensembles make more conservative predictions and outperform standard ensembles in various tasks.
RCAM-based ensemble combines binary classifiers using similarity and vote scheme.
problem Improving binary classification accuracy through ensemble methods.
method RCAM-based ensemble combining classifiers using similarity and recurrent consult-vote scheme.
result RCAM-based ensemble outperforms individual classifiers and majority voting.
Heterogeneous ensembles built from the predictions of a wide variety and large number of diverse base predictors represent a potent approach to building predictive models for problems where the ideal base/individual predictor may not be obvious. Ensemble selection is an especially promising approach here, not only for …
High-capacity neural network ensembles often benefit more from high-capacity models than from increased diversity.
problem The performance of high-capacity neural network ensembles is often harmed by interventions that promote predictive diversity.
method A large-scale study of nearly 600 neural network classification ensembles, examining various interventions and architectures.
result Discouraging predictive diversity can be benign in large-network ensembles, and higher-capacity models often yield better performance than diverse architectures.
New method reduces ensemble size for linear bandits, achieving near optimal regret.
problem Achieving near optimal regret in linear bandits with limited ensemble size.
method Ensemble sampling with a size of order d log T d \log T d log T for a d d d -dimensional stochastic linear bandit. result Regret is at most ( d log T ) 5 / 2 T (d \log T)^{5/2} \sqrt{T} ( d log T ) 5/2 T , improving over linear scaling with T T T . Jointly tuning ensemble models improves performance and uncertainty calibration.
problem Improving both predictive performance and uncertainty calibration in deep ensembles.
method Investigated the impact of jointly tuning weight decay, temperature scaling, and early stopping.
result Jointly tuning ensemble models generally matches or improves performance, with significant variation across tasks.
Proposes a new method for deep ensembles that improves accuracy and calibration.
problem Improving accuracy and calibration of deep ensembles.
method Estimates confusion matrices of ensemble members and weighs them according to their inferred performance.
result Empirically shows superiority of soft Dawid Skene over ensemble averaging.
Paper analyzes uncertainty metrics in ensemble learning for healthcare AI.
problem Selecting appropriate uncertainty metrics for ensemble learners in healthcare AI.
method Rigorous analysis of two uncertainty metrics: ensemble mean and variance.
result Ensemble mean is preferable to ensemble variance for decision making in healthcare AI.
Self-paced ensemble learning improves audio classification models.
problem Improving performance of individual models in speech and audio classification.
method A self-paced ensemble learning scheme where models learn from each other over several iterations.
result SPEL significantly outperforms baseline ensemble models.
Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance. At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation. In this work, we focus on in-domain uncertainty for im…
We propose and evaluate alternative ensemble schemes for a new instance based learning classifier, the Randomised Sphere Cover (RSC) classifier. RSC fuses instances into spheres, then bases classification on distance to spheres rather than distance to instances. The randomised nature of RSC makes it ideal for use in en…
A fast method estimates stability of ensemble feature selectors.
problem Improving stability of ensemble feature selectors for better prediction.
method Simulator of a feature selector to estimate stability.
result Reduces computation time for estimating stability.
Ensemble++ uses shared-factor ensembles to scale Thompson Sampling for linear and nonlinear bandits.
problem Computational challenges in Thompson Sampling for large-scale or non-conjugate settings.
method Ensemble++ with shared-factor architecture and random linear combinations.
result Ensemble++ achieves comparable regret to exact Thompson Sampling with significantly smaller ensemble sizes.
Ensemble unsupervised anomaly detection using IRT for hidden ground truth.
problem Challenges in constructing an ensemble from unsupervised anomaly detection methods.
method Use Item Response Theory to compute latent traits and construct an ensemble that downplays noisy methods.
result Demonstrated effectiveness of IRT ensemble on extensive data repository.
New method improves ensemble quality by exploring the pre-train basin more effectively.
problem Limited diversity in ensembles trained from a single pre-trained checkpoint.
method Proposed StarSSE modification of Snapshot Ensembles for transfer learning.
result Stronger ensembles and uniform model soups achieved.
Efficient neural network ensembles improve image classification reliability and uncertainty quantification.
problem Uncertainty in neural network predictions for industrial image classification.
method Investigated efficient neural network ensembles (snapshot, batch, multi-input multi-output) for image classification reliability and uncertainty quantification.
result Batch ensemble is a cost-effective and competitive alternative to deep ensembles, offering savings in training and test time.
Ensembles of deep neural networks significantly improve generalization accuracy. However, training neural network ensembles requires a large amount of computational resources and time. State-of-the-art approaches either train all networks from scratch leading to prohibitive training cost that allows only very small ens…
Paper analyzes ensemble Kalman updates for effective dimension and localization.
problem Why small ensemble sizes work well in inverse problems and data assimilation.
method Non-asymptotic analysis of ensemble Kalman updates, focusing on effective dimension and localization.
result Rigorously explains why a small ensemble size is sufficient when prior covariance has moderate effective dimension.
Ensembling DNNs improves minority group performance, leading to fairness.
problem Improving subgroup performances in DNN classifiers.
method Simple homogeneous ensembling of DNNs.
result Fairness naturally emerges from ensembling, improving minority group performance.
Repulsive deep ensembles improve diversity and Bayesian inference.
problem Challenges in maintaining diversity among ensemble members trained independently.
method Introducing a repulsive term in the update rule of deep ensembles.
result Training dynamics of repulsive ensembles follow a Wasserstein gradient flow of KL divergence with the true posterior.
This research tackles uncertainty in gradient boosting models using ensemble methods.
problem Quantifying uncertainty in gradient boosting models for high-risk applications.
method Probabilistic ensemble-based framework for gradient boosting classification and regression models.
result Ensembles of gradient boosting models detect anomalous inputs but have limited ability to improve total uncertainty.
Fibonacci Ensembles use Fibonacci weights to improve ensemble learning, inspired by natural growth patterns.
problem Improving ensemble learning methods to enhance model performance and interpretability.
method Introduces Fibonacci weights and a recursive ensemble dynamic to reduce variance and enrich representational depth.
result Fibonacci weighting can match or improve upon uniform averaging in ensemble learning experiments.
Ensembling improves performance when classifiers disagree more than average.
problem When do ensembles provide significant performance improvements in classification tasks?
method Theoretical and empirical analysis of ensemble improvement rate and disagreement-error ratio.
result Ensembling improves performance significantly when the disagreement rate is large relative to the average error rate.
Ensembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different \emph{forms} of uncertainty. However, ensembles come at a computational and memory cost which may be…
Ens-CGP synthesizes ensemble-based inference with Gaussian processes.
problem Ensemble-based inference and Gaussian process modeling.
method Formulates Ens-CGP as a conditional Gaussian process for ensemble moments.
result Ens-CGP provides a unified probabilistic foundation for Kalman-type methods.
Ensembles improve classifier performance by reducing bias, not variance.
problem Improving classifier performance through ensemble methods.
method Extended bias-variance decomposition for classification tasks, introducing dual reparameterization.
result Ensembling reduces bias in classifiers, contrary to the traditional view.
Ensembles of models have been empirically shown to improve predictive performance and to yield robust measures of uncertainty. However, they are expensive in computation and memory. Therefore, recent research has focused on distilling ensembles into a single compact model, reducing the computational and memory burden o…
Ensemble models refer to methods that combine a typically large number of classifiers into a compound prediction. The output of an ensemble method is the result of fitting a base-learning algorithm to a given data set, and obtaining diverse answers by reweighting the observations or by resampling them using a given pro…