Multi-headed ensembles boost model performance with faster training.
problem Limited computational resources hinder ensemble search performance.
method Extend NES to multi-headed ensembles, leveraging end-to-end training and one-shot NAS methods.
result Multi-headed ensemble search finds robust ensembles 3 times faster with comparable performance.
Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and remarkable capability to reduce the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computations that might not be affordable for researcher…
New method creates diverse neural ensembles for better uncertainty estimation and robustness.
problem Creating more robust neural networks for uncertainty estimation and dataset shift.
method Automatically constructing ensembles with varying architectures.
result Ensembles with varying architectures outperform deep ensembles in accuracy, uncertainty calibration, and robustness.
Paper proposes GP-NAS-ensemble for fast neural architecture performance prediction.
problem Estimating neural network performance without training time-consuming evaluations.
method GP-NAS-ensemble framework using ensemble learning improvements.
result Ranked second in a NAS performance prediction challenge.
IENs reduce neural network variance without increasing complexity.
problem Reducing variance in neural networks without increasing model complexity.
method IENs use ensemble parameters during training to reduce variance, removing them during testing.
result IENs reduce network variance by a factor of 1/mL−1, leading to significant error rate decreases. In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks. Amazingly, this model family admits tractable calculations for many information-theoretic quantities. We report analytical and empirical investigations in the search for signals that correlate with…
Ensembles of neural networks improve training dynamics and performance.
problem Improving neural network performance through model size increase.
method Defining collegial ensembles (CE) as multiple independent models trained as a single model, and using theoretical results on NTK to optimize architecture search.
result CE dynamics simplify and scale favorably, resembling wide models, and can be efficiently implemented using group convolutions and block diagonal layers.
We present a novel framework of knowledge distillation that is capable of learning powerful and efficient student models from ensemble teacher networks. Our approach addresses the inherent model capacity issue between teacher and student and aims to maximize benefit from teacher models during distillation by reducing t…
Current Neural Architecture Search techniques can suffer from a few shortcomings, including high computational cost, excessive bias from the search space, conceptual complexity or uncertain empirical benefits over random search. In this paper, we present ImmuNeCS, an attempt at addressing these issues with a method tha…
Bayesian Quadrature improves ensembling for neural networks with dispersed likelihood peaks.
problem Ensembling neural networks struggles with dispersed, narrow peaks in likelihood surfaces.
method Uses Bayesian Quadrature to construct weighted ensembles of architectures.
result Empirically outperforms state-of-the-art baselines in test likelihood, accuracy, and expected calibration error.
Tree ensemble kernels improve Bayesian optimization for mixed features and constraints.
problem Optimizing over mixed-feature spaces with known constraints.
method Kernel interpretation of tree ensembles as Gaussian Process prior, compatible optimization formulation for acquisition function, integration of known constraints.
result Framework outperforms state-of-the-art methods for mixed-feature spaces and constraints.
Finding the best neural network architecture requires significant time, resources, and human expertise. These challenges are partially addressed by neural architecture search (NAS) which is able to find the best convolutional layer or cell that is then used as a building block for the network. However, once a good buil…
Method detects neural network equivalence via matrix ensembles and spectral analysis.
problem Detecting equivalence among different deep learning architectures.
method Generating Mixed Matrix Ensembles (MMEs) and matching to conjugate circular ensembles.
result Empirical evidence shows vanishing differences in spectral densities with long tail decay rates.
This paper explores BDL hyperparameters for robust polynomial mapping with noise.
problem Designing BDL hyperparameters for robust function mapping with uncertainty quantification.
method Mapping Bayesian connectionist representations to polynomials of varying orders and noise types.
result Optimal network depth and ensemble size for prediction and uncertainty quantification.
New hyperparameter ensembles boost neural network performance and uncertainty.
problem Improving neural network robustness and uncertainty quantification.
method Designing ensembles over both weights and hyperparameters, stratified across random initializations.
result Hyper-deep and hyper-batch ensembles outperform deep and batch ensembles on various architectures.
This paper introduces TNTK to study infinite soft tree ensembles.
problem Understanding the behavior of infinite soft tree ensembles.
method Introduced Tree Neural Tangent Kernel (TNTK) to analyze infinite soft tree ensembles.
result Identified several non-trivial properties of infinite soft tree ensembles.
Neural architecture search (NAS) recently attracts much research attention because of its ability to identify better architectures than handcrafted ones. However, many NAS methods, which optimize the search process in a discrete search space, need many GPU days for convergence. Recently, DARTS, which constructs a diffe…
AgEBO-Tabular combines NAS and hyperparameter tuning for fast, high-performing tabular models.
problem Developing high-performing predictive models for large tabular data sets is challenging.
method Combines aging evolution NAS and asynchronous Bayesian optimization for hyperparameter tuning in data-parallel training.
result Automatically discovered neural network models outperform state-of-the-art AutoML ensembles in inference speed by two orders of magnitude.
NADS improves OoD detection accuracy by 57%.
problem Uncertainty in machine learning models when encountering out-of-distribution data.
method NADS searches for a distribution of architectures that perform well on a given task, optimizing a stochastic OoD detection objective.
result NADS achieves up to 57% improvement in accuracy over state-of-the-art methods.
AdaNet is a lightweight TensorFlow-based (Abadi et al., 2015) framework for automatically learning high-quality ensembles with minimal expert intervention. Our framework is inspired by the AdaNet algorithm (Cortes et al., 2017) which learns the structure of a neural network as an ensemble of subnetworks. We designed it…
This work explores how neural architecture search can improve adversarial robustness without adversarial training.
problem Improving adversarial robustness of neural networks without adversarial training.
method Experimented with hand-crafted and NAS-based architectures to compare robustness to PGD attacks.
result NAS-based architectures are more robust for small-scale attacks, but hand-crafted architectures are more robust for larger datasets and tasks.
In the travel industry, online customers book their travel itinerary according to several features, like cost and duration of the travel or the quality of amenities. To provide personalized recommendations for travel searches, an appropriate segmentation of customers is required. Clustering ensemble approaches were dev…
For network architecture search (NAS), it is crucial but challenging to simultaneously guarantee both effectiveness and efficiency. Towards achieving this goal, we develop a differentiable NAS solution, where the search space includes arbitrary feed-forward network consisting of the predefined number of connections. Be…
Bayesian optimization with preference learning using monotonic neural networks.
problem Optimizing complex systems with multiple conflicting objectives.
method Proposes a neural network ensemble for utility surrogate modeling, leveraging monotonicity.
result Demonstrates superior performance compared to existing methods.
A new stochastic method handles ensemble creation with cost constraints.
problem Creating ensembles under cost limitations in decision-making.
method Introducing a novel stochastic approach to solve the knapsack problem.
result The approach efficiently incorporates ensemble accuracy and cost constraints.
Gradient Boosting Decision Tree (GBDT) are popular machine learning algorithms with implementations such as LightGBM and in popular machine learning toolkits like Scikit-Learn. Many implementations can only produce trees in an offline manner and in a greedy manner. We explore ways to convert existing GBDT implementatio…
K-nearest neighbor (kNN) search has wide applications in many areas, including data mining, machine learning, statistics and many applied domains. Inspired by the success of ensemble methods and the flexibility of tree-based methodology, we propose random projection forests (rpForests), for kNN search. rpForests finds …
New framework learns interpretable rule ensembles without sacrificing accuracy.
problem Trade-off between accuracy and interpretability in rule ensembles.
method Introduces local interpretability and a regularizer to promote it, using coordinate descent with local search.
result Learns rule ensembles with fewer rules to explain individual predictions, maintaining comparable accuracy.
MBExplainer provides explanations for models combining graph embeddings and tabular features.
problem Explaining models using a mix of graph embeddings and tabular features.
method Model-agnostic approach using Shapley values and Monte Carlo Tree Search.
result MBExplainer efficiently finds human-readable explanations for model predictions.
Deep Neural Networks (DNNs) often rely on very large datasets for training. Given the large size of such datasets, it is conceivable that they contain certain samples that either do not contribute or negatively impact the DNN's optimization. Modifying the training distribution in a way that excludes such samples could …
Talent search and recommendation systems at LinkedIn strive to match the potential candidates to the hiring needs of a recruiter or a hiring manager expressed in terms of a search query or a job posting. Recent work in this domain has mainly focused on linear models, which do not take complex relationships between feat…
This paper surveys various methods for dimensionality reduction and nearest neighbor search.
problem Efficiently reducing high-dimensional data to lower dimensions while preserving essential information.
method Linear and nonlinear random projections, including sparse random projections, random Fourier Features, and Random Kitchen Sinks.
result Various methods for dimensionality reduction and nearest neighbor search are explained and compared.
Paper optimizes trading profits by predicting price direction using ensemble models.
problem Predicting short-term price movements in financial markets.
method Developed a two-layer ensemble architecture optimized through grid search.
result Improvement of 20% over benchmark strategies.
This paper develops a novel stochastic tree ensemble method for nonlinear regression, which we refer to as XBART, short for Accelerated Bayesian Additive Regression Trees. By combining regularization and stochastic search strategies from Bayesian modeling with computationally efficient techniques from recursive partiti…
We propose a generalized class of multimodal fusion operators for the task of visual question answering (VQA). We identify generalizations of existing multimodal fusion operators based on the Hadamard product, and show that specific non-trivial instantiations of this generalized fusion operator exhibit superior perform…
Improved hyperparameter optimization using simplified Transformer blocks.
problem Discovering optimal architectures in high-dimensional search spaces with limited exploration budgets.
method Simplified Transformer block for modeling hyper-parameter dependencies, actor-critic style algorithm, ensembling.
result Outperformed most algorithms on NAS-Bench-101 and Random Search in discovering more accurate model architectures.
High-capacity neural network ensembles often benefit more from high-capacity models than from increased diversity.
problem The performance of high-capacity neural network ensembles is often harmed by interventions that promote predictive diversity.
method A large-scale study of nearly 600 neural network classification ensembles, examining various interventions and architectures.
result Discouraging predictive diversity can be benign in large-network ensembles, and higher-capacity models often yield better performance than diverse architectures.
Embedded ensembles improve neural network performance efficiently.
problem Improving neural network performance with fewer resources.
method Analyzing the wide network limit of gradient descent dynamics using Neural-Tangent-Kernel.
result Embedded ensembles exhibit two regimes: independent and collective, affecting performance.
Bayesian Neural Networks improve geophysical model ensembles with reduced uncertainty.
problem Improving geophysical model projections and uncertainty quantification.
method Developed a Bayesian Neural Network ensemble strategy for geophysical models.
result Bayesian Neural Network ensemble outperforms existing methods in ozone prediction.
Bayesian deep ensembles improve prediction accuracy in various settings.
problem Improving prediction accuracy of deep ensembles in out-of-distribution settings.
method Introducing a randomised, untrainable function to each ensemble member, enabling a posterior predictive distribution interpretation.
result Bayesian deep ensembles make more conservative predictions and outperform standard ensembles in various tasks.
Quality assessments of models in unsupervised learning and clustering verification in particular have been a long-standing problem in the machine learning research. The lack of robust and universally applicable cluster validity scores often makes the algorithm selection and hyperparameter evaluation a tough guess. In t…
Neural networks learn from ensemble forecasts without considering their order.
problem Improving reliability of probabilistic weather forecasts.
method Permutation-invariant neural networks for postprocessing ensemble forecasts.
result Models achieve state-of-the-art prediction quality in surface temperature and wind gust forecasts.
Aggregating multiple learners through an ensemble of models aim to make better predictions by capturing the underlying distribution of the data more accurately. Different ensembling methods, such as bagging, boosting, and stacking/blending, have been studied and adopted extensively in research and practice. While baggi…
A hybrid ML model detects fraudulent transactions with high accuracy.
problem Detecting and preventing fraudulent credit card transactions.
method Intelligent combination of multiple algorithms with Grid search and IHT-LR.
result Achieves impressive accuracy rates of 99.66% for ENS model.
Ensembles of neural networks learn better by sharing information.
problem Improving performance of neural networks through collective learning.
method Modeling neural networks as socially interacting agents aiming to maximize their own performance and functional relations to others.
result Optimal collective performance emerges from local interactions between networks, leading to specialization and higher confidence.
Paper proposes ECOC for deep neural network ensembles to improve performance.
problem Designing an ensemble of deep networks is time-consuming and often not beneficial.
method ECOC framework applied to deep networks, with design strategies to balance accuracy and complexity.
result Proposed combinatory technique achieves highest classification performance.
Ensembles of deep neural networks significantly improve generalization accuracy. However, training neural network ensembles requires a large amount of computational resources and time. State-of-the-art approaches either train all networks from scratch leading to prohibitive training cost that allows only very small ens…
Single neural networks can match deep ensembles' benefits without the complexity.
problem The effectiveness and necessity of deep ensembles in neural network models.
method Demonstrated limitations of ensemble diversity and OOD performance in deep ensembles compared to a single larger model.
result A single neural network can replicate deep ensembles' benefits in uncertainty quantification and robustness.