New findings show that minimal feature diversity at neural network initialization is harmful but can be mitigated with noise.
problem The importance of feature diversity at neural network initialization.
method A series of experiments comparing different initialization schemes, including adding noise.
result Minimal feature diversity is harmful but can be mitigated with noise, even standard GPU noise is sufficient.
High-capacity neural network ensembles often benefit more from high-capacity models than from increased diversity.
problem The performance of high-capacity neural network ensembles is often harmed by interventions that promote predictive diversity.
method A large-scale study of nearly 600 neural network classification ensembles, examining various interventions and architectures.
result Discouraging predictive diversity can be benign in large-network ensembles, and higher-capacity models often yield better performance than diverse architectures.
Study 986 diverse networks to reveal structural diversity across domains.
problem Understanding structural diversity in networks across various domains.
method Machine learning techniques (random forest, confusion matrix) on 986 real-world networks and 575 generated networks.
result Networks in the same partition have similar underlying functions, constraints, and generative mechanisms, regardless of their origins.
This paper explores how diverse neural network ensembles improve prediction accuracy and robustness against deception.
problem Improving prediction accuracy and robustness of neural networks against adversarial attacks.
method Examines and measures ensemble diversity, develops algorithms for creating and combining diverse ensembles.
result Greater diversity in neural network ensembles leads to higher accuracy and robustness against deception.
New method shows random, diverse initializations are not essential for deep neural networks.
problem The necessity of random, diverse initializations in deep neural networks.
method Constructed a deep convolutional network with identical features by initializing weights to 0, enabling signal propagation and stable gradients.
result Random, diverse initializations are not necessary for training neural networks.
Neural networks generate their own weights using hypernetworks.
problem Generating diverse and non-trivial weights for neural networks.
method Formulate a compromise between accuracy and diversity, using multi-layered perceptrons for mapping.
result Generated weights are diverse and lie on a non-trivial manifold.
This paper improves ensemble robustness by promoting diversity among individual models.
problem Existing ensemble models are vulnerable to adversarial attacks.
method Introduces a new diversity promoting regularizer to enhance robustness.
result The method improves adversarial robustness while maintaining normal accuracy.
This work explores how XAI methods can visualize the diversity of feature representations in Bayesian Neural Networks.
problem Explaining the diversity of feature representations learned by Bayesian Neural Networks.
method Application of global XAI methods to visualize and quantify the diversity of feature representations.
result The diversity of learned feature representations correlates with uncertainty estimates and network width.
New method reduces overfitting in deep neural networks by measuring and regulating hidden unit diversity.
problem Overfitting in deep neural networks.
method Introduces a new redundancy measure based on mutual information to improve generalization.
result Reduction of redundancy improves generalization capacity, reducing overfitting.
Generative model creates diverse neural network weights efficiently.
problem Creating high-performance and diverse weights for neural networks.
method Trains a hypernetwork mapping latent vectors to high-performance weights, balancing accuracy and diversity.
result Generated weights form a diverse manifold, improving classification accuracy.
PaDGAN generates diverse, high-quality designs with improved performance.
problem Lack of diversity and performance improvement in generated designs.
method Integrates Determinantal Point Processes for diversity and quality, using GAN framework.
result PaDGAN generates higher quality designs with better diversity and without mode collapse.
HyperGAN generates diverse neural network parameters for improved performance and uncertainty.
problem Overconfidence of neural networks in out-of-distribution data.
method Generative model using a novel mixer to learn a distribution of neural network parameters.
result HyperGAN can generate parameters that perform competitively with fully supervised learning and provide better uncertainty estimates.
Paper tackles diversity in Airbnb search results.
problem Show diverse results to improve user experience.
method Heuristic based approaches → Deep learning solution using RNNs.
result Novel deep learning solution improves diversity in search results.
Graph Metanetworks process diverse neural architectures efficiently.
problem Processing diverse neural architectures efficiently.
method Builds metanetworks using graph neural networks to process graphs representing input neural networks.
result Proves GMNs are expressive and equivariant to parameter permutation symmetries.
Enhances sample diversity in SGMCMC for better uncertainty estimation in BNNs.
problem Limited sample diversity in SGMCMC affects uncertainty estimation and model performance.
method Reparameterizes neural network weights to produce a more diverse set of samples.
result The proposed approach achieves superior performance in image classification tasks, including OOD robustness.
MxPool learns graph features from diverse graphs using a hierarchical structure.
problem Learning graph features from diverse graphs with varying properties and sizes.
method MxPool uses a multiplex structure with multiple graph convolution/pooling networks in a hierarchical learning structure.
result MxPool outperforms state-of-the-art methods on graph classification benchmarks.
TimeMCL forecasts diverse time series futures using neural networks and WTA loss.
problem Forecasting multiple plausible time series futures.
method Multiple Choice Learning (MCL) framework with Winner-Takes-All (WTA) loss.
result TimeMCL efficiently predicts diverse time series futures at low computational cost.
New ACE cost function encourages diversity in neural networks.
problem Training multiple classifiers with controlled diversity.
method Mathematical derivation and gradient control.
result ACE yields better ensemble results than vanilla.
Paper perfect clusters sparse, diverse multilayer networks.
problem Clustering sparse, diverse multilayer networks.
method Tensor-based methodology pooling all layers' information.
result Achieves perfect clustering under sparser conditions than previous models.
New findings on representation learning beyond linear functions.
problem Achieving diversity in representation learning beyond linear prediction functions.
method Analysis of eluder dimension and empirical risks.
result Diversity holds even with nonlinear prediction functions and multiple layers in neural networks.
MOD improves ensemble-based uncertainty estimates by encouraging larger diversity.
problem Improving model uncertainty estimates for inputs not seen during training.
method Maximize Overall Diversity (MOD) approach to encourage larger diversity in ensemble predictions.
result Significantly improves predictive performance for out-of-distribution test examples.
Automates neural network design for diverse tasks.
problem Designing neural networks for new, under-explored domains.
method Introduces XD-Operations and a weight-sharing scheme to transform standard backbones into search spaces of operations.
result Models using XD-Operations achieve lower error than baseline and expert-designed networks on diverse tasks.
RNE tackles scalable recommendation for billion-scale scenarios.
problem Designing a scalable recommendation system for diverse and dynamic user interests.
method RNE uses a diversity- and dynamics-aware neighbor sampling method for scalable network embedding.
result RNE achieves high-quality and diverse results on a billion-scale user-item graph.
AI models struggle to generate diverse natural chemical structures.
problem Generating diverse chemical structures for drug discovery.
method Quantified internal chemical diversity; challenge with two models.
result AI models fail to reproduce natural chemical diversity.
Grassmannian packings improve CNN kernels' diversity and reduce sparsity.
problem Kernel sparsity and lack of diversity in CNNs decrease model capacity.
method Initialize CNN kernels with Grassmannian packings to maximize diversity and minimize sparsity.
result Grassmannian packings lead to diverse features and improved classification accuracy.
Proposes a method to generate diverse outputs in conditional GANs.
problem Mode-collapse in conditional GANs, where outputs are overly simplified.
method Explicit regularization to produce diverse outputs based on latent codes.
result Demonstrates improved diversity in image-to-image translation, inpainting, and future video prediction tasks.
BADGE samples diverse and uncertain points for deep neural nets.
problem Efficiently selecting batches for active learning with deep models.
method Samples groups of points that are disparate and high-magnitude in a hallucinated gradient space.
result BADGE consistently performs as well or better than other methods for active learning.
Improved uncertainty estimation through diverse sampling in neural networks.
problem Enhancing uncertainty estimation for machine learning models.
method Data-driven correlations and determinantal point processes-based sampling for dropout layers.
result State-of-the-art results in uncertainty estimation for regression and classification tasks.
Self-distillation improves model performance by increasing teacher diversity and smoothing predictions.
problem Improving model generalization and performance through self-distillation.
method Interpreting self-distillation as MAP estimation and proposing instance-specific label smoothing.
result Self-distillation enhances model performance by increasing teacher diversity and smoothing predictions.
DONNA rapidly finds optimal neural networks across diverse spaces.
problem Efficient scaling and handling of diverse architectural search-spaces in NAS.
method Three-phase pipeline: accuracy predictor, rapid evolutionary search, and optimal model finetuning.
result 100x faster than MNasNet in finding state-of-the-art architectures on-device.
Stacked models improve link prediction across diverse networks.
problem Predicting missing links in complex networks.
method Systematically evaluated 203 link predictor algorithms, combining them into stacked models.
result Stacked models achieve optimal or nearly optimal link prediction accuracy.
Efficiently builds diverse sub-model ensembles for robust self-supervised learning.
problem Challenges in diversity and efficiency of deep ensembles for self-supervised representation learning.
method Ensemble of independent sub-networks with a new loss function for diversity.
result Significantly improves prediction reliability and model calibration.
Improves GANs by enforcing diverse feature learning.
problem GANs can collapse to a single configuration and be unstable.
method Enforces diverse feature learning by penalizing correlated features.
result Enforces diverse features, stabilizes training, and improves image synthesis.
Prediction tasks over nodes and edges in networks require careful effort in engineering features used by learning algorithms. Recent research in the broader field of representation learning has led to significant progress in automating prediction by learning the features themselves. However, present feature learning ap…
New measures quantify how data augmentation improves model performance.
problem Understanding the effectiveness of data augmentation in deep learning.
method Introduced Affinity and Diversity measures to quantify augmentation performance.
result Augmentation performance is best achieved by optimizing both Affinity and Diversity.
Proposes GM Score to evaluate GANs considering diversity, disentanglement, and discriminability.
problem Evaluation of GANs for sample quality and diversity.
method Integrates various factors including intra-class and inter-class diversity, disentanglement, and discriminability metrics.
result Demonstrates improved evaluation of GANs on MNIST dataset.
Neural networks are a powerful class of functions that can be trained with simple gradient descent to achieve state-of-the-art performance on a variety of applications. Despite their practical success, there is a paucity of results that provide theoretical guarantees on why they are so effective. Lying in the center of…
Novel Bayesian neural network method for robustness.
problem Adversarial robustness without online training.
method Distributes uncertainty across all inputs.
result Demonstrates robustness on benchmark datasets.
DeepWeightFlow generates diverse neural network weights efficiently.
problem Generating complete neural network weights efficiently and accurately.
method Flow Matching in weight space with Git Re-Basin and TransFusion.
result DeepWeightFlow generates high-accuracy neural networks without fine-tuning.
Single neural network predicts ImageNet model parameters for faster training.
problem Training diverse ImageNet models requires significant resources and time.
method Trained a neural network to predict ImageNet model parameters and used them for initialization.
result Models initialized with predicted parameters converge faster and achieve competitive performance.
Improves low-shot learning with novel GAN for diverse example generation.
problem Overfitting and forgetting in small data settings.
method Covariance-Preserving Adversarial Augmentation Networks (CPGANs).
result Significant improvement on ImageNet benchmark.
DEN learns diverse tasks to generalize to unseen tasks.
problem Generalization from a diverse set of classification tasks with limited data.
method Three-block architecture: covariate transformation, distribution embedding, and classification.
result DEN outperforms existing methods in various synthetic and real tasks.
Hydra distills ensemble models into a single model while preserving diversity and uncertainty.
problem Loss of ensemble diversity and uncertainty in distilled models.
method Single multi-headed neural network with shared body network.
result Hydra improves distillation performance and preserves ensemble diversity and uncertainty.
Ensemble of diverse CNNs detects and mitigates adversarial attacks.
problem Detecting and defending against adversarial attacks.
method An ensemble of specialized CNNs with a voting mechanism.
result Significant reduction in adversarial attack risk rate.
Deep networks can approximate various activation functions with modest adjustments.
problem Expressive power of deep neural networks with diverse activation functions.
method Approximation of any activation function in set A by ReLU networks with specific scaling factors.
result Approximation of any activation function in a specific subset of A by ReLU networks with (1,1) scaling factors.
PDNAS optimizes GNN architectures for diverse datasets.
problem Inadequate adaptability and combinatorial search space in GNNs.
method Dual architecture search (micro- and macro-architectures) with gradient-based optimization.
result PDNAS finds deeper GNNs with better performance on diverse datasets.
New method learns diverse protein scaffolds for motif design.
problem Designing long, diverse protein scaffolds for specific motifs.
method E(3)-equivariant graph neural network for diffusion modeling.
result First to guarantee conditional sampling from diffusion models.
Proposes a new GAN framework using adversarial dropout to improve sample diversity and stability.
problem Mode collapse in GANs.
method Adversarial dropout in a dynamic ensemble of discriminators.
result Promotes sample diversity and stabilizes training.