New model closes gap in understanding equivariant set functions.
problem Understanding universality of equivariant set functions.
method Proves PointNet not equivariant universal and introduces PointNetST.
result PointNetST is the simplest permutation equivariant universal model.
A new causal deepset framework improves off-policy evaluation under complex interference.
problem Handling spatio-temporal interference in off-policy evaluation.
method Permutation invariance assumption and novel algorithms incorporating it.
result Significantly more precise estimations than existing methods.
Neural net reconstructs dark matter density from halo velocities.
problem Reconstructing local dark matter density from halo velocities.
method Hybrid architecture combining U-Net and DeepSets.
result Hybrid network recovers density amplitudes and phases better than U-Net.
We present a novel family of deep neural architectures, named partially exchangeable networks (PENs) that leverage probabilistic symmetries. By design, PENs are invariant to block-switch transformations, which characterize the partial exchangeability properties of conditionally Markovian processes. Moreover, we show th…
Dataset2Vec learns dataset meta-features without expert knowledge.
problem Learning meta-features for datasets requires expert domain knowledge.
method Dataset2Vec combines DeepSet architecture with hierarchical sets to learn meta-features.
result Meta-features learned by Dataset2Vec outperform engineered features.
Fishnets improve set and graph learning with scalable, robust aggregation.
problem Learning informative embeddings for sets and graphs with scalable and robust aggregation.
method Proposes Fishnets, a new aggregation strategy for set-based learning.
result Fishnets achieve state-of-the-art performance on graph datasets with fewer parameters and faster training.
Neural Local Wasserstein Regression models distribution-on-distribution regression with flexible, localized transport maps.
problem Estimating distribution-on-distribution regression with global optimal transport maps or linearization limitations.
method Proposes Neural Local Wasserstein Regression, a flexible nonparametric framework using locally defined transport maps in Wasserstein space.
result Demonstrates effective capture of nonlinear and high-dimensional distributional relationships.
New deep learning methods solve symmetric PDEs efficiently.
problem Solving nonlinear symmetric PDEs in high dimensions.
method Design of PointNet and DeepSet neural networks.
result DeepSet networks provide more accurate solutions and gradients.
Global pooling, such as max- or sum-pooling, is one of the key ingredients in deep neural networks used for processing images, texts, graphs and other types of structured data. Based on the recent DeepSets architecture proposed by Zaheer et al. (NIPS 2017), we introduce a Set Aggregation Network (SAN) as an alternative…
End-to-end multi-object tracking learns object interactions.
problem Object tracking ignores interactions between objects.
method End-to-end relational reasoning model MOHART.
result Relational reasoning improves tracking and prediction.
META2 improves taxonomic classification and abundance estimation in metagenomics with deep learning and memory efficiency.
problem Memory constraints and inefficiencies in taxonomic classification and abundance estimation for metagenomics.
method Developed a novel memory-efficient read classification technique combining deep learning and locality-sensitive hashing, and formulated abundance estimation as a Multiple Instance Learning problem.
result Our approach outperforms conventional methods in both single-read taxonomic classification and abundance estimation, especially when memory is limited.
New model learns multisets to predict containment and sizes of differences.
problem Learning permutation invariant representations for flexible containment.
method Formalize multisets, propose training on predicting symmetric difference sizes.
result Model outperforms DeepSets on predicting containment and sizes of symmetric differences.
SPAN learns functions over sets invariant to permutations, outperforming existing methods.
problem Learning functions over sets invariant to permutations.
method SPAN architecture that combines neural networks with adversarial permutations.
result SPAN achieves nearly permutation-invariant functions while maintaining accuracy.
New basis for permutation equivariant layers reduces computation costs.
problem Efficiently computing permutation equivariant layers in neural networks.
method Generalized partition algebra basis with low-rank tensors.
result Low-rank tensors enable faster computation compared to orbit basis.
Study improves neural network generalization for invariant and equivariant data.
problem Developing a generalization theory for invariant and equivariant neural networks.
method Introducing quotient feature spaces to measure the effect of group actions on properties and proving a generalization error bound.
result The volume of quotient feature spaces can describe the generalization error and invariance/equivariance significantly improve the bound.
Proposes a method to improve neural architectures reproducibly.
problem Lack of reproducibility in Neural Architecture Transformer (NAT).
method Differentiable Neural Architecture Transformation (DNAT).
result DNAT outperforms NAT and is applicable to various models and datasets.
HM-NAS improves neural architecture search by learning optimal architectures.
problem Limited flexibility in architecture candidates due to hand-designed heuristics.
method Incorporates multi-level encoding and hierarchical masking to automatically learn optimal architectures.
result Achieves better architecture search performance and competitive model accuracy.
Simpler neural architecture search method using random architectures and regression.
problem Complex algorithms in neural architecture search.
method Train N random architectures, use them to train a regression model, predict validation accuracies, and select top-K architectures.
result More sample efficient and competitive with complex approaches.
NAT optimizes neural architectures to improve performance without extra cost.
problem Redundant operations in neural architectures consume memory and degrade performance.
method Transformed Markov Decision Process (MDP) and reinforcement learning to replace redundant operations with more efficient ones.
result Transformed architectures outperform original and existing methods on CIFAR-10 and ImageNet datasets.
DNAS disentangles neural architecture search for better interpretability and performance.
problem Lack of interpretability in existing neural architecture search methods.
method DNAS disentangles the hidden representation of the controller into semantically meaningful concepts.
result DNAS achieves state-of-the-art performance and competitive architectures.
Conventional Neural Architecture Search (NAS) aims at finding a single architecture that achieves the best performance, which usually optimizes task related learning objectives such as accuracy. However, a single architecture may not be representative enough for the whole dataset with high diversity and variety. Intuit…
SAEP prunes sub-architectures to reduce search cost while maintaining performance.
problem Redundancy in ensemble sub-architectures leads to high computational cost.
method SAEP leverages diversity to prune sub-architectures, reducing ensemble size.
result SAEP reduces the number of sub-architectures without degrading performance.
City-GAN learns city architecture styles using a custom GAN architecture.
problem Learning architectural styles of cities using standard GAN and CGAN architectures.
method Proposes a custom GAN architecture to improve learning of city architectural styles.
result Demonstrates superior performance of custom GAN architecture for city architectural style learning.
Study reduces NAS search cost by generating multiple complex architectures in one shot.
problem Finding multiple neural architectures with varying complexities efficiently.
method Uses importance sampling to generate and update multiple distributions of architectures.
result Reduces search cost by finding multiple architectures with different complexities in a single search.
SGAS improves neural architecture search by choosing and pruning operations greedily.
problem NAS often fails to generalize in final evaluation.
method Divides search into sub-problems and chooses/prunes candidate operations greedily.
result SGAS finds state-of-the-art architectures with minimal computational cost.
Deep learning models' architectures, including depth and width, are key factors influencing models' performance, such as test accuracy and computation time. This paper solves two problems: given computation time budget, choose an architecture to maximize accuracy, and given accuracy requirement, choose an architecture …
DNArch learns CNN architectures by backpropagation.
problem Discovering optimal CNN architectures.
method Differentiable Neural Architectures (DNArch) learns CNN architectures by backpropagation, controlling kernel sizes, channels, downsampling positions, and depth.
result DNArch finds performant CNN architectures across various tasks.
Differentially-private FNAS protects privacy while collaboratively searching for neural architectures.
problem Collaborative neural architecture search with privacy concerns.
method Federated Neural Architecture Search (FNAS) with differential privacy (DP-FNAS).
result DP-FNAS can search for highly-performant neural architectures while protecting individual parties' privacy.
NAS favors wide and shallow cell structures, leading to fast convergence but not necessarily better generalization.
problem Understanding and improving the architectures generated by NAS algorithms.
method Empirical and theoretical study of existing NAS algorithms (DARTS, ENAS) and their architectures.
result Existing NAS algorithms favor wide and shallow cell structures, leading to fast convergence but not necessarily better generalization.
Improved neural architecture search through balanced training.
problem Inconsistent ranking of architectures under one-shot training.
method Balanced NAO: balanced training of supernet during search.
result Significant improvements in architecture discovery and performance.
AGNN automates GNN architecture search, achieving best performance.
problem Finding optimal GNN architectures is laborious and requires human expertise.
method AGNN uses reinforcement learning to search for optimal GNN architectures within a predefined space, with a novel parameter sharing strategy.
result AGNN identifies optimal GNN architectures achieving best performance.
PDNAS optimizes GNN architectures for diverse datasets.
problem Inadequate adaptability and combinatorial search space in GNNs.
method Dual architecture search (micro- and macro-architectures) with gradient-based optimization.
result PDNAS finds deeper GNNs with better performance on diverse datasets.
A graph VAE framework optimizes neural architectures in a continuous space.
problem Discovering efficient neural architectures in a discrete space.
method Graph VAE framework with VAE and GNN components, joint learning of predictors and decoders.
result The framework discovers powerful neural architectures with both excellent performance and high computational efficiency.
Neural architecture search has been shown to hold great promise towards the automation of deep learning. However in spite of its potential, neural architecture search remains quite costly. To this point, we propose a novel gradient-based framework for efficient architecture search by sharing information across several …
Bayesian optimization reduces RNN architecture search time.
problem Optimizing RNN architectures for time series prediction.
method Bayesian Optimization with a training-free performance metric.
result BO designs well-performing RNN architectures with reduced optimization time.
GATES improves neural architecture search by modeling operations as information transformation.
problem Improving predictor-based neural architecture search efficiency.
method GATES models operations as information transformation, covering both node and edge cell search spaces.
result GATES boosts sample efficiency and improves predictor performance.
Flat learning curves reveal no progress in ENAS controller.
problem Improving learning speed in neural architecture search.
method Evaluated learning progress of ENAS controller through architecture re-training.
result No observable progress in controller's generated architectures.
The process of designing neural architectures requires expert knowledge and extensive trial and error. While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components. We propose a domain-s…
A ranking network improves neural architecture search efficiency.
problem Efficiently search for neural network architectures without extensive training.
method Pairwise ranking loss for a performance predictor trained on task meta-features.
result The ranking network outperforms the performance predictor in architecture search.
SemiNAS reduces NAS cost by predicting accuracy of unlabeled architectures.
problem Costly evaluation of architectures limits NAS efficiency.
method SemiNAS uses unlabeled architectures to train an accuracy predictor.
result SemiNAS achieves comparable accuracy with less data.
This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architectu…
Generative Adversarial Networks (GANs) are a powerful class of generative models. Despite their successes, the most appropriate choice of a GAN network architecture is still not well understood. GAN models for image synthesis have adopted a deep convolutional network architecture, which eliminates or minimizes the use …
New model predicts neural network performance from early training epochs, incorporating architecture impact.
problem Predicting neural network performance from early training epochs, neglecting architecture impact.
method Architecture-aware graph ordinary differential equation model.
result Model outperforms state-of-the-art methods for MLP and CNN learning curves.
Gradient-based method prunes large models to create transferable architectures.
problem Creating transferable architectures from large models with limited fine-tuning data.
method Gradient-based algorithm for architecture pruning and subset selection.
result Successfully retrain architectures on new tasks with few fine-tuning data.
Latest algorithms for automatic neural architecture search perform remarkable but are basically directionless in search space and computational expensive in training of every intermediate architecture. In this paper, we propose a method for efficient architecture search called EENA (Efficient Evolution of Neural Archit…
BNAS improves neural architecture search with a scalable, fast, and efficient approach.
problem Efficiently searching for optimal neural architectures with high performance and low training time.
method Designing a broad scalable architecture (BCNN) with reinforcement learning and parameter sharing, and developing two variants.
result Significantly reduces training time and achieves state-of-the-art performance on CIFAR-10 and ImageNet.
Recent studies on neural architecture search have shown that automatically designed neural networks perform as good as expert-crafted architectures. While most existing works aim at finding architectures that optimize the prediction accuracy, these architectures may have complexity and is therefore not suitable being d…
The paper optimizes dynamic scheduling for ring architectures in deep learning training.
problem Optimizing deep learning training times with ring architectures.
method Formulated a non-convex, non-linear, NP-hard integer programming problem and developed a doubling heuristic.
result Dynamic scheduling can significantly reduce job completion times in ring architectures.