Proposes a method to improve neural architectures reproducibly.
problem Lack of reproducibility in Neural Architecture Transformer (NAT).
method Differentiable Neural Architecture Transformation (DNAT).
result DNAT outperforms NAT and is applicable to various models and datasets.
DNAS disentangles neural architecture search for better interpretability and performance.
problem Lack of interpretability in existing neural architecture search methods.
method DNAS disentangles the hidden representation of the controller into semantically meaningful concepts.
result DNAS achieves state-of-the-art performance and competitive architectures.
DNArch learns CNN architectures by backpropagation.
problem Discovering optimal CNN architectures.
method Differentiable Neural Architectures (DNArch) learns CNN architectures by backpropagation, controlling kernel sizes, channels, downsampling positions, and depth.
result DNArch finds performant CNN architectures across various tasks.
GATES improves neural architecture search by modeling operations as information transformation.
problem Improving predictor-based neural architecture search efficiency.
method GATES models operations as information transformation, covering both node and edge cell search spaces.
result GATES boosts sample efficiency and improves predictor performance.
Differentially-private FNAS protects privacy while collaboratively searching for neural architectures.
problem Collaborative neural architecture search with privacy concerns.
method Federated Neural Architecture Search (FNAS) with differential privacy (DP-FNAS).
result DP-FNAS can search for highly-performant neural architectures while protecting individual parties' privacy.
New model predicts neural network performance from early training epochs, incorporating architecture impact.
problem Predicting neural network performance from early training epochs, neglecting architecture impact.
method Architecture-aware graph ordinary differential equation model.
result Model outperforms state-of-the-art methods for MLP and CNN learning curves.
A graph VAE framework optimizes neural architectures in a continuous space.
problem Discovering efficient neural architectures in a discrete space.
method Graph VAE framework with VAE and GNN components, joint learning of predictors and decoders.
result The framework discovers powerful neural architectures with both excellent performance and high computational efficiency.
A method for faster neural architecture search using low-fidelity training.
problem Time-consuming evaluations in neural architecture search.
method Bayesian multi-fidelity method with knowledge distillation.
result Training for a few epochs with knowledge distillation leads to better architecture selection.
Graph Metanetworks process diverse neural architectures efficiently.
problem Processing diverse neural architectures efficiently.
method Builds metanetworks using graph neural networks to process graphs representing input neural networks.
result Proves GMNs are expressive and equivariant to parameter permutation symmetries.
Simplified NAS for GNN architectures improves efficiency and expressiveness.
problem Efficient and effective discovery of optimal GNN architectures.
method SNAG framework with a novel search space and reinforcement learning.
result SNAG framework outperforms human-designed and existing NAS methods.
Survey of 1000 NAS papers, automating neural architecture design.
problem Designing high-performing neural architectures for various tasks.
method Taxonomy of search spaces, algorithms, and speedup techniques.
result NAS has surpassed human-designed architectures on many tasks.
Recent studies on neural architecture search have shown that automatically designed neural networks perform as good as expert-crafted architectures. While most existing works aim at finding architectures that optimize the prediction accuracy, these architectures may have complexity and is therefore not suitable being d…
Method detects neural network equivalence via matrix ensembles and spectral analysis.
problem Detecting equivalence among different deep learning architectures.
method Generating Mixed Matrix Ensembles (MMEs) and matching to conjugate circular ensembles.
result Empirical evidence shows vanishing differences in spectral densities with long tail decay rates.
NAS helps find best neural network designs.
problem Designing optimal neural network architectures.
method Optimization algorithms and search spaces.
result Introduction to major advances in NAS for CNNs.
NPENAS improves neural architecture search efficiency and accuracy.
problem Efficient and accurate neural architecture search (NAS) for minimizing search costs.
method Proposes NPENAS, a neural predictor guided evolutionary algorithm that enhances exploration ability of evolutionary algorithms.
result NPENAS-BO and NPENAS-NP outperform existing NAS algorithms on NASBench-201, NASBench-101, and DARTS.
Latest algorithms for automatic neural architecture search perform remarkable but are basically directionless in search space and computational expensive in training of every intermediate architecture. In this paper, we propose a method for efficient architecture search called EENA (Efficient Evolution of Neural Archit…
Contrastive embeddings improve neural architecture search performance.
problem Improving performance of neural architecture search algorithms.
method Contrastive learning to identify networks based on data Jacobians and produce embeddings.
result Traditional black-box optimization algorithms can reach state-of-the-art performance with contrastive embeddings.
NeuralArTS categorizes neural ops in a type system for NAS.
problem Manual optimization of search spaces for NAS is inefficient.
method Developed NeuralArTS, a type system for categorizing network ops.
result NeuralArTS can be applied to convolutional layers.
Study reduces NAS search cost by generating multiple complex architectures in one shot.
problem Finding multiple neural architectures with varying complexities efficiently.
method Uses importance sampling to generate and update multiple distributions of architectures.
result Reduces search cost by finding multiple architectures with different complexities in a single search.
DC-NAS improves neural architecture search by clustering and evaluating sub-networks.
problem Inaccurate evaluation of neural architectures in large search spaces.
method Divide-and-Conquer approach: feature representation, clustering, and evaluation of clusters.
result Achieved 75.1% top-1 accuracy on ImageNet, surpassing state-of-the-art methods.
Graph Neural Networks (GNNs) have been popularly used for analyzing non-Euclidean data such as social network data and biological data. Despite their success, the design of graph neural networks requires a lot of manual work and domain knowledge. In this paper, we propose a Graph Neural Architecture Search method (Grap…
Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameter…
Extends NAS to learn both intra-cell and inter-cell architectures for language modeling.
problem Limited NAS systems restrict search to recurrent or convolutional cells.
method Designs a joint learning method to perform intra-cell and inter-cell NAS simultaneously.
result Significantly outperforms a strong baseline on PTB and WikiText data.
Paper proposes GP-NAS-ensemble for fast neural architecture performance prediction.
problem Estimating neural network performance without training time-consuming evaluations.
method GP-NAS-ensemble framework using ensemble learning improvements.
result Ranked second in a NAS performance prediction challenge.
Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and remarkable capability to reduce the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computations that might not be affordable for researcher…
Improves neural architecture search methods to be more stable and efficient.
problem Neural architecture search methods are unstable and sensitive to hyperparameters.
method Discusses practical considerations to improve stability and efficiency.
result Improves overall performance of neural architecture search methods.
We propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified…
TabNAS improves neural architecture search for tabular datasets by rejecting suboptimal architectures.
problem Finding optimal neural architectures for tabular datasets with resource constraints.
method Develops a reinforcement learning controller motivated by rejection sampling to handle resource constraints.
result TabNAS finds better models that obey resource constraints compared to previous methods.
Paper tackles NAS problem by modeling it as a sparse supernet.
problem Neural Architecture Search (NAS) problem, particularly Mixed-Path Search.
method Model NAS as a sparse supernet with sparsity constraints. Use hierarchical accelerated proximal gradient algorithm for optimization.
result Proposed method finds compact, general, and powerful neural architectures.
Smooth embedding space improves NAS performance.
problem Efficiently predicting good neural architectures.
method Two-sided variational graph autoencoder.
result Smooth embedding space facilitates extrapolation to unseen architectures.
In this paper, a neural architecture search (NAS) framework is proposed for 3D medical image segmentation, to automatically optimize a neural architecture from a large design space. Our NAS framework searches the structure of each layer including neural connectivities and operation types in both of the encoder and deco…
This study uncovers how neural architectures and weights interact in classification tasks.
problem Understanding the role of neural architecture and weights in classification performance.
method Developed a novel method to find optimal task-specific architectures as binary networks with {0, 1}-valued weights, using approximate gradient descent.
result Well-trained architectures may not require fine-tuning of weights, highlighting the importance of structure over weights.
Automatic neural architecture design has shown its potential in discovering powerful neural network architectures. Existing methods, no matter based on reinforcement learning or evolutionary algorithms (EA), conduct architecture search in a discrete space, which is highly inefficient. In this paper, we propose a simple…
SSNAS finds neural architectures without labeled data.
problem Limited labeled data for NAS.
method Self-supervised learning for NAS.
result Comparable results to supervised NAS with labeled data.
Optimal transport kernels improve neural architecture search efficiency.
problem Comparing complex neural architectures similarity using Euclidean metric fails.
method Developed a novel discrepancy using tree-Wasserstein (TW) for neural architectures.
result TW-based approaches outperform other methods in sequential and parallel NAS.
In neural architecture search (NAS), the space of neural network architectures is automatically explored to maximize predictive accuracy for a given task. Despite the success of recent approaches, most existing methods cannot be directly applied to large scale problems because of their prohibitive computational complex…
Neural Architecture Search methods are effective but often use complex algorithms to come up with the best architecture. We propose an approach with three basic steps that is conceptually much simpler. First we train N random architectures to generate N (architecture, validation accuracy) pairs and use them to train a …
New neural network architecture with height adds expressive power.
problem Expressiveness of neural networks limited by width and depth.
method Introduces height as a new hyper-parameter in neural network architecture.
result Neural networks with height achieve significantly better approximation of functions.
Neural Architecture Search has shown potential to automate the design of neural networks. Deep Reinforcement Learning based agents can learn complex architectural patterns, as well as explore a vast and compositional search space. On the other hand, evolutionary algorithms offer higher sample efficiency, which is criti…
Optimizes neural architecture search to generate novel lightweight models.
problem Over-reliance on expert knowledge limits NAS to local optima, preventing architectural breakthroughs.
method Casts NAS as an optimization problem, introduces a hierarchical graph-based search space, and uses Bayesian optimization.
result Generates extremely lightweight yet competitive models on six benchmark datasets.
Designing effective architectures is one of the key factors behind the success of deep neural networks. Existing deep architectures are either manually designed or automatically searched by some Neural Architecture Search (NAS) methods. However, even a well-searched architecture may still contain many non-significant o…
HardCoRe-NAS finds fitting neural networks adhering to hard resource constraints.
problem Finding fitting neural networks that adhere to hard resource constraints.
method Accurate formulation of resource requirement and scalable search method.
result HardCoRe-NAS generates state-of-the-art architectures strictly satisfying hard resource constraints.
Graph neural networks (GNN) has been successfully applied to operate on the graph-structured data. Given a specific scenario, rich human expertise and tremendous laborious trials are usually required to identify a suitable GNN architecture. It is because the performance of a GNN architecture is significantly affected b…
Single-stage neural architecture improves text-to-image synthesis.
problem Combining text and image generation is challenging.
method Used deep residual networks and sentence interpolation.
result Achieved state-of-the-art performance with single-stage training.
In neural architecture search, the structure of the neural network to best model a given dataset is determined by an automated search process. Efficient Neural Architecture Search (ENAS), proposed by Pham et al. (2018), has recently received considerable attention due to its ability to find excellent architectures with…
Architecture design has become a crucial component of successful deep learning. Recent progress in automatic neural architecture search (NAS) shows a lot of promise. However, discovered architectures often fail to generalize in the final evaluation. Architectures with a higher validation accuracy during the search phas…
New neural stack and Turing Machine architectures prove stability and computational power.
problem Designing stable neural network architectures for Turing Machine simulation.
method Introducing neural stack and Turing Machine architectures, proving stability and computational equivalence.
result Differentiable nnTM with bounded neurons can simulate Turing Machine in real-time and is equivalent to UTM.
In this paper we present a statistical analysis about the characteristics that we intend to influence in the performance of the neural networks in terms of assertiveness in the prediction of Brazilian stock returns. We created a population of architectures for analysis and extracted the sample that had the best asserti…