NAS-Bench-Suite simplifies NAS evaluation across diverse tasks.
problem Limited and inconsistent NAS benchmarks hinder research reproducibility.
method Developed a comprehensive, extensible NAS benchmark suite.
result Many NAS conclusions do not generalize across different benchmarks.
Study on encoding neural architectures for NAS, showing impact on performance.
problem Impact of different encodings on NAS performance.
method Formal definition and characterization of encodings, empirical study of encodings' effectiveness.
result Different encodings can significantly affect NAS performance.
Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks. Existing techniques rely on two stages: searching over the architecture space and validating the best architecture. NAS algorithms are currently compared solely based on their results on the downstream task. While intuitive, …
Hill-climbing is a powerful baseline for NAS, even with reduced noise.
problem Noise in evaluating neural architectures.
method Hill-climbing algorithm, denoising training pipeline.
result Hill-climbing outperforms state-of-the-art NAS algorithms with reduced noise.
New benchmarks provide full training data for NAS research.
problem Limited training data on popular benchmarks restricts multi-fidelity techniques.
method SVD and noise modeling to create surrogate benchmarks with full training info.
result Learning curve extrapolation framework improves single-fidelity algorithms.
BS-NAS broadens and shrinks search space for optimal neural architectures.
problem Suboptimal channel numbers and model averaging effects in One-Shot NAS methods.
method Broadening with spring block for channel search, shrinking with underperforming operations removal, evolutionary algorithm for optimal architecture search.
result BS-NAS achieves state-of-the-art performance on ImageNet.
A framework benchmarks and dissects one-shot neural architecture search methods.
problem Understanding the dynamics and components of one-shot NAS methods.
method General framework for one-shot NAS, benchmarking with NAS-Bench-101.
result Comparison of one-shot NAS methods and their sensitivity to hyperparameters.
NATS-Bench benchmarks NAS algorithms for architecture topology and size.
problem Incomparable performance of NAS algorithms due to different search spaces and training setups.
method Unified benchmarking platform for architecture topology and size searching.
result Validated benchmark for 15,625 topology and 32,768 size candidates.
This paper compares Grid Search, Random Search, and Genetic Algorithm for NAS.
problem Hyperparameter optimization for neural architecture search.
method Comparison of Grid Search, Random Search, and Genetic Algorithm.
result Genetic Algorithm outperforms Grid Search and Random Search in terms of accuracy and execution time.
New algorithms improve neural architecture search with faster convergence.
problem Improving efficiency and accuracy of neural architecture search.
method Geometry-aware gradient algorithms to optimize continuous relaxation of discrete search spaces.
result Exceeds state-of-the-art results on CIFAR and ImageNet benchmarks.
Scalable NAS by factorizing operators into subspaces.
problem Scaling up NAS search space while avoiding operator competition.
method Factorizing a large set of candidate operators into smaller subspaces.
result Achieved state-of-the-art performance on CIFAR10 and ImageNet.
Neural architecture search (NAS) searches architectures automatically for given tasks, e.g., image classification and language modeling. Improving the search efficiency and effectiveness have attracted increasing attention in recent years. However, few efforts have been devoted to understanding the generated architectu…
Paper evaluates robustness of NAS against poisoning attacks.
problem Robustness of Neural Architecture Search (NAS) against poisoning attacks.
method Evaluation of Efficient NAS (ENAS) against carefully designed ineffective operations in poisoning attacks.
result Demonstrates how poisoning attacks exploit design flaws in ENAS controller.
Study accelerates NAS research with a large dataset of ZC proxies.
problem Speeding up neural architecture search with ZC proxies.
method Created NAS-Bench-Suite, evaluated 13 ZC proxies across 28 tasks, and provided a unified codebase.
result ZC proxies capture substantial complementary information and can improve NAS algorithm performance.
Enhanced tabular benchmarks for energy-efficient neural architecture search.
problem Energy consumption in deep learning models.
method Introducing EC-NAS, an enhanced tabular benchmark with energy consumption data.
result EC-NAS reveals a balance between energy usage and accuracy in neural architecture search.
Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) …
Creating efficient deep neural networks involves repetitive manual optimization of the topology and the hyperparameters. This human intervention significantly inhibits the process. Recent publications propose various Neural Architecture Search (NAS) algorithms that automate this work. We have applied a customized NAS a…
Automates neural network design without training, speeding up search by seconds.
problem Time and effort in hand-designing deep neural networks.
method Predict trained accuracy from untrained network state using activation overlap.
result Search for powerful networks in seconds, verified on various benchmarks.
NAS helps find best neural network designs.
problem Designing optimal neural network architectures.
method Optimization algorithms and search spaces.
result Introduction to major advances in NAS for CNNs.
Survey tackles challenges in neural architecture design.
problem Challenges in designing optimal neural architectures.
method New classification perspective based on early NAS algorithms' characteristics, problems, and solutions.
result Comprehensive analysis and comparison of NAS works.
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public arc…
Deep learning (DL) techniques have penetrated all aspects of our lives and brought us great convenience. However, building a high-quality DL system for a specific task highly relies on human expertise, hindering the applications of DL to more areas. Automated machine learning (AutoML) becomes a promising solution to bu…
Paper proposes GP-NAS-ensemble for fast neural architecture performance prediction.
problem Estimating neural network performance without training time-consuming evaluations.
method GP-NAS-ensemble framework using ensemble learning improvements.
result Ranked second in a NAS performance prediction challenge.
NPENAS improves neural architecture search efficiency and accuracy.
problem Efficient and accurate neural architecture search (NAS) for minimizing search costs.
method Proposes NPENAS, a neural predictor guided evolutionary algorithm that enhances exploration ability of evolutionary algorithms.
result NPENAS-BO and NPENAS-NP outperform existing NAS algorithms on NASBench-201, NASBench-101, and DARTS.
While recent NAS algorithms are thousands of times faster than the pioneering works, it is often overlooked that they use fewer candidate operations, resulting in a significantly smaller search space. We present PR-DARTS, a NAS algorithm that discovers strong network configurations in a much larger search space and a s…
Bonsai-Net efficiently discovers state-of-the-art models with fewer parameters.
problem Efficiently discovering state-of-the-art neural architectures with minimal computational expense.
method Bonsai-Net uses a modified differential pruner to explore a relaxed search space.
result Bonsai-Net consistently discovers better architectures than random search with fewer parameters.
Paper uses CMAB to improve NAS efficiency and accuracy.
problem Improving efficiency and accuracy of NAS for DNNs.
method Formulated NAS as CMAB, used Nested Monte-Carlo Search.
result Discovered cell structure achieves comparable accuracy to state-of-the-art, 20x faster.
BRP-NAS uses GCNs to predict neural network performance for more efficient NAS.
problem Inaccurate performance metrics in NAS lead to suboptimal model designs.
method Proposes BRP-NAS, a hardware-aware NAS using GCNs for accurate performance prediction.
result BRP-NAS outperforms previous methods on NAS-Bench-101 and 201, improving model sample efficiency.
This paper evaluates heuristics and hyperparameters in weight-sharing NAS methods.
problem Improving the performance of weight-sharing NAS methods.
method Systematic evaluation of heuristics and hyperparameters in weight-sharing NAS algorithms.
result Some heuristics negatively impact super-net and stand-alone performance correlation.
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the runtime constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challen…
The use of automatic methods, often referred to as Neural Architecture Search (NAS), in designing neural network architectures has recently drawn considerable attention. In this work, we present an efficient NAS approach, named HM- NAS, that generalizes existing weight sharing based NAS approaches. Existing weight shar…
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the latency constraint of a mobile device? Neural Architecture Search (NAS) for ConvNet design is a challenging problem due to the combinatorially large design space and search time (at least 200 GPU-hours…
Guided Evolution improves NAS efficiency and accuracy.
problem NAS methods converge to local minima and are complex.
method G-EA: guided evolutionary approach with initialization evaluation and continuous knowledge extraction.
result G-EA achieves state-of-the-art results in CIFAR-10, CIFAR-100, and ImageNet16-120.
BONAS accelerates NAS while maintaining reliability.
problem Computational inefficiency in sample-based NAS.
method Bayesian Optimized Neural Architecture Search (BONAS) using weight-sharing.
result BONAS accelerates sample-based NAS significantly while maintaining reliability.
NAS evaluation is hard due to lack of standard protocols.
problem Difficulty in comparing NAS methods due to varying search spaces and evaluation protocols.
method Benchmarked 8 NAS methods on 5 datasets, proposing a relative improvement metric over random architectures.
result Many NAS techniques struggle to significantly outperform a randomly generated architecture.
We propose a neural architecture search (NAS) algorithm, Petridish, to iteratively add shortcut connections to existing network layers. The added shortcut connections effectively perform gradient boosting on the augmented layers. The proposed algorithm is motivated by the feature selection algorithm forward stage-wise …
Improved hyperparameter optimization using simplified Transformer blocks.
problem Discovering optimal architectures in high-dimensional search spaces with limited exploration budgets.
method Simplified Transformer block for modeling hyper-parameter dependencies, actor-critic style algorithm, ensembling.
result Outperformed most algorithms on NAS-Bench-101 and Random Search in discovering more accurate model architectures.
NeuralArTS categorizes neural ops in a type system for NAS.
problem Manual optimization of search spaces for NAS is inefficient.
method Developed NeuralArTS, a type system for categorizing network ops.
result NeuralArTS can be applied to convolutional layers.
Can we reduce the search cost of Neural Architecture Search (NAS) from days down to only few hours? NAS methods automate the design of Convolutional Networks (ConvNets) under hardware constraints and they have emerged as key components of AutoML frameworks. However, the NAS problem remains challenging due to the combin…
Recent advances in Neural Architecture Search (NAS) have produced state-of-the-art architectures on several tasks. NAS shifts the efforts of human experts from developing novel architectures directly to designing architecture search spaces and methods to explore them efficiently. The search space definition captures pr…
GTNs generate training data to accelerate AI learning.
problem Speeding up AI learning through better training data.
method Generative Teaching Networks (GTNs) learn to generate synthetic training data.
result GTNs accelerate AI learning and NAS evaluations.
New method improves neural architecture search by optimizing for both performance and diversity.
problem Traditional multi-objective NAS fails to address practical constraints and niches.
method Formulated as quality diversity optimization, introduces multifidelity optimizers.
result Quality diversity NAS outperforms multi-objective NAS in quality and efficiency.
Differentiable NAS frameworks grow networks wider and deeper, revealing biases in wiring evolution.
problem Understanding the evolution of neural architecture wiring in differentiable NAS methods.
method Unified view on searching algorithms, local cost minimization, empirical and theoretical analyses.
result Implicit inductive biases cause observed searching patterns in differentiable NAS methods.
BANANAS uses Bayesian optimization with neural predictors to improve neural architecture search.
problem Improving neural architecture search (NAS) performance.
method BANANAS combines Bayesian optimization with neural predictors, focusing on architecture encoding, predictor, uncertainty calibration, acquisition function, and optimization strategy.
result BANANAS achieves state-of-the-art performance on NAS search spaces.
Simplified NAS for GNN architectures improves efficiency and expressiveness.
problem Efficient and effective discovery of optimal GNN architectures.
method SNAG framework with a novel search space and reinforcement learning.
result SNAG framework outperforms human-designed and existing NAS methods.
New method speeds up image denoising models without sacrificing performance.
problem Efficiently training models for image denoising tasks.
method Introduces superkernel techniques for fast training of dense prediction models.
result Demonstrates effectiveness on SIDD+ benchmark with 6-8 RTX2080 GPU hours.
HardCoRe-NAS finds fitting neural networks adhering to hard resource constraints.
problem Finding fitting neural networks that adhere to hard resource constraints.
method Accurate formulation of resource requirement and scalable search method.
result HardCoRe-NAS generates state-of-the-art architectures strictly satisfying hard resource constraints.
Neural Architecture Search remains a very challenging meta-learning problem. Several recent techniques based on parameter-sharing idea have focused on reducing the NAS running time by leveraging proxy models, leading to architectures with competitive performance compared to those with hand-crafted designs. In this pape…