Paper proposes GP-NAS-ensemble for fast neural architecture performance prediction.
problem Estimating neural network performance without training time-consuming evaluations.
method GP-NAS-ensemble framework using ensemble learning improvements.
result Ranked second in a NAS performance prediction challenge.
A framework benchmarks and dissects one-shot neural architecture search methods.
problem Understanding the dynamics and components of one-shot NAS methods.
method General framework for one-shot NAS, benchmarking with NAS-Bench-101.
result Comparison of one-shot NAS methods and their sensitivity to hyperparameters.
Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks. Existing techniques rely on two stages: searching over the architecture space and validating the best architecture. NAS algorithms are currently compared solely based on their results on the downstream task. While intuitive, …
Can we reduce the search cost of Neural Architecture Search (NAS) from days down to only few hours? NAS methods automate the design of Convolutional Networks (ConvNets) under hardware constraints and they have emerged as key components of AutoML frameworks. However, the NAS problem remains challenging due to the combin…
Differentiable NAS frameworks grow networks wider and deeper, revealing biases in wiring evolution.
problem Understanding the evolution of neural architecture wiring in differentiable NAS methods.
method Unified view on searching algorithms, local cost minimization, empirical and theoretical analyses.
result Implicit inductive biases cause observed searching patterns in differentiable NAS methods.
New benchmarks provide full training data for NAS research.
problem Limited training data on popular benchmarks restricts multi-fidelity techniques.
method SVD and noise modeling to create surrogate benchmarks with full training info.
result Learning curve extrapolation framework improves single-fidelity algorithms.
BS-NAS broadens and shrinks search space for optimal neural architectures.
problem Suboptimal channel numbers and model averaging effects in One-Shot NAS methods.
method Broadening with spring block for channel search, shrinking with underperforming operations removal, evolutionary algorithm for optimal architecture search.
result BS-NAS achieves state-of-the-art performance on ImageNet.
In this paper, a neural architecture search (NAS) framework is proposed for 3D medical image segmentation, to automatically optimize a neural architecture from a large design space. Our NAS framework searches the structure of each layer including neural connectivities and operation types in both of the encoder and deco…
Over the past half-decade, many methods have been considered for neural architecture search (NAS). Bayesian optimization (BO), which has long had success in hyperparameter optimization, has recently emerged as a very promising strategy for NAS when it is coupled with a neural predictor. Recent work has proposed differe…
UNAS combines DNAS and RL for efficient architecture search.
problem Discovering high accuracy or low latency neural architectures.
method Unified framework combining differentiable and reinforcement learning approaches.
result UNAS achieves state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet datasets.
DSNAS optimizes neural architecture and parameters in one step.
problem Poor correlation between architecture performance in two stages of NAS methods.
method Task-specific end-to-end approach with DSNAS framework.
result DSNAS discovers comparable accuracy networks in less time.
Simplified NAS for GNN architectures improves efficiency and expressiveness.
problem Efficient and effective discovery of optimal GNN architectures.
method SNAG framework with a novel search space and reinforcement learning.
result SNAG framework outperforms human-designed and existing NAS methods.
The neural architecture search (NAS) algorithm with reinforcement learning can be a powerful and novel framework for the automatic discovering process of neural architectures. However, its application is restricted by noncontinuous and high-dimensional search spaces, which result in difficulty in optimization. To resol…
New algorithms improve neural architecture search with faster convergence.
problem Improving efficiency and accuracy of neural architecture search.
method Geometry-aware gradient algorithms to optimize continuous relaxation of discrete search spaces.
result Exceeds state-of-the-art results on CIFAR and ImageNet benchmarks.
NAS-Bench-Suite simplifies NAS evaluation across diverse tasks.
problem Limited and inconsistent NAS benchmarks hinder research reproducibility.
method Developed a comprehensive, extensible NAS benchmark suite.
result Many NAS conclusions do not generalize across different benchmarks.
NeuralArTS categorizes neural ops in a type system for NAS.
problem Manual optimization of search spaces for NAS is inefficient.
method Developed NeuralArTS, a type system for categorizing network ops.
result NeuralArTS can be applied to convolutional layers.
Weight-sharing is rarely helpful in NAS, especially on small datasets.
problem The efficiency of weight-sharing in Neural Architecture Search (NAS) is questionable.
method Comparison of a state-of-the-art weight-sharing approach to random search on the nasbench dataset.
result Weight-sharing is only rarely significantly helpful in NAS, highlighting the importance of the search space.
Improved software flaw detection using NAS on multimodal DL models.
problem Software flaw detection in multimodal deep learning models.
method Adapted NAS framework for multimodal learning, combined with multimodal deep learning models.
result Improved performance on the Juliet Test Suite.
Optimal transport kernels improve neural architecture search efficiency.
problem Comparing complex neural architectures similarity using Euclidean metric fails.
method Developed a novel discrepancy using tree-Wasserstein (TW) for neural architectures.
result TW-based approaches outperform other methods in sequential and parallel NAS.
A new framework generates large hierarchical search spaces for neural architectures.
problem Discovering neural architectures from simple blocks is hard.
method Context-free grammars for a unified, scalable search space.
result Efficiently searches over complete architectures, outperforming existing methods.
WeakNAS uses a set of weaker predictors to find top architectures with fewer samples.
problem Finding the best neural architecture with heavy computation costs.
method Proposes a paradigm shift from fitting the whole architecture space to progressively fitting a search path through a set of weaker predictors.
result WeakNAS produces coarse-to-fine iteration to gradually refine the ranking of sampling space, requiring fewer samples to find top-performance architectures.
BRP-NAS uses GCNs to predict neural network performance for more efficient NAS.
problem Inaccurate performance metrics in NAS lead to suboptimal model designs.
method Proposes BRP-NAS, a hardware-aware NAS using GCNs for accurate performance prediction.
result BRP-NAS outperforms previous methods on NAS-Bench-101 and 201, improving model sample efficiency.
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the runtime constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challen…
The use of automatic methods, often referred to as Neural Architecture Search (NAS), in designing neural network architectures has recently drawn considerable attention. In this work, we present an efficient NAS approach, named HM- NAS, that generalizes existing weight sharing based NAS approaches. Existing weight shar…
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the latency constraint of a mobile device? Neural Architecture Search (NAS) for ConvNet design is a challenging problem due to the combinatorially large design space and search time (at least 200 GPU-hours…
Novel RL-based NPG improves multi-objective NAS efficiency and performance.
problem Discovering optimal neural architectures with multiple conflicting objectives.
method Non-stationary policy gradient with adaptive reward functions and shared model.
result Framework efficiently approximates full Pareto front and achieves superior performance.
New method improves neural architecture search by optimizing for both performance and diversity.
problem Traditional multi-objective NAS fails to address practical constraints and niches.
method Formulated as quality diversity optimization, introduces multifidelity optimizers.
result Quality diversity NAS outperforms multi-objective NAS in quality and efficiency.
Paper tackles NAS problem by modeling it as a sparse supernet.
problem Neural Architecture Search (NAS) problem, particularly Mixed-Path Search.
method Model NAS as a sparse supernet with sparsity constraints. Use hierarchical accelerated proximal gradient algorithm for optimization.
result Proposed method finds compact, general, and powerful neural architectures.
HardCoRe-NAS finds fitting neural networks adhering to hard resource constraints.
problem Finding fitting neural networks that adhere to hard resource constraints.
method Accurate formulation of resource requirement and scalable search method.
result HardCoRe-NAS generates state-of-the-art architectures strictly satisfying hard resource constraints.
Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS). Although the community has made major strides in developing better NAS methods, the quality of scientific empiric…
Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) …
Neural Architecture Search (NAS) has shown great potentials in finding better neural network designs. Sample-based NAS is the most reliable approach which aims at exploring the search space and evaluating the most promising architectures. However, it is computationally very costly. As a remedy, the one-shot approach ha…
Enhanced tabular benchmarks for energy-efficient neural architecture search.
problem Energy consumption in deep learning models.
method Introducing EC-NAS, an enhanced tabular benchmark with energy consumption data.
result EC-NAS reveals a balance between energy usage and accuracy in neural architecture search.
Proposes baselines for joint NAS and HPO optimization.
problem Joint optimization of neural architecture and hyperparameters for multiple objectives.
method Extends existing methods to jointly optimize with multiple objectives.
result Serves as simple baselines for future multi-objective joint NAS + HPO research.
High sensitivity of neural architecture search (NAS) methods against their input such as step-size (i.e., learning rate) and search space prevents practitioners from applying them out-of-the-box to their own problems, albeit its purpose is to automate a part of tuning process. Aiming at a fast, robust, and widely-appli…
Recent breakthroughs in Neural Architectural Search (NAS) have achieved state-of-the-art performance in many tasks such as image classification and language understanding. However, most existing works only optimize for model accuracy and largely ignore other important factors imposed by the underlying hardware and devi…
This study analyzes NAS benchmarks and finds that only a subset of operations is crucial for generating high-performing architectures.
problem NAS benchmarks lack generability and provide skewed performance distributions, leading to unreliable comparisons.
method Empirical analysis of widely used NAS benchmarks (101, 201, TransNAS-Bench-101) focusing on operation importance and generability.
result Only a subset of operations is necessary to generate architectures close to the upper-bound performance range, and convolution layers have the highest impact.
A method to reduce memory usage in NAS by pruning the search space.
problem High GPU memory consumption in One-Shot NAS techniques.
method Utilising Zero-Shot NAS to prune the search space before applying One-Shot NAS.
result Reduces memory consumption by 81% while maintaining accuracy.
Study on encoding neural architectures for NAS, showing impact on performance.
problem Impact of different encodings on NAS performance.
method Formal definition and characterization of encodings, empirical study of encodings' effectiveness.
result Different encodings can significantly affect NAS performance.
Study explores calibration properties in neural architectures.
problem Calibration issues in deep neural networks despite improved accuracy.
method Leverages Neural Architecture Search (NAS) to evaluate 117,702 neural networks.
result Identifies key architectural designs beneficial for calibration.
PDNAS optimizes GNN architectures for diverse datasets.
problem Inadequate adaptability and combinatorial search space in GNNs.
method Dual architecture search (micro- and macro-architectures) with gradient-based optimization.
result PDNAS finds deeper GNNs with better performance on diverse datasets.
This work benchmarks and theorizes robust NAS under adversarial training.
problem Lack of benchmark evaluations and theoretical guarantees for robust NAS architectures under adversarial training.
method Released a comprehensive data set and established a generalization theory using the neural tangent kernel.
result Established a generalization theory for robust NAS architectures under adversarial training.
We propose Stochastic Neural Architecture Search (SNAS), an economical end-to-end solution to Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters in same round of back-propagation, while maintaining the completeness and differentiability of the NAS pipeline.…
Study evaluates 31 performance predictors in NAS, recommending best for different settings.
problem Understanding and comparing different performance prediction techniques in NAS.
method Analysis of 31 techniques, testing correlation and rank-based measures, speed-up potential.
result Certain predictor families can be combined for better predictive power.
Neural architecture search (NAS) has been proposed to automatically tune deep neural networks, but existing search algorithms, e.g., NASNet, PNAS, usually suffer from expensive computational cost. Network morphism, which keeps the functionality of a neural network while changing its neural architecture, could be helpfu…
Paper shows equivalence between NA and ACLMM in diffusion models.
problem No arbitrage condition and existence of ACLMM in general diffusion models.
method Investigates equivalence between NA and ACLMM in single asset diffusion market models.
result NA is equivalent to ACLMM plus mild conditions on scale function and absence of reflecting boundaries.
NAS-X improves inference and model learning for SLVMs.
problem Challenges in analytic inference and model learning for flexible SLVMs.
method NAS-X combines reweighted wake-sleep and smoothing sequential Monte Carlo.
result NAS-X provides low-bias and low-variance gradient estimates.
Deep learning (DL) techniques have penetrated all aspects of our lives and brought us great convenience. However, building a high-quality DL system for a specific task highly relies on human expertise, hindering the applications of DL to more areas. Automated machine learning (AutoML) becomes a promising solution to bu…