Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks. Existing techniques rely on two stages: searching over the architecture space and validating the best architecture. NAS algorithms are currently compared solely based on their results on the downstream task. While intuitive, …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “NAS evaluation”
NAS-Bench-Suite simplifies NAS evaluation across diverse tasks.
problem Limited and inconsistent NAS benchmarks hinder research reproducibility.
method Developed a comprehensive, extensible NAS benchmark suite.
result Many NAS conclusions do not generalize across different benchmarks.
Paper proposes GP-NAS-ensemble for fast neural architecture performance prediction.
problem Estimating neural network performance without training time-consuming evaluations.
method GP-NAS-ensemble framework using ensemble learning improvements.
result Ranked second in a NAS performance prediction challenge.
Are Neural Architecture Search Benchmarks Well Designed? A Deeper Look Into Operation Importancecs.LG
This study analyzes NAS benchmarks and finds that only a subset of operations is crucial for generating high-performing architectures.
problem NAS benchmarks lack generability and provide skewed performance distributions, leading to unreliable comparisons.
method Empirical analysis of widely used NAS benchmarks (101, 201, TransNAS-Bench-101) focusing on operation importance and generability.
result Only a subset of operations is necessary to generate architectures close to the upper-bound performance range, and convolution layers have the highest impact.
Hill-climbing is a powerful baseline for NAS, even with reduced noise.
problem Noise in evaluating neural architectures.
method Hill-climbing algorithm, denoising training pipeline.
result Hill-climbing outperforms state-of-the-art NAS algorithms with reduced noise.
NAS evaluation is hard due to lack of standard protocols.
problem Difficulty in comparing NAS methods due to varying search spaces and evaluation protocols.
method Benchmarked 8 NAS methods on 5 datasets, proposing a relative improvement metric over random architectures.
result Many NAS techniques struggle to significantly outperform a randomly generated architecture.
Paper evaluates robustness of NAS against poisoning attacks.
problem Robustness of Neural Architecture Search (NAS) against poisoning attacks.
method Evaluation of Efficient NAS (ENAS) against carefully designed ineffective operations in poisoning attacks.
result Demonstrates how poisoning attacks exploit design flaws in ENAS controller.
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public arc…
Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS). Although the community has made major strides in developing better NAS methods, the quality of scientific empiric…
The use of automatic methods, often referred to as Neural Architecture Search (NAS), in designing neural network architectures has recently drawn considerable attention. In this work, we present an efficient NAS approach, named HM- NAS, that generalizes existing weight sharing based NAS approaches. Existing weight shar…
A framework benchmarks and dissects one-shot neural architecture search methods.
problem Understanding the dynamics and components of one-shot NAS methods.
method General framework for one-shot NAS, benchmarking with NAS-Bench-101.
result Comparison of one-shot NAS methods and their sensitivity to hyperparameters.
NAS helps find best neural network designs.
problem Designing optimal neural network architectures.
method Optimization algorithms and search spaces.
result Introduction to major advances in NAS for CNNs.
Study accelerates NAS research with a large dataset of ZC proxies.
problem Speeding up neural architecture search with ZC proxies.
method Created NAS-Bench-Suite, evaluated 13 ZC proxies across 28 tasks, and provided a unified codebase.
result ZC proxies capture substantial complementary information and can improve NAS algorithm performance.
This paper evaluates heuristics and hyperparameters in weight-sharing NAS methods.
problem Improving the performance of weight-sharing NAS methods.
method Systematic evaluation of heuristics and hyperparameters in weight-sharing NAS algorithms.
result Some heuristics negatively impact super-net and stand-alone performance correlation.
Neural Architecture Search (NAS) has shown great success in automating the design of neural networks, but the prohibitive amount of computations behind current NAS methods requires further investigations in improving the sample efficiency and the network evaluation cost to get better results in a shorter time. In this …
Guided Evolution improves NAS efficiency and accuracy.
problem NAS methods converge to local minima and are complex.
method G-EA: guided evolutionary approach with initialization evaluation and continuous knowledge extraction.
result G-EA achieves state-of-the-art results in CIFAR-10, CIFAR-100, and ImageNet16-120.
BONAS accelerates NAS while maintaining reliability.
problem Computational inefficiency in sample-based NAS.
method Bayesian Optimized Neural Architecture Search (BONAS) using weight-sharing.
result BONAS accelerates sample-based NAS significantly while maintaining reliability.
Enhanced tabular benchmarks for energy-efficient neural architecture search.
problem Energy consumption in deep learning models.
method Introducing EC-NAS, an enhanced tabular benchmark with energy consumption data.
result EC-NAS reveals a balance between energy usage and accuracy in neural architecture search.
New benchmarks provide full training data for NAS research.
problem Limited training data on popular benchmarks restricts multi-fidelity techniques.
method SVD and noise modeling to create surrogate benchmarks with full training info.
result Learning curve extrapolation framework improves single-fidelity algorithms.
Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) …
This work benchmarks and theorizes robust NAS under adversarial training.
problem Lack of benchmark evaluations and theoretical guarantees for robust NAS architectures under adversarial training.
method Released a comprehensive data set and established a generalization theory using the neural tangent kernel.
result Established a generalization theory for robust NAS architectures under adversarial training.
DSNAS optimizes neural architecture and parameters in one step.
problem Poor correlation between architecture performance in two stages of NAS methods.
method Task-specific end-to-end approach with DSNAS framework.
result DSNAS discovers comparable accuracy networks in less time.
New approach to handle ranking function variation in zero-shot NAS.
problem Variation in ranking function outputs due to randomness.
method Viewing ranking function output as a random variable and constructing a stochastic ordering.
result Stochastic ordering boosts performance in neural architecture search.
A benchmark for NLP models trained on text datasets.
problem Limited access to high-performance clusters for NAS experiments.
method Created a search space for recurrent neural networks on text datasets and trained 14k architectures.
result Demonstrated the potential of precomputed NAS results for NLP.
Weight-sharing is rarely helpful in NAS, especially on small datasets.
problem The efficiency of weight-sharing in Neural Architecture Search (NAS) is questionable.
method Comparison of a state-of-the-art weight-sharing approach to random search on the nasbench dataset.
result Weight-sharing is only rarely significantly helpful in NAS, highlighting the importance of the search space.
The emergence of neural architecture search (NAS) has greatly advanced the research on network design. Recent proposals such as gradient-based methods or one-shot approaches significantly boost the efficiency of NAS. In this paper, we formulate the NAS problem from a Bayesian perspective. We propose explicitly estimati…
MTL-NAS combines NAS with GP-MTL for task-agnostic multi-task learning.
problem Designing architectures for diverse tasks with varying priors.
method Disentangled GP-MTL networks, hierarchical feature sharing, and gradient-based search.
result General-purpose model trained once can adapt to multiple tasks.
GTNs generate training data to accelerate AI learning.
problem Speeding up AI learning through better training data.
method Generative Teaching Networks (GTNs) learn to generate synthetic training data.
result GTNs accelerate AI learning and NAS evaluations.
Study explores calibration properties in neural architectures.
problem Calibration issues in deep neural networks despite improved accuracy.
method Leverages Neural Architecture Search (NAS) to evaluate 117,702 neural networks.
result Identifies key architectural designs beneficial for calibration.
Smooth embedding space improves NAS performance.
problem Efficiently predicting good neural architectures.
method Two-sided variational graph autoencoder.
result Smooth embedding space facilitates extrapolation to unseen architectures.
WeakNAS uses a set of weaker predictors to find top architectures with fewer samples.
problem Finding the best neural architecture with heavy computation costs.
method Proposes a paradigm shift from fitting the whole architecture space to progressively fitting a search path through a set of weaker predictors.
result WeakNAS produces coarse-to-fine iteration to gradually refine the ranking of sampling space, requiring fewer samples to find top-performance architectures.
Study evaluates 31 performance predictors in NAS, recommending best for different settings.
problem Understanding and comparing different performance prediction techniques in NAS.
method Analysis of 31 techniques, testing correlation and rank-based measures, speed-up potential.
result Certain predictor families can be combined for better predictive power.
AgEBO-Tabular combines NAS and hyperparameter tuning for fast, high-performing tabular models.
problem Developing high-performing predictive models for large tabular data sets is challenging.
method Combines aging evolution NAS and asynchronous Bayesian optimization for hyperparameter tuning in data-parallel training.
result Automatically discovered neural network models outperform state-of-the-art AutoML ensembles in inference speed by two orders of magnitude.
DC-NAS improves neural architecture search by clustering and evaluating sub-networks.
problem Inaccurate evaluation of neural architectures in large search spaces.
method Divide-and-Conquer approach: feature representation, clustering, and evaluation of clusters.
result Achieved 75.1% top-1 accuracy on ImageNet, surpassing state-of-the-art methods.
Survey tackles challenges in neural architecture design.
problem Challenges in designing optimal neural architectures.
method New classification perspective based on early NAS algorithms' characteristics, problems, and solutions.
result Comprehensive analysis and comparison of NAS works.
Novel RL-based NPG improves multi-objective NAS efficiency and performance.
problem Discovering optimal neural architectures with multiple conflicting objectives.
method Non-stationary policy gradient with adaptive reward functions and shared model.
result Framework efficiently approximates full Pareto front and achieves superior performance.
Synthetic Petri Dish predicts neural architecture performance faster.
problem Expensive NAS evaluation process with ground-truth data.
method Instantiates motifs in small networks, evaluates with few synthetic samples.
result Significantly higher accuracy in predicting motif performance.
SemiNAS reduces NAS cost by predicting accuracy of unlabeled architectures.
problem Costly evaluation of architectures limits NAS efficiency.
method SemiNAS uses unlabeled architectures to train an accuracy predictor.
result SemiNAS achieves comparable accuracy with less data.
BRP-NAS uses GCNs to predict neural network performance for more efficient NAS.
problem Inaccurate performance metrics in NAS lead to suboptimal model designs.
method Proposes BRP-NAS, a hardware-aware NAS using GCNs for accurate performance prediction.
result BRP-NAS outperforms previous methods on NAS-Bench-101 and 201, improving model sample efficiency.
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the runtime constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challen…
Estimates neural architecture performance speedily.
problem Accurately evaluating neural architectures' generalization performance.
method Estimates final test performance based on training speed.
result Consistently outperforms other alternatives in correlation with true test performance.
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the latency constraint of a mobile device? Neural Architecture Search (NAS) for ConvNet design is a challenging problem due to the combinatorially large design space and search time (at least 200 GPU-hours…
Can we reduce the search cost of Neural Architecture Search (NAS) from days down to only few hours? NAS methods automate the design of Convolutional Networks (ConvNets) under hardware constraints and they have emerged as key components of AutoML frameworks. However, the NAS problem remains challenging due to the combin…
New method improves neural architecture search by optimizing for both performance and diversity.
problem Traditional multi-objective NAS fails to address practical constraints and niches.
method Formulated as quality diversity optimization, introduces multifidelity optimizers.
result Quality diversity NAS outperforms multi-objective NAS in quality and efficiency.
HardCoRe-NAS finds fitting neural networks adhering to hard resource constraints.
problem Finding fitting neural networks that adhere to hard resource constraints.
method Accurate formulation of resource requirement and scalable search method.
result HardCoRe-NAS generates state-of-the-art architectures strictly satisfying hard resource constraints.
BS-NAS broadens and shrinks search space for optimal neural architectures.
problem Suboptimal channel numbers and model averaging effects in One-Shot NAS methods.
method Broadening with spring block for channel search, shrinking with underperforming operations removal, evolutionary algorithm for optimal architecture search.
result BS-NAS achieves state-of-the-art performance on ImageNet.
Proposes baselines for joint NAS and HPO optimization.
problem Joint optimization of neural architecture and hyperparameters for multiple objectives.
method Extends existing methods to jointly optimize with multiple objectives.
result Serves as simple baselines for future multi-objective joint NAS + HPO research.
VINNAS uses variational inference to avoid mode collapse in neural architecture search.
problem Mode collapse in gradient-based NAS methods, leading to suboptimal architectures.
method Differentiable variational inference with variational dropout and automatic relevance determination.
result State-of-the-art accuracy with up to twice fewer non-zero parameters.