FiGS searches over a larger space of architectures for efficient mobile models.
problem Designing small, efficient deep networks for mobile devices.
method Differentiable search method using sparse regularization and Logistic-Sigmoid distribution.
result FiGS produces state-of-the-art parameter-efficient models on ImageNet and improves object detection performance.
A real-time federated neural architecture search approach reduces costs and improves performance.
problem High communication and computational demands in federated learning for large models.
method Evolutionary approach with double-sampling technique to optimize model performance and reduce costs.
result Effective real-time federated neural architecture search for deep models on edge devices.
AutoShrink optimizes neural architectures by shrinking cell structures.
problem Resource constraints in deploying DNNs on mobile devices.
method Topology-aware node-based Neural Architecture Search (NAS).
result AutoShrink achieves up to 48% parameter reduction and 34% MACs savings.
FrostNet improves INT8 quantization efficiency in mobile networks.
problem The importance of network architecture for optimal INT8 quantization.
method Quantization-aware training (QAT) with StatAssist and GradBoost, hardware-aware NAS.
result FrostNets achieve higher recognition accuracy with comparable latency when quantized.
The evolution of MobileNets has laid a solid foundation for neural network applications on mobile end. With the latest MobileNetV3, neural architecture search again claimed its supremacy in network design. Unfortunately, till today all mobile methods mainly focus on CPU latencies instead of GPU, the latter, however, is…
DrNAS improves neural architecture search with Dirichlet distribution and progressive learning.
problem Efficiently search for neural architectures with improved generalization and exploration.
method Formulates architecture search as a distribution learning problem using Dirichlet distribution and gradient-based optimization. Introduces a progressive learning scheme to handle large-scale tasks.
result Achieves state-of-the-art results on CIFAR-10 and ImageNet, demonstrating improved generalization and exploration.
Recent studies on automatic neural architectures search have demonstrated significant performance, competitive to or even better than hand-crafted neural architectures. However, most of the existing network architecture tend to use residual, parallel structures and concatenation block between shallow and deep features …
This work automates CNN model compression for mobile devices.
problem Deploying trained CNNs to mobile devices requires balancing speed, memory, and accuracy.
method Reinforcement learning with architecture search and knowledge distillation.
result An automated model compression algorithm improves the trade-off between speed, memory, and accuracy.
This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice. Its optimization criterion is well fitted for an architecture-selection, i.e., it minimizes the regret incurred by a sub-optimal selection of operations. Unlike previous s…
Winograd convolutions are used to improve quantized neural networks.
problem Improving quantized neural networks using Winograd convolutions.
method Proposed a Winograd-aware formulation of convolution layers to expose numerical inaccuracies to learning.
result Up to 10% higher classification accuracy on CIFAR-10 with Winograd-aware layers.
Recent breakthroughs in Neural Architectural Search (NAS) have achieved state-of-the-art performance in many tasks such as image classification and language understanding. However, most existing works only optimize for model accuracy and largely ignore other important factors imposed by the underlying hardware and devi…
We implement a differentiable Neural Architecture Search (NAS) method inspired by FBNet for discovering neural networks that are heavily optimized for a particular target device. The FBNet NAS method discovers a neural network from a given search space by optimizing over a loss function which accounts for accuracy and …
The ability to rank candidate architectures is the key to the performance of neural architecture search~(NAS). One-shot NAS is proposed to reduce the expense but shows inferior performance against conventional NAS and is not adequately stable. We investigate into this and find that the ranking correlation between archi…
Can we reduce the search cost of Neural Architecture Search (NAS) from days down to only few hours? NAS methods automate the design of Convolutional Networks (ConvNets) under hardware constraints and they have emerged as key components of AutoML frameworks. However, the NAS problem remains challenging due to the combin…
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the runtime constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challen…
Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models. Our goal is to apply NAS to search for a better alternative to the Transformer. We first construct a large search space in…
Galen algorithm compresses neural networks for specific hardware with reduced latency.
problem Finding optimal compression policies for neural networks on specific hardware.
method Reinforcement learning using pruning and quantization to optimize inference latency.
result Compressed ResNet18 for ARM processor reduced inference latency by 80%.
Optimized neural networks for Edge TPU achieve high accuracy in real-time image classification.
problem Designing neural networks for hardware accelerators to achieve optimal performance.
method Hardware-aware neural architecture search and model customization for Edge TPU.
result Improved accuracy-latency tradeoff on Pixel 4's Edge TPU compared to existing models.
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the latency constraint of a mobile device? Neural Architecture Search (NAS) for ConvNet design is a challenging problem due to the combinatorially large design space and search time (at least 200 GPU-hours…
We introduce a new function-preserving transformation for efficient neural architecture search. This network transformation allows reusing previously trained networks and existing successful architectures that improves sample efficiency. We aim to address the limitation of current network transformation operations that…
DNAS disentangles neural architecture search for better interpretability and performance.
problem Lack of interpretability in existing neural architecture search methods.
method DNAS disentangles the hidden representation of the controller into semantically meaningful concepts.
result DNAS achieves state-of-the-art performance and competitive architectures.
Differentially-private FNAS protects privacy while collaboratively searching for neural architectures.
problem Collaborative neural architecture search with privacy concerns.
method Federated Neural Architecture Search (FNAS) with differential privacy (DP-FNAS).
result DP-FNAS can search for highly-performant neural architectures while protecting individual parties' privacy.
SGAS improves neural architecture search by choosing and pruning operations greedily.
problem NAS often fails to generalize in final evaluation.
method Divides search into sub-problems and chooses/prunes candidate operations greedily.
result SGAS finds state-of-the-art architectures with minimal computational cost.
Simplified NAS for GNN architectures improves efficiency and expressiveness.
problem Efficient and effective discovery of optimal GNN architectures.
method SNAG framework with a novel search space and reinforcement learning.
result SNAG framework outperforms human-designed and existing NAS methods.
NeuralArTS categorizes neural ops in a type system for NAS.
problem Manual optimization of search spaces for NAS is inefficient.
method Developed NeuralArTS, a type system for categorizing network ops.
result NeuralArTS can be applied to convolutional layers.
Contrastive embeddings improve neural architecture search performance.
problem Improving performance of neural architecture search algorithms.
method Contrastive learning to identify networks based on data Jacobians and produce embeddings.
result Traditional black-box optimization algorithms can reach state-of-the-art performance with contrastive embeddings.
A new framework generates large hierarchical search spaces for neural architectures.
problem Discovering neural architectures from simple blocks is hard.
method Context-free grammars for a unified, scalable search space.
result Efficiently searches over complete architectures, outperforming existing methods.
GATES improves neural architecture search by modeling operations as information transformation.
problem Improving predictor-based neural architecture search efficiency.
method GATES models operations as information transformation, covering both node and edge cell search spaces.
result GATES boosts sample efficiency and improves predictor performance.
NAS helps find best neural network designs.
problem Designing optimal neural network architectures.
method Optimization algorithms and search spaces.
result Introduction to major advances in NAS for CNNs.
Paper tackles NAS problem by modeling it as a sparse supernet.
problem Neural Architecture Search (NAS) problem, particularly Mixed-Path Search.
method Model NAS as a sparse supernet with sparsity constraints. Use hierarchical accelerated proximal gradient algorithm for optimization.
result Proposed method finds compact, general, and powerful neural architectures.
Study reduces NAS search cost by generating multiple complex architectures in one shot.
problem Finding multiple neural architectures with varying complexities efficiently.
method Uses importance sampling to generate and update multiple distributions of architectures.
result Reduces search cost by finding multiple architectures with different complexities in a single search.
GeneCAI optimizes DNN compression hyper-parameters for mobile devices.
problem Efficient deployment of complex DNNs on resource-limited devices.
method GeneCAI uses genetic algorithm to learn optimal hyper-parameters.
result GeneCAI finds models with better accuracy-complexity trade-off.
NPENAS improves neural architecture search efficiency and accuracy.
problem Efficient and accurate neural architecture search (NAS) for minimizing search costs.
method Proposes NPENAS, a neural predictor guided evolutionary algorithm that enhances exploration ability of evolutionary algorithms.
result NPENAS-BO and NPENAS-NP outperform existing NAS algorithms on NASBench-201, NASBench-101, and DARTS.
Improves neural architecture search methods to be more stable and efficient.
problem Neural architecture search methods are unstable and sensitive to hyperparameters.
method Discusses practical considerations to improve stability and efficiency.
result Improves overall performance of neural architecture search methods.
SSNAS finds neural architectures without labeled data.
problem Limited labeled data for NAS.
method Self-supervised learning for NAS.
result Comparable results to supervised NAS with labeled data.
DC-NAS improves neural architecture search by clustering and evaluating sub-networks.
problem Inaccurate evaluation of neural architectures in large search spaces.
method Divide-and-Conquer approach: feature representation, clustering, and evaluation of clusters.
result Achieved 75.1% top-1 accuracy on ImageNet, surpassing state-of-the-art methods.
Optimizes neural architecture search to generate novel lightweight models.
problem Over-reliance on expert knowledge limits NAS to local optima, preventing architectural breakthroughs.
method Casts NAS as an optimization problem, introduces a hierarchical graph-based search space, and uses Bayesian optimization.
result Generates extremely lightweight yet competitive models on six benchmark datasets.
einspace expands NAS search space to include diverse neural architectures.
problem NAS results are often limited to existing structures; new designs are rare.
method einspace uses a probabilistic context-free grammar to create a versatile search space.
result einspace discovers novel and improved architectures, including convolutions and attention.
In neural architecture search, the structure of the neural network to best model a given dataset is determined by an automated search process. Efficient Neural Architecture Search (ENAS), proposed by Pham et al. (2018), has recently received considerable attention due to its ability to find excellent architectures with…
Neural Architecture Search has shown potential to automate the design of neural networks. Deep Reinforcement Learning based agents can learn complex architectural patterns, as well as explore a vast and compositional search space. On the other hand, evolutionary algorithms offer higher sample efficiency, which is criti…
Extends NAS to learn both intra-cell and inter-cell architectures for language modeling.
problem Limited NAS systems restrict search to recurrent or convolutional cells.
method Designs a joint learning method to perform intra-cell and inter-cell NAS simultaneously.
result Significantly outperforms a strong baseline on PTB and WikiText data.
This paper compares Grid Search, Random Search, and Genetic Algorithm for NAS.
problem Hyperparameter optimization for neural architecture search.
method Comparison of Grid Search, Random Search, and Genetic Algorithm.
result Genetic Algorithm outperforms Grid Search and Random Search in terms of accuracy and execution time.
Memory-augmented neural networks (MANNs) are designed for question-answering tasks. It is difficult to run a MANN effectively on accelerators designed for other neural networks (NNs), in particular on mobile devices, because MANNs require recurrent data paths and various types of operations related to external memory a…
Paper uses CMAB to improve NAS efficiency and accuracy.
problem Improving efficiency and accuracy of NAS for DNNs.
method Formulated NAS as CMAB, used Nested Monte-Carlo Search.
result Discovered cell structure achieves comparable accuracy to state-of-the-art, 20x faster.
Survey of 1000 NAS papers, automating neural architecture design.
problem Designing high-performing neural architectures for various tasks.
method Taxonomy of search spaces, algorithms, and speedup techniques.
result NAS has surpassed human-designed architectures on many tasks.
Predict accuracy of neural architectures using non-neural models.
problem Improving efficiency and effectiveness of neural architecture search.
method Used gradient boosting decision tree (GBDT) for accuracy prediction and pruned the search space based on features learned from GBDT.
result GBDT-based predictor achieves comparable or better accuracy than neural network-based predictors and is 22x more sample efficient on NASBench-101.
Recent studies on neural architecture search have shown that automatically designed neural networks perform as good as expert-crafted architectures. While most existing works aim at finding architectures that optimize the prediction accuracy, these architectures may have complexity and is therefore not suitable being d…
This work proposes searching for optimal operation distribution in neural architecture search.
problem Finding optimal neural architecture with specific operations and connections.
method Search for the optimal operation distribution, providing a stochastic and approximate solution.
result Operation distribution holds enough discriminating power to reliably identify a solution and is easier to optimise than traditional encodings.