PDNAS optimizes GNN architectures for diverse datasets.
problem Inadequate adaptability and combinatorial search space in GNNs.
method Dual architecture search (micro- and macro-architectures) with gradient-based optimization.
result PDNAS finds deeper GNNs with better performance on diverse datasets.
New neural network architecture with height adds expressive power.
problem Expressiveness of neural networks limited by width and depth.
method Introduces height as a new hyper-parameter in neural network architecture.
result Neural networks with height achieve significantly better approximation of functions.
A graph VAE framework optimizes neural architectures in a continuous space.
problem Discovering efficient neural architectures in a discrete space.
method Graph VAE framework with VAE and GNN components, joint learning of predictors and decoders.
result The framework discovers powerful neural architectures with both excellent performance and high computational efficiency.
Graph Neural Networks (GNNs) have been popularly used for analyzing non-Euclidean data such as social network data and biological data. Despite their success, the design of graph neural networks requires a lot of manual work and domain knowledge. In this paper, we propose a Graph Neural Architecture Search method (Grap…
A ranking network improves neural architecture search efficiency.
problem Efficiently search for neural network architectures without extensive training.
method Pairwise ranking loss for a performance predictor trained on task meta-features.
result The ranking network outperforms the performance predictor in architecture search.
NAS helps find best neural network designs.
problem Designing optimal neural network architectures.
method Optimization algorithms and search spaces.
result Introduction to major advances in NAS for CNNs.
Graph Metanetworks process diverse neural architectures efficiently.
problem Processing diverse neural architectures efficiently.
method Builds metanetworks using graph neural networks to process graphs representing input neural networks.
result Proves GMNs are expressive and equivariant to parameter permutation symmetries.
Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameter…
In this paper, we propose Generative Adversarial Network (GAN) architectures that use Capsule Networks for image-synthesis. Based on the principal of positional-equivariance of features, Capsule Network's ability to encode spatial relationships between the features of the image helps it become a more powerful critic in…
Optimizes SGD for anytime neural networks, improving accuracy.
problem Training networks that produce increasingly accurate outputs over time.
method Orthogonalized SGD optimizer for nested architectures.
result Significantly improves generalization accuracy of anytime networks.
This study uncovers how neural architectures and weights interact in classification tasks.
problem Understanding the role of neural architecture and weights in classification performance.
method Developed a novel method to find optimal task-specific architectures as binary networks with {0, 1}-valued weights, using approximate gradient descent.
result Well-trained architectures may not require fine-tuning of weights, highlighting the importance of structure over weights.
Generative Adversarial Networks (GANs) are a powerful class of generative models. Despite their successes, the most appropriate choice of a GAN network architecture is still not well understood. GAN models for image synthesis have adopted a deep convolutional network architecture, which eliminates or minimizes the use …
New model predicts neural network performance from early training epochs, incorporating architecture impact.
problem Predicting neural network performance from early training epochs, neglecting architecture impact.
method Architecture-aware graph ordinary differential equation model.
result Model outperforms state-of-the-art methods for MLP and CNN learning curves.
DNArch learns CNN architectures by backpropagation.
problem Discovering optimal CNN architectures.
method Differentiable Neural Architectures (DNArch) learns CNN architectures by backpropagation, controlling kernel sizes, channels, downsampling positions, and depth.
result DNArch finds performant CNN architectures across various tasks.
The paper analyzes different neural network architectures for 3D point cloud processing.
problem Understanding the effects of specific intermediate-layer network architectures on 3D point cloud processing.
method Designing five metrics to diagnose various types of DNNs and conducting comparative studies.
result The hypotheses on the effects of specific intermediate-layer network architectures on the representation capacity of DNNs are verified.
This work explores how neural network architecture affects robustness to noisy labels.
problem The impact of neural network architecture on robustness to noisy labels.
method Formal framework connecting robustness to architecture alignments, measured by predictive power in representations.
result Network robustness to noisy labels improves when its architecture is more aligned with the target function.
RobNets improve robustness of deep neural networks against adversarial attacks.
problem Vulnerability of deep neural networks to adversarial attacks.
method One-shot neural architecture search, finetuning sub-networks, and analyzing robustness indicators.
result RobNets achieve superior robustness performance across various datasets.
Bayesian optimization reduces RNN architecture search time.
problem Optimizing RNN architectures for time series prediction.
method Bayesian Optimization with a training-free performance metric.
result BO designs well-performing RNN architectures with reduced optimization time.
Simplified NAS for GNN architectures improves efficiency and expressiveness.
problem Efficient and effective discovery of optimal GNN architectures.
method SNAG framework with a novel search space and reinforcement learning.
result SNAG framework outperforms human-designed and existing NAS methods.
Deep learning boosts building energy load forecasting.
problem Short-term load forecasting in buildings.
method Stacked Boosters Network architecture with sparse interactions, parameter sharing, and equivariant representations.
result Outperforms state-of-the-art models in short-term load forecasting tasks.
DCNs mimic neuronal networks for improved neural classification.
problem Lack of topological similarity between DNNs and biological neural networks.
method Developed DCNs with topologies inspired by real-world neuronal networks.
result High classification accuracy achieved by DCNs.
For network architecture search (NAS), it is crucial but challenging to simultaneously guarantee both effectiveness and efficiency. Towards achieving this goal, we develop a differentiable NAS solution, where the search space includes arbitrary feed-forward network consisting of the predefined number of connections. Be…
Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Gra…
We propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified…
DC-NAS improves neural architecture search by clustering and evaluating sub-networks.
problem Inaccurate evaluation of neural architectures in large search spaces.
method Divide-and-Conquer approach: feature representation, clustering, and evaluation of clusters.
result Achieved 75.1% top-1 accuracy on ImageNet, surpassing state-of-the-art methods.
A new autoencoder architecture captures multiscale data.
problem Multiscale spatio-temporal data representation.
method Integrates multigrid methods, convolutional autoencoders, and transfer learning.
result Adaptive, hierarchical architecture captures different scaled features dynamically.
The use of automatic methods, often referred to as Neural Architecture Search (NAS), in designing neural network architectures has recently drawn considerable attention. In this work, we present an efficient NAS approach, named HM- NAS, that generalizes existing weight sharing based NAS approaches. Existing weight shar…
Method detects neural network equivalence via matrix ensembles and spectral analysis.
problem Detecting equivalence among different deep learning architectures.
method Generating Mixed Matrix Ensembles (MMEs) and matching to conjugate circular ensembles.
result Empirical evidence shows vanishing differences in spectral densities with long tail decay rates.
Automatic neural architecture design has shown its potential in discovering powerful neural network architectures. Existing methods, no matter based on reinforcement learning or evolutionary algorithms (EA), conduct architecture search in a discrete space, which is highly inefficient. In this paper, we propose a simple…
Deep learning (DL) advances state-of-the-art reinforcement learning (RL), by incorporating deep neural networks in learning representations from the input to RL. However, the conventional deep neural network architecture is limited in learning representations for multi-task RL (MT-RL), as multiple tasks can refer to di…
Study examines how network architecture handles increasing data complexity.
problem Understanding how network architecture affects performance with complex data.
method Empirical study comparing various network architectures on an image classification task with increasing class numbers.
result Modern architectures show better generalization performance with increasing data complexity.
One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the architecture parameters from an over-parameterized network. However, there are two issues associated with most one-shot NAS methods…
Deep neural network architectures have traditionally been designed and explored with human expertise in a long-lasting trial-and-error process. This process requires huge amount of time, expertise, and resources. To address this tedious problem, we propose a novel algorithm to optimally find hyperparameters of a deep n…
New deep learning architecture learns martingales efficiently.
problem Efficiently learning martingales in financial derivatives pricing.
method High-order weak approximation algorithms of Runge-Kutta type.
result Deep neural networks based on this architecture learn martingales effectively.
A method for faster neural architecture search using low-fidelity training.
problem Time-consuming evaluations in neural architecture search.
method Bayesian multi-fidelity method with knowledge distillation.
result Training for a few epochs with knowledge distillation leads to better architecture selection.
Recently, deep residual networks have been successfully applied in many computer vision and natural language processing tasks, pushing the state-of-the-art performance with deeper and wider architectures. In this work, we interpret deep residual networks as ordinary differential equations (ODEs), which have long been s…
Designing the architecture for an artificial neural network is a cumbersome task because of the numerous parameters to configure, including activation functions, layer types, and hyper-parameters. With the large number of parameters for most networks nowadays, it is intractable to find a good configuration for a given …
Deep learning models' architectures, including depth and width, are key factors influencing models' performance, such as test accuracy and computation time. This paper solves two problems: given computation time budget, choose an architecture to maximize accuracy, and given accuracy requirement, choose an architecture …
In this paper we present a statistical analysis about the characteristics that we intend to influence in the performance of the neural networks in terms of assertiveness in the prediction of Brazilian stock returns. We created a population of architectures for analysis and extracted the sample that had the best asserti…
Comparative study of neural networks for short-term FOREX forecasting.
problem Simulating expert judgment in foreign exchange market forecasting.
method Implemented and compared LSTM and ANN architectures for short-term FOREX forecasting.
result ANN custom architecture outperforms LSTM in prediction quality and resource efficiency.
Study compares LSTM and ANN architectures for forex prediction, finding ANN more efficient.
problem Forex time series prediction efficiency and resource usage.
method Comparison of Long Short-Term Memory (LSTM) and specialized ANN architectures.
result Specialized ANN architecture performs better with fewer resources and faster execution.
A lightweight network and NAS method improve ASC tasks.
problem Heavy computational burden in acoustic scene classification.
method Inspired by MobileNetV2, unidirectional convolutions; dynamic NAS with evolutionary algorithm.
result 90.3% F1-score on DCASE2018 task 5, 25% fewer FLOPs.
New language models improve essay scoring accuracy.
problem Improving essay scoring accuracy using AI.
method Used BERT and XLNet language models to compare with traditional methods.
result Achieved above human-level accuracy on AES dataset.
NAS-Navigator automates neural network architecture search with visual steering.
problem Difficulty in configuring large neural networks efficiently.
method Formulates architecture optimization as graph space exploration, trains all candidate architectures in one-shot.
result Allows analysts to effectively select and guide the search for optimal neural network architectures.
The paper proposes a neural network architecture inspired by Langevin Monte Carlo for sampling from target distributions.
problem Sampling from complex target distributions efficiently.
method A neural network architecture inspired by Langevin Monte Carlo is proposed to map samples from a simple reference distribution to samples from the target.
result The proposed neural network architecture achieves approximation rates in the Wasserstein-2 distance for smooth, log-concave target distributions.
RARTS improves neural architecture search efficiency and accuracy.
problem Efficiency and accuracy issues in first-order DARTS.
method Formulates a relaxed architecture search (RARTS) method that uses whole dataset splitting.
result RARTS outperforms DARTS and its variants in accuracy and efficiency.
FrostNet improves INT8 quantization efficiency in mobile networks.
problem The importance of network architecture for optimal INT8 quantization.
method Quantization-aware training (QAT) with StatAssist and GradBoost, hardware-aware NAS.
result FrostNets achieve higher recognition accuracy with comparable latency when quantized.
One-shot neural architecture search limits depth search space and prunes networks for better performance and uncertainty.
problem Finding optimal depth in residual networks for efficient training and inference.
method Formulated a variational objective to approximate the depth distribution and pruned networks based on this distribution.
result Pruned networks achieve competitive accuracy with unpruned networks and better uncertainty calibration.