Graph Metanetworks process diverse neural architectures efficiently.
problem Processing diverse neural architectures efficiently.
method Builds metanetworks using graph neural networks to process graphs representing input neural networks.
result Proves GMNs are expressive and equivariant to parameter permutation symmetries.
A graph VAE framework optimizes neural architectures in a continuous space.
problem Discovering efficient neural architectures in a discrete space.
method Graph VAE framework with VAE and GNN components, joint learning of predictors and decoders.
result The framework discovers powerful neural architectures with both excellent performance and high computational efficiency.
New model predicts neural network performance from early training epochs, incorporating architecture impact.
problem Predicting neural network performance from early training epochs, neglecting architecture impact.
method Architecture-aware graph ordinary differential equation model.
result Model outperforms state-of-the-art methods for MLP and CNN learning curves.
NAS helps find best neural network designs.
problem Designing optimal neural network architectures.
method Optimization algorithms and search spaces.
result Introduction to major advances in NAS for CNNs.
Graph Neural Networks (GNNs) have been popularly used for analyzing non-Euclidean data such as social network data and biological data. Despite their success, the design of graph neural networks requires a lot of manual work and domain knowledge. In this paper, we propose a Graph Neural Architecture Search method (Grap…
DNArch learns CNN architectures by backpropagation.
problem Discovering optimal CNN architectures.
method Differentiable Neural Architectures (DNArch) learns CNN architectures by backpropagation, controlling kernel sizes, channels, downsampling positions, and depth.
result DNArch finds performant CNN architectures across various tasks.
Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameter…
Simplified NAS for GNN architectures improves efficiency and expressiveness.
problem Efficient and effective discovery of optimal GNN architectures.
method SNAG framework with a novel search space and reinforcement learning.
result SNAG framework outperforms human-designed and existing NAS methods.
Method detects neural network equivalence via matrix ensembles and spectral analysis.
problem Detecting equivalence among different deep learning architectures.
method Generating Mixed Matrix Ensembles (MMEs) and matching to conjugate circular ensembles.
result Empirical evidence shows vanishing differences in spectral densities with long tail decay rates.
We propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified…
New neural network architecture with height adds expressive power.
problem Expressiveness of neural networks limited by width and depth.
method Introduces height as a new hyper-parameter in neural network architecture.
result Neural networks with height achieve significantly better approximation of functions.
A method for faster neural architecture search using low-fidelity training.
problem Time-consuming evaluations in neural architecture search.
method Bayesian multi-fidelity method with knowledge distillation.
result Training for a few epochs with knowledge distillation leads to better architecture selection.
DC-NAS improves neural architecture search by clustering and evaluating sub-networks.
problem Inaccurate evaluation of neural architectures in large search spaces.
method Divide-and-Conquer approach: feature representation, clustering, and evaluation of clusters.
result Achieved 75.1% top-1 accuracy on ImageNet, surpassing state-of-the-art methods.
This study uncovers how neural architectures and weights interact in classification tasks.
problem Understanding the role of neural architecture and weights in classification performance.
method Developed a novel method to find optimal task-specific architectures as binary networks with {0, 1}-valued weights, using approximate gradient descent.
result Well-trained architectures may not require fine-tuning of weights, highlighting the importance of structure over weights.
Braid theory optimizes neural network structures.
problem Optimizing the architecture of neural networks.
method Using braid theory to describe and construct neural network structures.
result Braid-based networks outperform other architectures in classification tasks.
In this paper we present a statistical analysis about the characteristics that we intend to influence in the performance of the neural networks in terms of assertiveness in the prediction of Brazilian stock returns. We created a population of architectures for analysis and extracted the sample that had the best asserti…
Residual neural networks don't help overcome sampling complexity issues.
problem Learning invertible residual neural networks from samples is hard due to the curse of dimensionality.
method Investigated invertible residual neural networks and their sampling complexity.
result Invertible residual neural networks still suffer from the curse of dimensionality in sampling complexity.
NAS-Navigator automates neural network architecture search with visual steering.
problem Difficulty in configuring large neural networks efficiently.
method Formulates architecture optimization as graph space exploration, trains all candidate architectures in one-shot.
result Allows analysts to effectively select and guide the search for optimal neural network architectures.
The paper proposes a neural network architecture inspired by Langevin Monte Carlo for sampling from target distributions.
problem Sampling from complex target distributions efficiently.
method A neural network architecture inspired by Langevin Monte Carlo is proposed to map samples from a simple reference distribution to samples from the target.
result The proposed neural network architecture achieves approximation rates in the Wasserstein-2 distance for smooth, log-concave target distributions.
Empirical study shows standard CNNs deviate from NTK predictions.
problem Understanding how standard finite-width CNNs behave compared to their infinite-width NTK counterparts.
method Empirical analysis of AlexNet and LeNet architectures.
result Standard CNNs deviate significantly from their NTK counterparts, but deviation decreases with wider networks.
Contrastive embeddings improve neural architecture search performance.
problem Improving performance of neural architecture search algorithms.
method Contrastive learning to identify networks based on data Jacobians and produce embeddings.
result Traditional black-box optimization algorithms can reach state-of-the-art performance with contrastive embeddings.
New framework combines simple machines into complex ones for better neural network performance.
problem Improving neural network performance with limited training data.
method Developed a framework using topology and functional analysis to combine simple machines into complex ones, and used kernel methods to find optimal architectures.
result Kernel-inspired networks can outperform classical neural networks when training data is small.
HardCoRe-NAS finds fitting neural networks adhering to hard resource constraints.
problem Finding fitting neural networks that adhere to hard resource constraints.
method Accurate formulation of resource requirement and scalable search method.
result HardCoRe-NAS generates state-of-the-art architectures strictly satisfying hard resource constraints.
Review of neural network expressivity and architectures.
problem Understanding neural network expressivity across different architectures.
method Comprehensive overview of approximation results for various neural network types.
result Deep neural networks offer advantages over shallow ones for specific function classes.
Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Gra…
Recent studies on neural architecture search have shown that automatically designed neural networks perform as good as expert-crafted architectures. While most existing works aim at finding architectures that optimize the prediction accuracy, these architectures may have complexity and is therefore not suitable being d…
VINNAS uses variational inference to avoid mode collapse in neural architecture search.
problem Mode collapse in gradient-based NAS methods, leading to suboptimal architectures.
method Differentiable variational inference with variational dropout and automatic relevance determination.
result State-of-the-art accuracy with up to twice fewer non-zero parameters.
Polynomial neural networks explore thresholds for maximum expressiveness.
problem Understanding the limits of polynomial neural networks' expressiveness.
method Introducing activation degree threshold to measure network expressiveness and proving its existence and upper bounds.
result Polynomial neural networks with equi-width architectures achieve the maximum expressiveness.
This work explores how neural network architecture affects robustness to noisy labels.
problem The impact of neural network architecture on robustness to noisy labels.
method Formal framework connecting robustness to architecture alignments, measured by predictive power in representations.
result Network robustness to noisy labels improves when its architecture is more aligned with the target function.
The interplay between inter-neuronal network topology and cognition has been studied deeply by connectomics researchers and network scientists, which is crucial towards understanding the remarkable efficacy of biological neural networks. Curiously, the deep learning revolution that revived neural networks has not paid …
NTK theory fails to predict practical behavior of large-width neural networks.
problem Theoretical limits of NTK do not match practical neural network architectures.
method Empirical investigation of NTK's applicability to large-width architectures.
result Practically relevant behavior of large-width architectures differs from NTK theory.
Designing the architecture for an artificial neural network is a cumbersome task because of the numerous parameters to configure, including activation functions, layer types, and hyper-parameters. With the large number of parameters for most networks nowadays, it is intractable to find a good configuration for a given …
Automatic neural architecture design has shown its potential in discovering powerful neural network architectures. Existing methods, no matter based on reinforcement learning or evolutionary algorithms (EA), conduct architecture search in a discrete space, which is highly inefficient. In this paper, we propose a simple…
Deep neural networks have shown remarkable performance across a wide range of vision-based tasks, particularly due to the availability of large-scale datasets for training and better architectures. However, data seen in the real world are often affected by distortions that not accounted for by the training datasets. In…
In this paper we propose a Bayesian method for estimating architectural parameters of neural networks, namely layer size and network depth. We do this by learning concrete distributions over these parameters. Our results show that regular networks with a learnt structure can generalise better on small datasets, while f…
New neural networks combine additive regression with traditional architectures.
problem Performance limitations and high parameter requirements of traditional neural networks.
method Introduce hybrid deep additive neural networks with simpler activation and basis functions.
result Hybrid neural networks achieve better performance with fewer parameters.
Comparative study of neural networks for short-term FOREX forecasting.
problem Simulating expert judgment in foreign exchange market forecasting.
method Implemented and compared LSTM and ANN architectures for short-term FOREX forecasting.
result ANN custom architecture outperforms LSTM in prediction quality and resource efficiency.
Taking inspiration from biological evolution, we explore the idea of "Can deep neural networks evolve naturally over successive generations into highly efficient deep neural networks?" by introducing the notion of synthesizing new highly efficient, yet powerful deep neural networks over successive generations via an ev…
Study reveals how neural network architectures bias their learning based on feature directions.
problem Understanding how neural network architectures bias their learning based on feature directions.
method Defined neural anisotropy directions (NADs) to encapsulate the directional inductive bias of architectures and provided an efficient method to identify them.
result NADs characterize the features used by CNNs to discriminate between different classes for the CIFAR-10 dataset.
SSNAS finds neural architectures without labeled data.
problem Limited labeled data for NAS.
method Self-supervised learning for NAS.
result Comparable results to supervised NAS with labeled data.
Deep learning (DL) advances state-of-the-art reinforcement learning (RL), by incorporating deep neural networks in learning representations from the input to RL. However, the conventional deep neural network architecture is limited in learning representations for multi-task RL (MT-RL), as multiple tasks can refer to di…
Neural architecture search (NAS) has been proposed to automatically tune deep neural networks, but existing search algorithms, e.g., NASNet, PNAS, usually suffer from expensive computational cost. Network morphism, which keeps the functionality of a neural network while changing its neural architecture, could be helpfu…
This paper classifies G-invariant shallow neural networks.
problem Designing optimal G-invariant neural architectures for G-invariant target functions. method Proving theorems about the classification and morphisms of G-invariant single-hidden-layer neural networks. result Classification of G-invariant shallow neural networks and characterization of morphisms. The learnability of different neural architectures can be characterized directly by computable measures of data complexity. In this paper, we reframe the problem of architecture selection as understanding how data determines the most expressive and generalizable architectures suited to that data, beyond inductive bias.…
New BNN architectures reduce computational cost for uncertainty quantification.
problem High computational cost in Bayesian neural networks.
method Partial trace-class Bayesian neural networks (PaTraC BNNs).
result Comparable uncertainty quantification with fewer parameters.
New deep learning architecture learns martingales efficiently.
problem Efficiently learning martingales in financial derivatives pricing.
method High-order weak approximation algorithms of Runge-Kutta type.
result Deep neural networks based on this architecture learn martingales effectively.
A new optimisation framework for neural networks without hyperparameters.
problem Lack of explicit architectural information in optimisation frameworks.
method Extends mirror descent to account for neural architecture, transforming a Bregman divergence.
result Automatic gradient descent: a first-order optimiser without hyperparameters.
BINAS improves neural architecture search with interpretable models.
problem Finding efficient neural networks under resource constraints.
method BINAS uses a bilinear formulation for accuracy and resource estimation, with a scalable search method.
result BINAS generates comparable or better architectures than state-of-the-art methods, while strictly satisfying resource constraints.