BNAS improves neural architecture search with a scalable, fast, and efficient approach.
problem Efficiently searching for optimal neural architectures with high performance and low training time.
method Designing a broad scalable architecture (BCNN) with reinforcement learning and parameter sharing, and developing two variants.
result Significantly reduces training time and achieves state-of-the-art performance on CIFAR-10 and ImageNet.
BayesNAS uses Bayesian learning to improve neural architecture search efficiency.
problem Improper treatment of zero operations and architecture parameter pruning issues in one-shot NAS methods.
method Employing hierarchical automatic relevance determination (HARD) priors for Bayesian learning to model architecture parameters.
result Found architecture on CIFAR-10 in just 0.2 GPU days using a single GPU.
AGNN automates GNN architecture search, achieving best performance.
problem Finding optimal GNN architectures is laborious and requires human expertise.
method AGNN uses reinforcement learning to search for optimal GNN architectures within a predefined space, with a novel parameter sharing strategy.
result AGNN identifies optimal GNN architectures achieving best performance.
In practical Bayesian optimization, we must often search over structures with differing numbers of parameters. For instance, we may wish to search over neural network architectures with an unknown number of layers. To relate performance data gathered for different architectures, we define a new kernel for conditional p…
New neural network architecture with height adds expressive power.
problem Expressiveness of neural networks limited by width and depth.
method Introduces height as a new hyper-parameter in neural network architecture.
result Neural networks with height achieve significantly better approximation of functions.
Graph Metanetworks process diverse neural architectures efficiently.
problem Processing diverse neural architectures efficiently.
method Builds metanetworks using graph neural networks to process graphs representing input neural networks.
result Proves GMNs are expressive and equivariant to parameter permutation symmetries.
DSNAS optimizes neural architecture and parameters in one step.
problem Poor correlation between architecture performance in two stages of NAS methods.
method Task-specific end-to-end approach with DSNAS framework.
result DSNAS discovers comparable accuracy networks in less time.
In this paper we propose a Bayesian method for estimating architectural parameters of neural networks, namely layer size and network depth. We do this by learning concrete distributions over these parameters. Our results show that regular networks with a learnt structure can generalise better on small datasets, while f…
New method finds minimal neural networks without weights.
problem Importance of weight parameters in neural networks.
method Search for minimal neural network architectures without explicit weight training.
result Minimal neural networks can perform tasks without weight training.
Differentiable NAS method optimizes network architecture and parameters efficiently.
problem Challenging to simultaneously guarantee effectiveness and efficiency in network architecture search.
method Differentiable architecture search with ensemble Gumbel-Softmax estimator.
result End-to-end mechanism for searching network architectures, discovering high-performance architectures efficiently.
A new neural network architecture reduces parameters by 94% while maintaining performance.
problem Reduction of trainable parameters in neural networks.
method Spatially-coupled sparse construction to allocate trainable parameters efficiently.
result Performance comparable to traditional neural networks with 94% fewer parameters.
This study examines how neural network architecture parameters affect loss surface modality.
problem Understanding the relationship between neural architecture parameters and loss surface modality.
method Fitness landscape analysis of neural network loss surfaces under various architecture settings.
result An increase in problem dimensionality, hidden layer width, and architecture depth affects the modality of loss surfaces.
Paper proposes a method to estimate scientific parameters in hybrid models without relying on model architecture.
problem Estimating unknown parameters in hybrid models combining machine learning and scientific models.
method Sharpness-aware minimization adapted for hybrid modeling, focusing on model simplicity.
result Demonstrates effectiveness of SAM-based hybrid model learning for scientific parameter estimation.
Deep learning predicts neural network parameters efficiently.
problem Optimizing neural network parameters remains inefficient.
method Used graph neural networks to predict parameters of unseen networks.
result Achieved surprisingly good performance on unseen networks.
Method learns equivariances from data without custom architecture design.
problem Learning equivariances for tasks without manually designed architectures.
method Reparameterization to learn equivariant parameter sharing.
result Can learn equivariances for any finite group of transformations.
We propose Stochastic Neural Architecture Search (SNAS), an economical end-to-end solution to Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters in same round of back-propagation, while maintaining the completeness and differentiability of the NAS pipeline.…
Deep learning boosts building energy load forecasting.
problem Short-term load forecasting in buildings.
method Stacked Boosters Network architecture with sparse interactions, parameter sharing, and equivariant representations.
result Outperforms state-of-the-art models in short-term load forecasting tasks.
Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and pe…
Neural network architectures found by sophistic search algorithms achieve strikingly good test performance, surpassing most human-crafted network models by significant margins. Although computationally efficient, their design is often very complex, impairing execution speed. Additionally, finding models outside of the …
Automatically finds efficient multi-task models with less data.
problem Training separate models requires more data, parameters, and time.
method Compact search space for multi-task architectures, feature distillation for quick evaluation.
result Automatically identifies multi-task architectures that balance resource requirements and performance.
Higher granularity in MoE models boosts expressivity exponentially.
problem Expressivity of Mixture-of-Experts models with varying granularity.
method Comparing models with different numbers of active experts (granularity).
result Exponential separation in network expressivity based on granularity.
The study compares different neural network architectures for option pricing accuracy and training time.
problem Evaluating the impact of network architectures on option pricing accuracy and training time.
method Empirical investigation of various neural network architectures (plain feed forward, highway, DGM) on option pricing problems.
result Generalized highway network architecture achieves the best performance in terms of mean squared error and training time.
State-of-the-art named entity recognition (NER) systems have been improving continuously using neural architectures over the past several years. However, many tasks including NER require large sets of annotated data to achieve such performance. In particular, we focus on NER from clinical notes, which is one of the mos…
Designing the architecture for an artificial neural network is a cumbersome task because of the numerous parameters to configure, including activation functions, layer types, and hyper-parameters. With the large number of parameters for most networks nowadays, it is intractable to find a good configuration for a given …
Deep learning models predict generalization gaps without specific task or architecture.
problem Predicting when deep learning works across different tasks and architectures.
method Created a dataset of 13,500 neural networks trained on various spiral datasets and parameters. Used this dataset to train predictors for generalization gaps.
result DNNs and RNNs outperform linear models in predicting generalization gaps, with RNNs achieving R2=0.584. Paper stabilizes DARTS algorithm for better neural architecture search.
problem Weak stability of DARTS algorithm leading to unreliable results.
method Amended gradient estimation method to bridge optimization gap.
result Significant improvement in search stability and larger search spaces explored.
New findings on hidden symmetries in ReLU networks.
problem Understanding the redundancy and symmetries in ReLU network parameter space.
method Analyzing parameter settings and function classes for various network architectures.
result For certain network architectures, there are no hidden symmetries.
SigMA uses signatures and attention to estimate parameters in fBm-driven SDEs.
problem Estimating parameters in SDEs driven by fBm is challenging due to non-Markovian and semimartingale issues.
method SigMA integrates path signatures with multi-head self-attention, using convolutional and MLP layers.
result SigMA outperforms other methods in accuracy, robustness, and model compactness.
We introduce a parameter sharing scheme, in which different layers of a convolutional neural network (CNN) are defined by a learned linear combination of parameter tensors from a global bank of templates. Restricting the number of templates yields a flexible hybridization of traditional CNNs and recurrent networks. Com…
We consider a general framework for reducing the number of trainable model parameters in deep learning networks by decomposing linear operators as a product of sums of simpler linear operators. Recently proposed deep learning architectures such as CNN, KFC, Dilated CNN, etc. are all subsumed in this framework and we il…
Finding the best neural network architecture requires significant time, resources, and human expertise. These challenges are partially addressed by neural architecture search (NAS) which is able to find the best convolutional layer or cell that is then used as a building block for the network. However, once a good buil…
We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. The controller is trained with policy gradient to select a su…
New method selects neural network architectures without needing data.
problem Choosing efficient deep neural network architectures.
method Developed the deep frame potential to quantify network capacity.
result Deep frame potential correlates with generalization error.
Multi-task learning (MTL) allows deep neural networks to learn from related tasks by sharing parameters with other networks. In practice, however, MTL involves searching an enormous space of possible parameter sharing architectures to find (a) the layers or subspaces that benefit from sharing, (b) the appropriate amoun…
We introduce a new function-preserving transformation for efficient neural architecture search. This network transformation allows reusing previously trained networks and existing successful architectures that improves sample efficiency. We aim to address the limitation of current network transformation operations that…
DONNA rapidly finds optimal neural networks across diverse spaces.
problem Efficient scaling and handling of diverse architectural search-spaces in NAS.
method Three-phase pipeline: accuracy predictor, rapid evolutionary search, and optimal model finetuning.
result 100x faster than MNasNet in finding state-of-the-art architectures on-device.
We study mechanisms to characterize how the asymptotic convergence of backpropagation in deep architectures, in general, is related to the network structure, and how it may be influenced by other design choices including activation type, denoising and dropout rate. We seek to analyze whether network architecture and in…
New algorithms improve neural architecture search with faster convergence.
problem Improving efficiency and accuracy of neural architecture search.
method Geometry-aware gradient algorithms to optimize continuous relaxation of discrete search spaces.
result Exceeds state-of-the-art results on CIFAR and ImageNet benchmarks.
ThriftyNet uses a single convolutional layer recursively to maximize parameter usage.
problem Maximizing the use of parameters in deep convolutional neural networks.
method A single convolutional layer is used recursively, with normalization, non-linearities, downsampling, and shortcuts to maintain model expressivity.
result ThriftyNet achieves competitive performance with significantly fewer parameters.
EENA efficiently searches neural architectures with minimal resources.
problem Lack of direction and high computational cost in neural architecture search.
method EENA uses guided evolution with mutation and crossover operations.
result EENA designs highly effective neural architectures with minimal resources.
New CNN layer selects important channels to improve model capacity.
problem Improving model capacity under resource constraints.
method Selective allocation of channels in convolutional layers.
result New layer allows new optima that generalize better.
CogScale benchmarks AI architectures for sequential processing.
problem Evaluating AI architectures' ability to process sequential information efficiently.
method 14 scalable synthetic tasks designed to isolate cognitive and memory abilities at different scales.
result Attention mechanisms and modern state-space models consistently maintain high performance as task difficulty scales.
ForecastNet uses a time-variant deep feed-forward neural network for better multi-step-ahead time series forecasting.
problem Time-invariant architectures limit multi-step-ahead forecasting.
method ForecastNet employs a deep feed-forward architecture with time-variant parameters and interleaved outputs.
result ForecastNet outperforms other models on multi-step-ahead time series forecasting tasks.
New framework for neural networks converging to low loss without overparameterization.
problem Training deep neural networks without overparameterization assumptions.
method Construction of random sparse lifts and analysis using algebraic topology and random graph theory.
result Provable convergence to low loss for large sparse neural networks.
HMCNAS generates competitive neural architectures without human-defined parameters.
problem Lack of human-defined parameters in Neural Architecture Search.
method Combines Hidden Markov Chains and Bayesian Optimization for autonomous search space generation and competitive model generation.
result HMCNAS generates competitive models in a short time without human-defined parameters.
A hybrid neural network improves robustness in estimating vehicle parameters from noisy data.
problem Estimating parameters of a mechanical vehicle model from noisy acceleration data.
method Introduced a convolutional neural network with two objective functions: naive and hybrid.
result The hybrid objective function outperforms the naive one in robustness on noisy input data.
We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters. To overcome this, w…
Neural architecture search (NAS) has a great impact by automatically designing effective neural network architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. 104 GPU hours) makes it difficult to \emph{directly} search the architectures on large-scale tasks (e.g. ImageNet).…