Latest algorithms for automatic neural architecture search perform remarkable but are basically directionless in search space and computational expensive in training of every intermediate architecture. In this paper, we propose a method for efficient architecture search called EENA (Efficient Evolution of Neural Archit…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “neural architecture evolution”
Differentiable NAS frameworks grow networks wider and deeper, revealing biases in wiring evolution.
problem Understanding the evolution of neural architecture wiring in differentiable NAS methods.
method Unified view on searching algorithms, local cost minimization, empirical and theoretical analyses.
result Implicit inductive biases cause observed searching patterns in differentiable NAS methods.
Evolution in Groups: A deeper look at synaptic cluster driven evolution of deep neural networkscs.NE
A promising paradigm for achieving highly efficient deep neural networks is the idea of evolutionary deep intelligence, which mimics biological evolution processes to progressively synthesize more efficient networks. A crucial design factor in evolutionary deep intelligence is the genetic encoding scheme used to simula…
Contrastive embeddings improve neural architecture search performance.
problem Improving performance of neural architecture search algorithms.
method Contrastive learning to identify networks based on data Jacobians and produce embeddings.
result Traditional black-box optimization algorithms can reach state-of-the-art performance with contrastive embeddings.
Deep learning models' architectures, including depth and width, are key factors influencing models' performance, such as test accuracy and computation time. This paper solves two problems: given computation time budget, choose an architecture to maximize accuracy, and given accuracy requirement, choose an architecture …
Neural Architecture Search methods are effective but often use complex algorithms to come up with the best architecture. We propose an approach with three basic steps that is conceptually much simpler. First we train N random architectures to generate N (architecture, validation accuracy) pairs and use them to train a …
New model predicts neural network performance from early training epochs, incorporating architecture impact.
problem Predicting neural network performance from early training epochs, neglecting architecture impact.
method Architecture-aware graph ordinary differential equation model.
result Model outperforms state-of-the-art methods for MLP and CNN learning curves.
Guided Evolution improves NAS efficiency and accuracy.
problem NAS methods converge to local minima and are complex.
method G-EA: guided evolutionary approach with initialization evaluation and continuous knowledge extraction.
result G-EA achieves state-of-the-art results in CIFAR-10, CIFAR-100, and ImageNet16-120.
Current Deep Reinforcement Learning algorithms still heavily rely on handcrafted neural network architectures. We propose a novel approach to automatically find strong topologies for continuous control tasks while only adding a minor overhead in terms of interactions in the environment. To achieve this, we combine Neur…
Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts. While recent approaches have achieved state-of-the-art predictive performance for image recognition, they are problematic under resource constraints for two reasons: (1)the ne…
DeepStreamCE detects new classes in streaming deep neural networks.
problem Detecting new classes in deep neural networks in a streaming environment.
method Uses autoencoder and MCOD stream-based clustering for real-time concept evolution detection.
result DeepStreamCE outperforms OpenMax in identifying concept evolution.
NPENAS improves neural architecture search efficiency and accuracy.
problem Efficient and accurate neural architecture search (NAS) for minimizing search costs.
method Proposes NPENAS, a neural predictor guided evolutionary algorithm that enhances exploration ability of evolutionary algorithms.
result NPENAS-BO and NPENAS-NP outperform existing NAS algorithms on NASBench-201, NASBench-101, and DARTS.
Neural networks for structured data like graphs have been studied extensively in recent years. To date, the bulk of research activity has focused mainly on static graphs. However, most real-world networks are dynamic since their topology tends to change over time. Predicting the evolution of dynamic graphs is a task of…
A key feature of intelligent behavior is the ability to learn abstract strategies that transfer to unfamiliar problems. Therefore, we present a novel architecture, based on memory-augmented networks, that is inspired by the von Neumann and Harvard architectures of modern computers. This architecture enables the learnin…
In this paper, we model Density Evolution (DE) using Recurrent Neural Networks (RNNs) with the aim of designing capacity-approaching Irregular Low-Density Parity-Check (LDPC) codes for binary erasure channels. In particular, we present a method for determining the coefficients of the degree distributions, characterizin…
We propose Stochastic Neural Architecture Search (SNAS), an economical end-to-end solution to Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters in same round of back-propagation, while maintaining the completeness and differentiability of the NAS pipeline.…
Taking inspiration from biological evolution, we explore the idea of "Can deep neural networks evolve naturally over successive generations into highly efficient deep neural networks?" by introducing the notion of synthesizing new highly efficient, yet powerful deep neural networks over successive generations via an ev…
Emergence of Globally Attracting Fixed Points in Deep Neural Networks With Nonlinear Activationscs.LG
The paper studies how neural networks evolve representations, finding a unique fixed point for nonlinear activations.
problem Understanding how neural networks transform input data across layers.
method Theoretical framework for the evolution of the kernel sequence, using mean-field regime and Hermite polynomials.
result For nonlinear activations, the kernel sequence converges globally to a unique fixed point.
Predict accuracy of neural architectures using non-neural models.
problem Improving efficiency and effectiveness of neural architecture search.
method Used gradient boosting decision tree (GBDT) for accuracy prediction and pruned the search space based on features learned from GBDT.
result GBDT-based predictor achieves comparable or better accuracy than neural network-based predictors and is 22x more sample efficient on NASBench-101.
Neural SVEs model complex systems with memory, outperforming traditional methods.
problem Modeling systems with memory effects and irregular behavior.
method Introducing neural stochastic Volterra equations as a physics-inspired architecture.
result Neural SVEs outperform neural SDEs and DeepONets in various applications.
The evolution of MobileNets has laid a solid foundation for neural network applications on mobile end. With the latest MobileNetV3, neural architecture search again claimed its supremacy in network design. Unfortunately, till today all mobile methods mainly focus on CPU latencies instead of GPU, the latter, however, is…
AgEBO-Tabular combines NAS and hyperparameter tuning for fast, high-performing tabular models.
problem Developing high-performing predictive models for large tabular data sets is challenging.
method Combines aging evolution NAS and asynchronous Bayesian optimization for hyperparameter tuning in data-parallel training.
result Automatically discovered neural network models outperform state-of-the-art AutoML ensembles in inference speed by two orders of magnitude.
DyNODE uses neural ODEs to model system dynamics in continuous control tasks.
problem Modeling the dynamics of systems in continuous control tasks.
method Neural Ordinary Differential Equations (ODEs) combined with actor-critic RL.
result DyNODE outperforms standard neural networks in sample efficiency and predictive performance.
We propose a population-based Evolutionary Stochastic Gradient Descent (ESGD) framework for optimizing deep neural networks. ESGD combines SGD and gradient-free evolutionary algorithms as complementary algorithms in one framework in which the optimization alternates between the SGD step and evolution step to improve th…
Neural Architecture Search (NAS) has shown great success in automating the design of neural networks, but the prohibitive amount of computations behind current NAS methods requires further investigations in improving the sample efficiency and the network evaluation cost to get better results in a shorter time. In this …
New method predicts dynamic relationships in terrorist networks.
problem Dynamic co-evolution of multiplex graphs and nodal attributes in terrorism networks.
method Time-varying stochastic latent factor models with neural network Gaussian processes.
result Superior performance in predicting unobserved dynamic relationships.
We present a new method of blackbox optimization via gradient approximation with the use of structured random orthogonal matrices, providing more accurate estimators than baselines and with provable theoretical guarantees. We show that this algorithm can be successfully applied to learn better quality compact policies …
NDM incorporates geometric structure into neural networks for better optimization and interpretability.
problem Efficient and interpretable deep learning architectures.
method NDM is a neural network architecture that explicitly incorporates geometric structure into its design, using a Coordinate Layer, Geometric Layer, and Evolution Layer.
result NDM provides intrinsic regularization, enhancing generalization and robustness.
LAVARNET predicts multivariate time series by estimating causal variable relationships.
problem Forecasting multivariate time series requires understanding causal interrelationships among variables.
method LAVARNET is a neural network architecture that estimates causal effects and predicts future values.
result LAVARNET outperforms other models on various real-world data sets.
ReGENN improves time series forecasting by considering inter and intra-temporal relationships.
problem Achieving reliable predictions in real-world time series applications.
method ReGENN combines graph evolution with deep recurrent learning to model dynamic dependencies among multiple variables.
result Sound improvement of up to 64.87% over competing algorithms in time-series forecasting.
There has been significant recent interest towards achieving highly efficient deep neural network architectures. A promising paradigm for achieving this is the concept of evolutionary deep intelligence, which attempts to mimic biological evolution processes to synthesize highly-efficient deep neural networks over succe…
Multi-objective Neural Architecture Search (NAS) aims to discover novel architectures in the presence of multiple conflicting objectives. Despite recent progress, the problem of approximating the full Pareto front accurately and efficiently remains challenging. In this work, we explore the novel reinforcement learning …
NAS improves gliomas segmentation on MRI scans.
problem Designing optimal deep learning architectures for brain tumor segmentation.
method Neural architecture search with probabilistic parameter learning for MRI brain tumor segmentation.
result Two optimal neural architectures for brain tumor segmentation were discovered.
A neural flow method minimizes Willmore energy for 2-surfaces in 3D space.
problem Minimizing Willmore energy for closed oriented 2-surfaces in 3D space.
method Introducing neural Willmore flow to model and minimize the Willmore energy using neural architectures.
result The neural flow reproduces expected round sphere and Clifford torus for genus 0 and 1 surfaces, respectively, and finds minimal Willmore surfaces for genus 2.
Recurrent neural networks (RNNs) are notoriously difficult to train. When the eigenvalues of the hidden to hidden weight matrix deviate from absolute value 1, optimization becomes difficult due to the well studied issue of vanishing and exploding gradients, especially when trying to learn long-term dependencies. To cir…
Generative models create personalized patient health simulations.
problem Creating accurate digital twins for personalized medicine.
method Neural network architecture for conditional generative models of clinical trajectories.
result Same architecture generates accurate twins across 13 indications.
DAFNO learns surrogates for complex systems on irregular geometries.
problem Learning accurate surrogates for complex physical systems on irregular geometries.
method DAFNO incorporates a smoothed characteristic function in the integral layer architecture of FNOs, leveraging FFT for rapid computations.
result DAFNO achieves state-of-the-art accuracy on material modeling and airfoil simulation datasets.
Deep neural networks continue to show improved performance with increasing depth, an encouraging trend that implies an explosion in the possible permutations of network architectures and hyperparameters for which there is little intuitive guidance. To address this increasing complexity, we propose Evolutionary DEep Net…
SGRNN models evolving graph data for better property prediction.
problem Modeling evolving graph data for property prediction.
method SGRNN uses stochastic latent variables to capture both node attribute and topology evolution, with semi-implicit variational inference and KL-divergence simplification.
result SGRNN improves property prediction on real-world datasets.
The study explains how neural networks align their kernels to target functions during training.
problem Understanding how neural networks align their kernels to target functions during training.
method Theoretical analysis of kernel evolution in toy models and deep networks.
result Kernel alignment naturally emerges during training to accelerate convergence and improve generalization.
CTRNNs improve blood glucose forecasting in ICU, outperforming traditional models.
problem Forecasting blood glucose in ICU with irregular measurements.
method Continuous time autoregressive recurrent neural networks (CTRNNs) using neural ODE or neural flow layers.
result CTRNNs generally outperform traditional autoregressive models in probabilistic forecasting of blood glucose.
Proposes NSSNNs to predict nonseparable Hamiltonian systems.
problem Predicting nonseparable Hamiltonian systems with coupled dynamics.
method Augmented symplectic time integrator to decouple position and momentum.
result Long-term, accurate, and robust predictions for large-scale Hamiltonian systems.
A Distribution-to-Distribution Neural Probabilistic Forecasting Framework for Dynamical Systemsstat.ML
Develops a neural framework for probabilistic forecasting of dynamical systems.
problem Uncertainty quantification in dynamical systems using trajectory-oriented approaches.
method D2D neural probabilistic forecasting framework using kernel mean embeddings and mixture density networks.
result The D2D model captures distributional evolution in chaotic systems and produces skillful probabilistic forecasts.
We focus on the problem of streaming recommender system and explore novel collaborative filtering algorithms to handle the data dynamicity and complexity in a streaming manner. Although deep neural networks have demonstrated the effectiveness of recommendation tasks, it is lack of explorations on integrating probabilis…
Graph neural network using Beltrami flow for feature and topology evolution.
problem Efficient feature learning and topology evolution on graphs.
method Discretized Beltrami flow applied to graph neural networks with positional encodings.
result Achieves state-of-the-art results on various benchmarks.
Deep learning dynamics and NTK evolution studied through diverse measures.
problem Understanding the training dynamics of deep neural networks and their loss landscapes.
method Phenomenological analysis of training dynamics in multiple architectures and datasets.
result Training dynamics exhibit a chaotic initial transient followed by a stable phase, with the NTK evolving to match full network performance.
GD-VAEs learn dynamics from observations using geometric and topological information.
problem Learning parsimonious representations of nonlinear dynamics from observations.
method Develops data-driven methods incorporating geometric and topological information using Variational Autoencoders (VAEs).
result GD-VAEs provide methods for learning reduced dimensional representations of nonlinear dynamics.
VHVM models financial time series with varying volatility.
problem Modeling heteroscedastic behavior in multivariate financial time series.
method Variational autoencoder and recurrent neural network for capturing relationships and temporal dynamics.
result VHVM outperforms GARCH and SV models on FX datasets.