This study proposes a logic architecture for the high-speed and power efficiently training of a gradient boosting decision tree model of binary classification. We implemented the proposed logic architecture on an FPGA and compared training time and power efficiency with three general GBDT software libraries using CPU a…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Efficient Architecture”
Latest algorithms for automatic neural architecture search perform remarkable but are basically directionless in search space and computational expensive in training of every intermediate architecture. In this paper, we propose a method for efficient architecture search called EENA (Efficient Evolution of Neural Archit…
Improves neural architecture search methods to be more stable and efficient.
problem Neural architecture search methods are unstable and sensitive to hyperparameters.
method Discusses practical considerations to improve stability and efficiency.
result Improves overall performance of neural architecture search methods.
We introduce a new function-preserving transformation for efficient neural architecture search. This network transformation allows reusing previously trained networks and existing successful architectures that improves sample efficiency. We aim to address the limitation of current network transformation operations that…
Efficient search methods can outperform random search on challenging tasks.
problem Comparing the performance of efficient and random search methods in neural architecture search.
method Comparison of weight sharing and random search methods on progressively larger search spaces for image classification and detection.
result Efficient search methods can provide substantial gains over random search on large, realistic tasks.
A graph VAE framework optimizes neural architectures in a continuous space.
problem Discovering efficient neural architectures in a discrete space.
method Graph VAE framework with VAE and GNN components, joint learning of predictors and decoders.
result The framework discovers powerful neural architectures with both excellent performance and high computational efficiency.
Neural architecture search has been shown to hold great promise towards the automation of deep learning. However in spite of its potential, neural architecture search remains quite costly. To this point, we propose a novel gradient-based framework for efficient architecture search by sharing information across several …
This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architectu…
Simplified NAS for GNN architectures improves efficiency and expressiveness.
problem Efficient and effective discovery of optimal GNN architectures.
method SNAG framework with a novel search space and reinforcement learning.
result SNAG framework outperforms human-designed and existing NAS methods.
We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human ex…
FiGS searches over a larger space of architectures for efficient mobile models.
problem Designing small, efficient deep networks for mobile devices.
method Differentiable search method using sparse regularization and Logistic-Sigmoid distribution.
result FiGS produces state-of-the-art parameter-efficient models on ImageNet and improves object detection performance.
MiLeNAS improves neural architecture search by reducing approximation errors and achieving better accuracy.
problem Improving efficiency and accuracy in neural architecture search (NAS).
method Mixed-level reformulation (MiLeNAS) to optimize efficiently and reliably.
result MiLeNAS achieves lower validation error and higher accuracy than bilevel optimization methods.
Neural network architectures found by sophistic search algorithms achieve strikingly good test performance, surpassing most human-crafted network models by significant margins. Although computationally efficient, their design is often very complex, impairing execution speed. Additionally, finding models outside of the …
New deep learning architecture learns martingales efficiently.
problem Efficiently learning martingales in financial derivatives pricing.
method High-order weak approximation algorithms of Runge-Kutta type.
result Deep neural networks based on this architecture learn martingales effectively.
Recent advances in Neural Architecture Search (NAS) have produced state-of-the-art architectures on several tasks. NAS shifts the efforts of human experts from developing novel architectures directly to designing architecture search spaces and methods to explore them efficiently. The search space definition captures pr…
For network architecture search (NAS), it is crucial but challenging to simultaneously guarantee both effectiveness and efficiency. Towards achieving this goal, we develop a differentiable NAS solution, where the search space includes arbitrary feed-forward network consisting of the predefined number of connections. Be…
RARTS improves neural architecture search efficiency and accuracy.
problem Efficiency and accuracy issues in first-order DARTS.
method Formulates a relaxed architecture search (RARTS) method that uses whole dataset splitting.
result RARTS outperforms DARTS and its variants in accuracy and efficiency.
The use of automatic methods, often referred to as Neural Architecture Search (NAS), in designing neural network architectures has recently drawn considerable attention. In this work, we present an efficient NAS approach, named HM- NAS, that generalizes existing weight sharing based NAS approaches. Existing weight shar…
Smooth embedding space improves NAS performance.
problem Efficiently predicting good neural architectures.
method Two-sided variational graph autoencoder.
result Smooth embedding space facilitates extrapolation to unseen architectures.
Study compares LSTM and ANN architectures for forex prediction, finding ANN more efficient.
problem Forex time series prediction efficiency and resource usage.
method Comparison of Long Short-Term Memory (LSTM) and specialized ANN architectures.
result Specialized ANN architecture performs better with fewer resources and faster execution.
Enhanced tabular benchmarks for energy-efficient neural architecture search.
problem Energy consumption in deep learning models.
method Introducing EC-NAS, an enhanced tabular benchmark with energy consumption data.
result EC-NAS reveals a balance between energy usage and accuracy in neural architecture search.
A new LSTM architecture improves time series forecasting efficiency.
problem Efficiency and accuracy in time series forecasting using linear models.
method Attention-free LSTM architecture for time series prediction.
result Improved prediction capacity and efficiency compared to LSTM.
Designing energy-efficient networks is of critical importance for enabling state-of-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited. Recently, Liu et al. (2019) framed the search of efficient neural architectures into a continuous splitting process: it itera…
Graph Metanetworks process diverse neural architectures efficiently.
problem Processing diverse neural architectures efficiently.
method Builds metanetworks using graph neural networks to process graphs representing input neural networks.
result Proves GMNs are expressive and equivariant to parameter permutation symmetries.
Efficient algorithm finds fast Transformer models.
problem Slow inference time of Transformer models.
method Decompose Transformer architecture into components, use sampling-based one-shot search.
result Achieved 10% to 30% speedup on pre-trained BERT and 70% on top of a previous state-of-the-art model.
Neural architecture search (NAS) has been proposed to automatically tune deep neural networks, but existing search algorithms, e.g., NASNet, PNAS, usually suffer from expensive computational cost. Network morphism, which keeps the functionality of a neural network while changing its neural architecture, could be helpfu…
Multi-task learning holds the promise of less data, parameters, and time than training of separate models. We propose a method to automatically search over multi-task architectures while taking resource constraints into consideration. We propose a search space that compactly represents different parameter sharing strat…
The neural architecture search (NAS) algorithm with reinforcement learning can be a powerful and novel framework for the automatic discovering process of neural architectures. However, its application is restricted by noncontinuous and high-dimensional search spaces, which result in difficulty in optimization. To resol…
The recent advent of `Internet of Things' (IOT) has increased the demand for enabling AI-based edge computing. This has necessitated the search for efficient implementations of neural networks in terms of both computations and storage. Although extreme quantization has proven to be a powerful tool to achieve significan…
One-Shot Neural architecture search (NAS) attracts broad attention recently due to its capacity to reduce the computational hours through weight sharing. However, extensive experiments on several recent works show that there is no positive correlation between the validation accuracy with inherited weights from the supe…
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal. Likewise, we demonstrate that the common practice of using very few …
In neural architecture search, the structure of the neural network to best model a given dataset is determined by an automated search process. Efficient Neural Architecture Search (ENAS), proposed by Pham et al. (2018), has recently received considerable attention due to its ability to find excellent architectures with…
NPENAS improves neural architecture search efficiency and accuracy.
problem Efficient and accurate neural architecture search (NAS) for minimizing search costs.
method Proposes NPENAS, a neural predictor guided evolutionary algorithm that enhances exploration ability of evolutionary algorithms.
result NPENAS-BO and NPENAS-NP outperform existing NAS algorithms on NASBench-201, NASBench-101, and DARTS.
MixerFlow combines MLP-Mixer with normalizing flows for efficient image modeling.
problem Efficiently modeling complex image densities using generative models.
method Proposes MixerFlow, a novel architecture based on MLP-Mixer for normalizing flows.
result Demonstrates improved density estimation and better scaling with higher image resolutions.
Paper uses CMAB to improve NAS efficiency and accuracy.
problem Improving efficiency and accuracy of NAS for DNNs.
method Formulated NAS as CMAB, used Nested Monte-Carlo Search.
result Discovered cell structure achieves comparable accuracy to state-of-the-art, 20x faster.
Unified transformer-based LT-TTD improves ranking efficiency and quality.
problem Decoupled L1 and L2 models in recommendation and search systems cause irreversible error propagation and suboptimal ranking.
method LT-TTD combines two-tower models with transformer expressivity in a unified listwise learning framework, providing theoretical guarantees and UPQE evaluation.
result LT-TTD reduces irretrievable relevant items and achieves better global optimization than disjoint training.
Proposes a method to improve neural architectures reproducibly.
problem Lack of reproducibility in Neural Architecture Transformer (NAT).
method Differentiable Neural Architecture Transformation (DNAT).
result DNAT outperforms NAT and is applicable to various models and datasets.
New findings on how convolutional architectures approximate time series data.
problem Understanding the approximation properties of convolutional architectures in time series modeling.
method Mathematical analysis of convolutional architectures applied to time series modeling.
result A new definition of spectrum-based regularity for measuring temporal relationships under convolutional approximation.
Gradients of neural networks can be computed efficiently for any architecture, but some applications require differential operators with higher time complexity. We describe a family of restricted neural network architectures that allow efficient computation of a family of differential operators involving dimension-wise…
SOTERIA optimizes neural networks for secure inference with minimal overhead.
problem Protecting user privacy in ML-as-a-service models with low overhead.
method Neural architecture search with dual objectives of accuracy and cryptographic efficiency.
result SOTERIA constructs efficient models for secure inference.
We propose Stochastic Neural Architecture Search (SNAS), an economical end-to-end solution to Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters in same round of back-propagation, while maintaining the completeness and differentiability of the NAS pipeline.…
A method to reduce memory usage in NAS by pruning the search space.
problem High GPU memory consumption in One-Shot NAS techniques.
method Utilising Zero-Shot NAS to prune the search space before applying One-Shot NAS.
result Reduces memory consumption by 81% while maintaining accuracy.
Applying deep neural networks (DNNs) in mobile and safety-critical systems, such as autonomous vehicles, demands a reliable and efficient execution on hardware. Optimized dedicated hardware accelerators are being developed to achieve this. However, the design of efficient and reliable hardware has become increasingly d…
Deep learning models' architectures, including depth and width, are key factors influencing models' performance, such as test accuracy and computation time. This paper solves two problems: given computation time budget, choose an architecture to maximize accuracy, and given accuracy requirement, choose an architecture …
The design of neural network architectures is an important component for achieving state-of-the-art performance with machine learning systems across a broad array of tasks. Much work has endeavored to design and build architectures automatically through clever construction of a search space paired with simple learning …
Bonsai-Net efficiently discovers state-of-the-art models with fewer parameters.
problem Efficiently discovering state-of-the-art neural architectures with minimal computational expense.
method Bonsai-Net uses a modified differential pruner to explore a relaxed search space.
result Bonsai-Net consistently discovers better architectures than random search with fewer parameters.
New method grows deep networks efficiently by dynamically pruning and growing layers.
problem Training deep networks is computationally expensive and inefficient.
method Structured continuous sparsification starting from a small seed architecture.
result 49.7% inference FLOPs and 47.4% training FLOPs savings with 75.2% top-1 accuracy.
A new framework generates large hierarchical search spaces for neural architectures.
problem Discovering neural architectures from simple blocks is hard.
method Context-free grammars for a unified, scalable search space.
result Efficiently searches over complete architectures, outperforming existing methods.