LSTMs and GRUs are the most common recurrent neural network architectures used to solve temporal sequence problems. The two architectures have differing data flows dealing with a common component called the cell state (also referred to as the memory). We attempt to enhance the memory by presenting a modification that w…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,786 papers · 148 categories
Trend · papers per month
48 results for “compact architecture”
Paper tackles NAS problem by modeling it as a sparse supernet.
problem Neural Architecture Search (NAS) problem, particularly Mixed-Path Search.
method Model NAS as a sparse supernet with sparsity constraints. Use hierarchical accelerated proximal gradient algorithm for optimization.
result Proposed method finds compact, general, and powerful neural architectures.
In this paper, we introduce Channel-wise recurrent convolutional neural networks (RecNets), a family of novel, compact neural network architectures for computer vision tasks inspired by recurrent neural networks (RNNs). RecNets build upon Channel-wise recurrent convolutional (CRC) layers, a novel type of convolutional …
New architecture improves decision-making in dense traffic.
problem Designing accurate and compact learning architectures for autonomous vehicles in crowded conditions.
method Attention-based architecture that accounts for interactions between vehicles.
result Significant performance gains and interpretable interaction patterns.
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public arc…
We present a new method of blackbox optimization via gradient approximation with the use of structured random orthogonal matrices, providing more accurate estimators than baselines and with provable theoretical guarantees. We show that this algorithm can be successfully applied to learn better quality compact policies …
Automates graph convolutional network design for semi-supervised node classification.
problem Designing optimal graph convolutional network architectures for semi-supervised node classification.
method An automatic process to define a problem-specific architecture based on graph structure.
result The proposed method outperforms existing methods in classification performance and network compactness.
NAT optimizes neural architectures to improve performance without extra cost.
problem Redundant operations in neural architectures consume memory and degrade performance.
method Transformed Markov Decision Process (MDP) and reinforcement learning to replace redundant operations with more efficient ones.
result Transformed architectures outperform original and existing methods on CIFAR-10 and ImageNet datasets.
Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscs.LG
Dynamic Sparse Training finds efficient sparse networks from scratch.
problem Finding efficient sparse neural networks.
method Jointly optimizes network parameters and sparsity with trainable thresholds.
result Achieves state-of-the-art performance with minimal performance loss.
DeepPeep attacks DNN architectures to reveal design details, posing IP theft risks.
problem Protecting DNN architecture from IP theft in cloud-based services.
method Two-stage attack methodology exploiting design characteristics.
result DeepPeep successfully reverses-engineers compact DNN architectures.
A new autoencoder architecture captures multiscale data.
problem Multiscale spatio-temporal data representation.
method Integrates multigrid methods, convolutional autoencoders, and transfer learning.
result Adaptive, hierarchical architecture captures different scaled features dynamically.
We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can also surpass their unstructured or fully connected counterparts. As we show, even more compact topologi…
Long short-term memory (LSTM) has been widely used for sequential data modeling. Researchers have increased LSTM depth by stacking LSTM cells to improve performance. This incurs model redundancy, increases run-time delay, and makes the LSTMs more prone to overfitting. To address these problems, we propose a hidden-laye…
New framework models neural systems with random architecture on manifolds.
problem Complex, uncertain systems with non-Gaussian outputs.
method Latent random field on compact manifold generates neural architecture and weights.
result Synthetic neural systems can produce stochastic outputs for deterministic inputs.
NPAS trains neural networks with a fixed parameter budget, improving performance and compactness.
problem Training neural networks requires memory, and existing methods struggle with arbitrary parameter budgets.
method NPAS learns to share parameters automatically, covering low and high budgets.
result NPAS and SSNs improve network performance and compactness across various tasks.
New architectures improve KANs, making them more interpretable and accurate.
problem Improving Kolmogorov-Arnold networks while maintaining interpretability.
method Overprovisioned architectures combined with sparsification, deep supervision, and depth selection, optimized with a minimum description length objective.
result Combining sparsification with depth selection achieves competitive or superior accuracy while discovering smaller models.
Deep neural network learns compact representations for driving tasks.
problem Improving autonomous driving through better neural network representations.
method Inspired by human brain's hierarchical structure and predictive nature, the paper proposes a deep learning framework that learns compact representations of driving concepts.
result The paper successfully learns compact representations using as few as 16 neural units for car and lane concepts.
New method constructs neural network architecture iteratively to optimize learning.
problem Difficulty in setting optimal random weights and biases for feedforward neural networks.
method Iteratively constructs hidden nodes, accepting only those that significantly reduce training error.
result Faster convergence and more compact network architecture with fewer significant neurons.
We explore Deep Reinforcement Learning in a parameterized action space. Specifically, we investigate how to achieve sample-efficient end-to-end training in these tasks. We propose a new compact architecture for the tasks where the parameter policy is conditioned on the output of the discrete action policy. We also prop…
Robotic table tennis learns efficient policies to return balls at 100Hz.
problem Developing efficient robotic table tennis strategies.
method Model-free reinforcement learning using evolutionary search on CNN-based policies.
result Robots can develop multi-modal styles (forehand and backhand) with 80% return rate.
New method grows deep networks efficiently by dynamically pruning and growing layers.
problem Training deep networks is computationally expensive and inefficient.
method Structured continuous sparsification starting from a small seed architecture.
result 49.7% inference FLOPs and 47.4% training FLOPs savings with 75.2% top-1 accuracy.
Various forms of representations may arise in the many layers embedded in deep neural networks (DNNs). Of these, where can we find the most compact representation? We propose to use a pruning framework to answer this question: How compact can each layer be compressed, without losing performance? Most of the existing DN…
New framework SEU solves lifelong learning's catastrophic forgetting issue.
problem Catastrophic forgetting in lifelong learning.
method Introduces Neural Architecture Search into lifelong learning to dynamically adapt model structures for different tasks.
result Achieves higher accuracy with significantly smaller model size (25-33% of state-of-the-art methods).
The mixture of experts (MoE) model is a popular neural network architecture for nonlinear regression and classification. The class of MoE mean functions is known to be uniformly convergent to any unknown target function, assuming that the target function is from Sobolev space that is sufficiently differentiable and tha…
This paper reviews methods to create compact neural networks for IoT applications.
problem Complex deep neural networks are costly and slow, hindering real-world deployment.
method Automatic synthesis of compact, accurate DNN/LSTM models.
result Compact neural networks reduce energy consumption, memory, and inference time.
Predict accuracy of neural architectures using non-neural models.
problem Improving efficiency and effectiveness of neural architecture search.
method Used gradient boosting decision tree (GBDT) for accuracy prediction and pruned the search space based on features learned from GBDT.
result GBDT-based predictor achieves comparable or better accuracy than neural network-based predictors and is 22x more sample efficient on NASBench-101.
Bayesian Dark Knowledge is a method for compressing the posterior predictive distribution of a neural network model into a more compact form. Specifically, the method attempts to compress a Monte Carlo approximation to the parameter posterior into a single network representing the posterior predictive distribution. Fur…
We present LumièreNet, a simple, modular, and completely deep-learning based architecture that synthesizes, high quality, full-pose headshot lecture videos from instructor's new audio narration of any length. Unlike prior works, LumièreNet is entirely composed of trainable neural network modules to learn mapping functi…
GNMC reduces XCSF population size while preserving function approximation and policy accuracy.
problem Population bloat in XCSF compaction.
method Introduced GNMC, a novel compaction algorithm.
result GNMC reduces population size significantly without compromising function approximation or policy accuracy.
We present a novel cost function for semi-supervised learning of neural networks that encourages compact clustering of the latent space to facilitate separation. The key idea is to dynamically create a graph over embeddings of labeled and unlabeled samples of a training batch to capture underlying structure in feature …
To discover powerful yet compact models is an important goal of neural architecture search. Previous two-stage one-shot approaches are limited by search space with a fixed depth. It seems handy to include an additional skip connection in the search space to make depths variable. However, it creates a large range of per…
LLoCa makes any network Lorentz-equivariant, achieving high accuracy and efficiency.
problem Limitations of specialized layers in Lorentz-equivariant neural networks.
method LLoCa framework using local reference frames and geometric message passing.
result Models achieve competitive and state-of-the-art accuracy on particle physics tasks.
MeliusNet improves binary neural networks to match MobileNet-v1 accuracy.
problem Achieving high accuracy with binary neural networks on mobile devices.
method Alternating DenseBlocks and ImprovementBlocks to increase feature capacity and quality.
result MeliusNet matches MobileNet-v1 accuracy on ImageNet, improving binary network performance.
PDE-NetGen 1.0: from symbolic PDE representations of physical processes to trainable neural network representationsphysics.comp-ph
PDE-NetGen converts physical equations to neural networks for various scientific problems.
problem Bridging physics and deep learning for efficient neural network architectures.
method Combines symbolic calculus and neural network generation to translate PDEs into NN architectures.
result Generates compact, computationally-efficient physics-informed NN architectures.
Technological development aims to produce generations of increasingly efficient robots able to perform complex tasks. This requires considerable efforts, from the scientific community, to find new algorithms that solve computer vision problems, such as object recognition. The diffusion of RGB-D cameras directed the stu…
Brain computer interfaces (BCI) enable direct communication with a computer, using neural activity as the control signal. This neural signal is generally chosen from a variety of well-studied electroencephalogram (EEG) signals. For a given BCI paradigm, feature extractors and classifiers are tailored to the distinct ch…
New conditions ensure deep neural networks can approximate any function on non-Euclidean spaces.
problem Understanding how to modify neural network architectures to approximate functions on non-Euclidean spaces.
method Developed conditions for feature and readout maps that preserve universal approximation capabilities.
result Modified architectures can deterministically approximate any classifier on non-Euclidean spaces.
AdaptiveNet tackles disease progression prediction in rheumatoid arthritis using deep neural networks.
problem Predicting disease progression in rheumatoid arthritis using clinical data.
method AdaptiveNet, a novel recurrent neural network architecture, that handles multiple lists of different events and missing data.
result AdaptiveNet outperforms classical baselines in disease progression prediction.
Model compresses event-like contexts using gated surprise signals.
problem Perceiving a dynamic world as organized events.
method Hierarchical, surprise-gated recurrent neural network architecture.
result Achieves best performance on multiple event processing tasks.
Regularizes RNNs to be invariant to input order.
problem Making RNNs invariant to input order.
method Stochastic regularization to enforce permutation invariance.
result Improves model performance on permutation invariant tasks.
Bayesian neural network predicts planetary instability.
problem Predicting planetary instability in compact systems.
method Novel Bayesian neural network trained on raw orbital elements.
result Model predicts planetary instability times with high accuracy and robust generalization.
We introduce a deep multitask architecture to integrate multityped representations of multimodal objects. This multitype exposition is less abstract than the multimodal characterization, but more machine-friendly, and thus is more precise to model. For example, an image can be described by multiple visual views, which …
Nowozin \textit{et al} showed last year how to extend the GAN \textit{principle} to all -divergences. The approach is elegant but falls short of a full description of the supervised game, and says little about the key player, the generator: for example, what does the generator actually converge to if solving the GAN…
Neural networks can approximate functions uniformly across various measures.
problem Universal approximation of functions across different probability measures.
method Proving neural networks are dense in Orlicz spaces, extending classical theorems.
result Neural networks uniformly approximate functions for weakly compact families of measures.
Dirichlet pruning compresses neural networks by removing unimportant units.
problem Compressing large neural network models without sacrificing performance.
method Assigns Dirichlet distribution over network layers' units and uses variational inference to estimate parameters.
result Achieves state-of-the-art compression performance on larger architectures like VGG and ResNet.
We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian). The approach reduces the otherwise tedious manual derivation of these mat…
We show that compact fully connected (FC) deep learning networks trained to classify wireless protocols using a hierarchy of multiple denoising autoencoders (AEs) outperform reference FC networks trained in a typical way, i.e., with a stochastic gradient based optimization of a given FC architecture. Not only is the co…
Generates high-quality images using sparse DCT representations.
problem Challenges in generating images due to high dimensionality.
method Transformers trained on sparse DCT block sequences.
result Competitive image generation quality with state-of-the-art methods.