ARMIN improves memory efficiency and lightness in neural networks.
problem Complex memory addressing and inefficient memory information exploitation in previous MANNs.
method ARMIN uses hidden state for automatic memory addressing and a novel RNN cell for memory integration.
result ARMIN achieves lower computational overhead and similar performances compared to vanilla LSTM.
LEMN improves memory networks for learning from streaming data.
problem Efficiently processing large data streams in memory-augmented neural networks.
method LEMN uses a RNN-based retention agent to learn and replace less important memory entries based on their importance and historical context.
result LEMN achieves significant improvements over existing methods in learning from streaming data.
Linear Memory Network separates memory and function in RNNs.
problem Complex transduction problems requiring memory and input-output exploitation.
method Conceptual separation between memory and function, using feedforward and autoencoder components.
result Efficient training and competitive performance on polyphonic music datasets.
Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is often addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which …
New memory in neural networks mimics computer architectures.
problem Learning algorithms and complex tasks with neural networks.
method Introducing a new memory to store weights for a neural controller, similar to stored-program memory in computers.
result Neural Stored-program Memory enhances neural networks' adaptability and learning capabilities.
Generative diffusion models mimic biological memory networks, encoding associative dynamics in deep neural weights.
problem Understanding long-term memory mechanisms in neuroscience and AI.
method Interpreting generative diffusion models as energy-based models and comparing them to Hopfield networks.
result Generative diffusion models can encode associative dynamics of Hopfield networks in deep neural weights.
mGRN improves multivariate time series prediction by managing marginal and joint memories.
problem Extracting dependencies in multivariate sequential data with strong serial and cross-sectional dependencies.
method Developed a novel recurrent network architecture, Memory-Gated Recurrent Networks (mGRN), with gates for marginal and joint memories.
result mGRN consistently outperforms state-of-the-art architectures on various public datasets.
Metalearned neural memory improves learning across various tasks.
problem Improving neural network adaptability and memory function.
method Augmenting neural networks with a metalearned external memory mechanism.
result The model achieves strong performance on diverse learning problems.
Augmenting a neural network with memory that can grow without growing the number of trained parameters is a recent powerful concept with many exciting applications. We propose a design of memory augmented neural networks (MANNs) called Labeled Memory Networks (LMNs) suited for tasks requiring online adaptation in class…
NEST optimizes deep learning training by placing devices efficiently across networks and memory.
problem Inefficient device placement in distributed deep learning leads to high communication and memory overhead.
method NEST uses network-, compute-, and memory-aware dynamic programming to optimize device placement.
result NEST achieves up to 2.43 times higher throughput and better memory efficiency.
DAM with MRL improves relational reasoning in MANNs.
problem Limited performance of associative memory networks on complex relational reasoning tasks.
method Distributed Associative Memory architecture with Memory Refreshing Loss.
result Enhanced relation reasoning performance of MANNs on long temporal sequence data.
In this paper, a taxonomy for memory networks is proposed based on their memory organization. The taxonomy includes all the popular memory networks: vanilla recurrent neural network (RNN), long short term memory (LSTM ), neural stack and neural Turing machine and their variants. The taxonomy puts all these networks und…
A novel memory network improves interpretability and training for EHR analysis.
problem Limited interpretability and training difficulty in traditional memory networks for EHR analysis.
method Proposes a hybrid memory architecture with explicit and blurred memory blocks for improved interpretability and training efficiency.
result The model performs comparably to state-of-the-art methods and enables ready interpretation of results.
LTM tackles long sequence language modeling by avoiding vanishing and exploding gradients.
problem Language models struggle with long sequences due to gradient issues.
method Introduces Long Term Memory network (LTM) that scales memory and weights input, avoiding overfitting.
result LTM achieves state-of-the-art perplexity results with fewer cells than previous models.
Memory-augmented neural networks improve machine translation performance.
problem Improving machine translation accuracy and flexibility.
method Evaluation of Neural Turing Machines and Differentiable Neural Computers for machine translation tasks.
result Memory-augmented neural networks perform similarly to attentional encoders on Vietnamese to English tasks but have lower BLEU scores on Romanian to English tasks.
Efficient memory layer improves graph neural networks for graph classification and regression.
problem Efficiently learning node representations and graph coarsening for arbitrary graph topology.
method Introduces a memory layer for GNNs that learns node representations and graph coarsening, and two new networks: MemGNN and GMN.
result Proposed models achieve state-of-the-art results in graph classification and regression benchmarks.
Memory split advantage: thinner networks outperform a single wide network.
problem Optimizing deep learning models with limited memory.
method Investigated training a single wide network vs. an ensemble of thinner networks with the same total number of parameters.
result An ensemble of several thinner networks outperforms a single wide network for large memory budgets.
The study reduces neural network training memory by up to 60.7x on CIFAR-10.
problem Reducing memory requirements for neural network training.
method Profiled memory usage, evaluated four techniques: sparsity, low precision, microbatching, gradient checkpointing.
result Appropriate combinations of techniques can reduce memory usage up to 60.7x on CIFAR-10.
Low-rank training improves neural network training on edge devices with non-volatile memory.
problem Training neural networks on edge devices with non-volatile memory, especially in terms of write density and auxiliary memory.
method Low-rank training scheme to address write density and auxiliary memory limitations.
result The low-rank training technique outperforms standard SGD in accuracy and weight writes.
Proposes a novel SAM operator for separate item and relational memories.
problem Limited memory interactions in neural networks.
method Introduces a Self-attentive Associative Memory (SAM) operator to separate item and relational memories.
result Achieves competitive results in various tasks, including geometry, graph, reinforcement learning, and question answering.
NeuZip compresses neural network weights to save memory during training and inference.
problem Memory constraints in neural network training and inference.
method Entropy-based dynamic weight compression.
result Significant reduction in memory usage without performance loss.
Efficiently discovers Bayesian network structure with reduced memory usage.
problem Bayesian network structure discovery is NP-hard and memory-intensive.
method Progressively leveled scoring approach with hierarchical computation.
result Achieved processing of a 28-variable Bayesian network using only memory.
Enhances sequence memory capacity in neural networks.
problem Limited sequence capacity in Hopfield-like neural networks.
method Introducing a nonlinear interaction term and a generalized pseudoinverse rule.
result Significantly increased sequence capacity with novel scaling laws.
Paper proposes low-rank gradient approximation to save memory for deep neural network training.
problem Memory limitation on mobile devices for deep neural network training.
method Approximating gradient matrices using low-rank parameterization.
result Reduces training memory by about 33.0% for Adam optimization and 4.5% relative lower word error rate on ASR personalization task.
Generalizes memory and forecasting capacities for nonlinear recurrent networks with dependent inputs.
problem Understanding memory and forecasting capabilities in networks with dependent inputs.
method Formulated bounds for memory and forecasting capacities in terms of network size and input properties.
result Proved that memory capacity for linear recurrent networks with independent inputs is given by the rank of the controllability matrix.
Paper questions RNN and LSTM's long-term memory and introduces a new definition.
problem Whether RNN and LSTM have long-term memory.
method Introduced a new definition of long-term memory and modified RNN and LSTM to test it.
result RNN and LSTM do not meet the new definition of long-term memory.
Researchers save memory on MCUs by reordering neural network operators.
problem Memory constraints on microcontrollers for neural network inference.
method Operator reordering to save memory, orthogonal to other compression methods.
result Reduced memory footprint of a CNN to deploy on an MCU with 512KB SRAM.
Learning and memory are intertwined in our brain and their relationship is at the core of several recent neural network models. In particular, the Attention-Gated MEmory Tagging model (AuGMEnT) is a reinforcement learning network with an emphasis on biological plausibility of memory dynamics and learning. We find that …
The memory capacity of linear echo state networks is accurately calculated using new numerical methods.
problem Numerical evaluations of memory capacity in recurrent neural networks often contradict theoretical bounds.
method Developed robust numerical approaches exploiting MC neutrality with respect to the input mask matrix.
result Memory curves fully agree with theory when using the proposed methods.
MemNet optimizes neural architectures for memory efficiency.
problem Memory constraints in mobile devices limit the use of large neural networks.
method Augment-trim learning with memory consumption ranking score.
result MemNet finds architectures with 24.17% less memory usage compared to state-of-the-art methods.
This paper introduces a hierarchical associative memory model with multiple layers.
problem Limitations of traditional associative memory models with only one hidden layer.
method Develops a fully recurrent model with arbitrary layers, including locally connected ones, and a corresponding energy function.
result The model can dynamically assemble memories using weights from lower layers and higher layers' rules.
Efficient echo state network with explicit memory performs well on benchmark tasks.
problem Training differentiable neural computers is difficult and time-consuming.
method Echo state network with an explicit memory.
result Echo state network can recognize all regular languages, including those contractive networks cannot.
Neural Turing Machines (NTMs) are an instance of Memory Augmented Neural Networks, a new class of recurrent neural networks which decouple computation from memory by introducing an external memory unit. NTMs have demonstrated superior performance over Long Short-Term Memory Cells in several sequence learning tasks. A n…
Deep neural networks with memory learn reduced equations from partial data.
problem Constructing governing equations for unknown dynamical systems from limited data.
method Formulate a discrete approximation of memory integrals, use deep neural networks to incorporate history terms.
result Deep neural networks can learn reduced equations with memory from partial data.
Model combines long-term and short-term memory using conceptors.
problem Transfer between long-term and short-term memory.
method Recurrent neural network with gated reservoir for short-term memory and conceptors for long-term memory.
result Standard operations on conceptors allow combining long-term memories and describing their effect on short-term memory.
Memory-enhanced model predicts tennis shots based on player history.
problem Predicting shot location and type in tennis.
method Semi-supervised Generative Adversarial Network with neural memory modules.
result The model learns player-specific behavioral patterns from match data.
A new model MTNet uses memory networks for better time series forecasting.
problem Challenges in modeling complex patterns and dependencies in multivariate time series data.
method Memory Time-series network (MTNet) with a large memory component, three encoders, and an autoregressive component.
result MTNet effectively captures long-term patterns and incorporates information from other variables.
We describe a new class of learning models called memory networks. Memory networks reason with inference components combined with a long-term memory component; they learn how to use these jointly. The long-term memory can be read and written to, with the goal of using it for prediction. We investigate these models in t…
Serenity optimizes neural network execution for edge devices by scheduling with optimal memory footprint.
problem Order of nodes in irregular neural networks affects memory footprint, complicating execution under resource constraints.
method Memory-aware compiler using dynamic programming and graph rewriting to find optimal schedules.
result Achieves optimal peak memory and further improves it with graph rewriting, reducing memory usage by 1.68x-1.86x compared to TensorFlow Lite.
MEMGAN uses memory to improve anomaly detection by isolating abnormal data.
problem Weak guarantees for detecting anomalous data in classical algorithms.
method Memory-augmented Generative Adversarial Networks (MEMGAN) with a memory module.
result MEMGAN provides strong guarantees for anomaly detection with improved reconstruction.
Extended LSTMs improve volatility prediction by 20%.
problem Predicting asset price volatility with long memory.
method Extended LSTMs with multiple flexible timescales.
result Extended LSTMs outperform rough volatility predictions by 20%.
New model DWM mimics human memory, learning to retain, ignore or forget.
problem Lack of effective separation between episodic and working memory in neural networks.
method Designed Differentiable Working Memory (DWM) inspired by psychological studies.
result DWM learns psychology-inspired tasks faster and generalizes to longer sequences.
A new memory-efficient sign language translation model reduces weight usage.
problem Memory constraints in real-time sign language translation.
method Variational Bayesian sequence-to-sequence network with Gaussian posterior and Indian Buffet Process prior.
result The proposed model achieves substantial weight compression without compromising performance.
New neural networks model complex phenomena with fewer parameters.
problem Challenges in studying higher-order interactions in neural networks.
method Introducing curved neural networks using the maximum entropy principle.
result Curved neural networks accelerate memory retrieval and exhibit explosive phase transitions.
New method optimizes memory usage in neural networks, improving sequential learning.
problem Current memory models in neural networks waste memory and computation.
method Formulated an optimization problem to maximize information storage, introduced Cached Uniform Writing.
result Proved Cached Uniform Writing optimizes memory usage and outperforms other methods.
HMN detects fake faces using memory networks and visual cues.
problem Detecting fake faces from unseen manipulation types.
method Hierarchical Memory Network (HMN) architecture.
result Superior performance in fake and fraudulent face detection.
SMGD trains low-bit neural networks with memory constraints.
problem Training large neural networks with limited memory.
method Stochastic Markov Gradient Descent (SMGD).
result Encouraging numerical results and theoretical guarantees.
Neural network uses local credit assignment to learn distant cause-effect relationships.
problem Learning distant cause-effect relationships in sequential data.
method Sparse coding in a recurrent neural network memory with local and immediate credit assignment.
result Network can predict partially-observable higher-order sequences and navigate mazes.