This paper uses deep learning to improve memory prefetching.
problem Improving memory performance in software optimizations.
method Relating prefetching strategies to n-gram models and using recurrent neural networks.
result Neural networks consistently outperform traditional prefetching strategies in benchmark tests.
DEAP Cache learns prefetching, eviction, and admission using machine learning.
problem Improving cache performance through better prefetching, eviction, and admission strategies.
method End-to-end pipeline using machine learning, inspired by pretraining on large corpora and online reinforcement learning.
result Optimal policy distribution between two orthogonal eviction strategies based on frequency and recency.
We present a general framework for accelerating a large class of widely used Markov chain Monte Carlo (MCMC) algorithms. Our approach exploits fast, iterative approximations to the target density to speculatively evaluate many potential future steps of the chain in parallel. The approach can accelerate computation of t…
New model learns execution of code using GNNs.
problem Stagnation of computer system performance due to Moore's Law.
method Multi-task GNN over low-level code and program state.
result Improved performance on dynamic tasks (26% and 45% over state-of-the-art).
Optimizes parallel training of linear models, improving convergence.
problem Improving convergence of parallel training of linear models.
method Data partitioning scheme across threads to improve convergence.
result Achieved up to 42x speedup in convergence compared to state of the art implementations.
RHOMP improves prediction accuracy for user trails.
problem Predicting user actions in web browsing and geolocation.
method Retrospective higher-order Markov process (RHOMP) for sequences of data.
result RHOMP outperforms higher-order Markov chains and other methods in prediction accuracy.
Proposes a novel SAM operator for separate item and relational memories.
problem Limited memory interactions in neural networks.
method Introduces a Self-attentive Associative Memory (SAM) operator to separate item and relational memories.
result Achieves competitive results in various tasks, including geometry, graph, reinforcement learning, and question answering.
Dual memory improves reinforcement learning efficiency.
problem Training inefficiency in reinforcement learning.
method Introduces a dual memory structure with a main and cache memory.
result Dual memory structure leads to higher scores in reinforcement learning environments.
Mathematical models link perception and memory formation.
problem Linking perception and memory formation.
method Tensor decompositions and latent representations.
result Active semantic decoding process in perception.
Stable Hadamard Memory improves reinforcement learning by efficiently managing memory.
problem Memory models struggle in partially observable reinforcement learning environments.
method Introduces a novel memory model using the Hadamard product for efficient memory management and updates.
result Significantly outperforms state-of-the-art memory-based methods on challenging benchmarks.
This paper improves RNN memory capacity for long sequences through learning associative memory update rules.
problem Challenges in RNNs remembering long sequences.
method Jointly learns memory update rule with task objective and uses multiple associative memories.
result Improves memory capacity for long sequence encoding.
ARMIN improves memory efficiency and lightness in neural networks.
problem Complex memory addressing and inefficient memory information exploitation in previous MANNs.
method ARMIN uses hidden state for automatic memory addressing and a novel RNN cell for memory integration.
result ARMIN achieves lower computational overhead and similar performances compared to vanilla LSTM.
DAM with MRL improves relational reasoning in MANNs.
problem Limited performance of associative memory networks on complex relational reasoning tasks.
method Distributed Associative Memory architecture with Memory Refreshing Loss.
result Enhanced relation reasoning performance of MANNs on long temporal sequence data.
New memory allocation scheme improves image generation performance.
problem Improving episodic and semantic memory representation in neural networks.
method Developed a hierarchical latent variable model with differentiable, locally block allocated latent memory.
result Improved conditional likelihood values on various datasets.
A new memory system learns and generates like new data.
problem Training generative models on new data.
method Hierarchical conditional generative model with distributed memory.
result The memory system significantly improves generative models.
LEMN improves memory networks for learning from streaming data.
problem Efficiently processing large data streams in memory-augmented neural networks.
method LEMN uses a RNN-based retention agent to learn and replace less important memory entries based on their importance and historical context.
result LEMN achieves significant improvements over existing methods in learning from streaming data.
mGRN improves multivariate time series prediction by managing marginal and joint memories.
problem Extracting dependencies in multivariate sequential data with strong serial and cross-sectional dependencies.
method Developed a novel recurrent network architecture, Memory-Gated Recurrent Networks (mGRN), with gates for marginal and joint memories.
result mGRN consistently outperforms state-of-the-art architectures on various public datasets.
New model DWM mimics human memory, learning to retain, ignore or forget.
problem Lack of effective separation between episodic and working memory in neural networks.
method Designed Differentiable Working Memory (DWM) inspired by psychological studies.
result DWM learns psychology-inspired tasks faster and generalizes to longer sequences.
Hierarchical memory network tackles large-scale memory access with reduced computation and training complexity.
problem Efficiently accessing large memories in neural networks.
method Hybrid approach combining soft and hard attention mechanisms, using Maximum Inner Product Search (MIPS).
result Hierarchical memory network achieves faster and more trainable memory access than flat memory structures.
Model combines long-term and short-term memory using conceptors.
problem Transfer between long-term and short-term memory.
method Recurrent neural network with gated reservoir for short-term memory and conceptors for long-term memory.
result Standard operations on conceptors allow combining long-term memories and describing their effect on short-term memory.
AuGMEnT network struggles with long-term memory for hierarchical tasks.
problem Learning and memory in neural networks, especially hierarchical tasks.
method Introduced hybrid AuGMEnT with leaky and non-leaky memory units.
result Hybrid AuGMEnT solves hierarchical and distractor tasks.
New memory in neural networks mimics computer architectures.
problem Learning algorithms and complex tasks with neural networks.
method Introducing a new memory to store weights for a neural controller, similar to stored-program memory in computers.
result Neural Stored-program Memory enhances neural networks' adaptability and learning capabilities.
Metalearned neural memory improves learning across various tasks.
problem Improving neural network adaptability and memory function.
method Augmenting neural networks with a metalearned external memory mechanism.
result The model achieves strong performance on diverse learning problems.
CMT efficiently manages memory by inserting and querying memories in logarithmic time.
problem Managing large memory stores efficiently for quick access and updates.
method Designing a Contextual Memory Tree (CMT) that inserts and retrieves memories in logarithmic time.
result CMT improves classification algorithms and image-captioning tasks, demonstrating better computational efficiency.
Proposes new economic accelerators with memory in discrete time.
problem Describing economic processes with power-law memory and periodic sharp splashes.
method Uses capital stock adjustment principle and discrete maps derived from fractional-order differential equations.
result Discrete accelerators with memory accurately describe economic processes.
Generative memory helps AI learn new tasks without forgetting old ones.
problem Catastrophic forgetting in lifelong reinforcement learning.
method Generative memory model for task-agnostic latent space.
result Generative memory can improve lifelong reinforcement learning performance.
Enhanced Hopfield model boosts memory retrieval capacity.
problem Memory retrieval in modern Hopfield models with limited capacity.
method Introduces a learnable feature map transforming energy function into kernel space, minimizing separation loss for uniform memory distribution.
result Significant reduction in metastable states, enhancing memory capacity and retrieval accuracy.
LMNs enhance neural networks with memory to adapt online.
problem Online adaptation of neural networks for domain-relevant data.
method LMNs use label-based memory replacement and write to memory only for instances with non-zero loss.
result Significant accuracy gains on various tasks including word-modelling and few-shot learning.
New memory architecture improves meta-learning performance in small datasets.
problem Meta-learning challenges with small datasets and memory interference.
method Introduced a Feature-Label Memory Network (FLMN) that splits memory into feature and label components.
result FLMN outperforms MANN in supervised one-shot classification tasks.
Memory-constrained algorithms need superlinear memory for efficient convex optimization.
problem Efficiently minimizing convex functions with limited memory.
method Analyzing first-order algorithms with superlinear memory constraints.
result Superlinear memory is necessary for optimal performance in convex optimization.
The study reduces neural network training memory by up to 60.7x on CIFAR-10.
problem Reducing memory requirements for neural network training.
method Profiled memory usage, evaluated four techniques: sparsity, low precision, microbatching, gradient checkpointing.
result Appropriate combinations of techniques can reduce memory usage up to 60.7x on CIFAR-10.
A taxonomy classifies memory networks based on their memory organization.
problem Classifying and understanding the expressive power of different memory networks.
method Developed a taxonomy including RNN, LSTM, neural stack, and neural RAM, analyzing their differences and commonality.
result Showed the relative expressive power of memory networks and how they relate to specific tasks.
Two GPU memory management approaches reduce deep learning model memory usage.
problem Limited GPU memory in deep learning systems.
method Two orthogonal approaches exploiting iterative training algorithm to optimize memory usage.
result Up to 34.2% reduction in memory usage without communication overhead.
Generative diffusion models mimic biological memory networks, encoding associative dynamics in deep neural weights.
problem Understanding long-term memory mechanisms in neuroscience and AI.
method Interpreting generative diffusion models as energy-based models and comparing them to Hopfield networks.
result Generative diffusion models can encode associative dynamics of Hopfield networks in deep neural weights.
This paper successfully implements Neural Turing Machines for sequence learning tasks.
problem Implementing stable and high-performing Neural Turing Machines.
method Detailed description and successful implementation of NTMs, focusing on memory contents initialization.
result Memory contents initialized to small constant values converge 2 times faster.
A new model for sequential memory using temporal predictive coding.
problem Forming accurate memory of sequential stimuli in the brain.
method Proposes a novel PC-based model called temporal predictive coding (tPC).
result Shows that tPC models can accurately memorize and retrieve sequential inputs.
A new memory system handles non-stationary environments by self-sizing and retaining memories.
problem Non-stationary environments where memories arrive over time and must be distinguished from noise.
method A self-sizing continual associative memory that generalizes Hopfield's network, handling adaptation and novelty.
result The memory system grows to the intrinsic memory demand of the environment and retains memories without forgetting.
Paper questions RNN and LSTM's long-term memory and introduces a new definition.
problem Whether RNN and LSTM have long-term memory.
method Introduced a new definition of long-term memory and modified RNN and LSTM to test it.
result RNN and LSTM do not meet the new definition of long-term memory.
A new algorithm uses reservoir sampling to enhance a reinforcement learning agent's memory.
problem Efficiently maintaining and recalling past states for reinforcement learning.
method Reservoir sampling to maintain a fixed number of past states for an external memory.
result The method allows for efficient online computation of gradient estimates.
Estimates memory usage of reinforcement learning policies by mutual information.
problem Understanding the amount of memory used by reinforcement learning policies.
method Estimates mutual information between behavior histories and current actions.
result Provides a lower bound on minimal memory capacity of any agent.
A new method reduces memory usage for large models by three orders of magnitude.
problem Memory limitations in training large models.
method Extreme tensoring for adaptive preconditioning.
result Significant reduction in memory usage without performance degradation.
EMR learns to read and remember from streaming data for QA.
problem Scalable QA from streaming data without knowing questions.
method End-to-end deep network model (EMR) with RL agent for memory management.
result EMR achieves significant improvements over existing methods on synthetic and real-world datasets.
New method optimizes memory usage in neural networks, improving sequential learning.
problem Current memory models in neural networks waste memory and computation.
method Formulated an optimization problem to maximize information storage, introduced Cached Uniform Writing.
result Proved Cached Uniform Writing optimizes memory usage and outperforms other methods.
We study the long-term memory in diverse stock market indices and foreign exchange rates using the Detrended Fluctuation Analysis(DFA). For all daily and high-frequency market data studied, no significant long-term memory property is detected in the return series, while a strong long-term memory property is found in th…
Linear Memory Network separates memory and function in RNNs.
problem Complex transduction problems requiring memory and input-output exploitation.
method Conceptual separation between memory and function, using feedforward and autoencoder components.
result Efficient training and competitive performance on polyphonic music datasets.
New memory effect discovered in gravitational wave behavior.
problem Understanding gravitational wave behavior in spacetimes with angular momentum.
method Mathematical analysis of Minkowski spacetime and Kerr black holes.
result Angular momentum memory effect observed at future null infinity.
Tensor models decode human perception and memory using SPO triples.
problem Understanding implicit and explicit perception and memory in the brain.
method Tensor models with SPO triples, dual representations, and four layers.
result Semantic memory is crucial for explicit perception and declarative memories.
MEMGAN uses memory to improve anomaly detection by isolating abnormal data.
problem Weak guarantees for detecting anomalous data in classical algorithms.
method Memory-augmented Generative Adversarial Networks (MEMGAN) with a memory module.
result MEMGAN provides strong guarantees for anomaly detection with improved reconstruction.