arXiv research

New analysis tightens memory capacity of Hopfield models using spherical codes.

The study evaluates memory and capacity of graph embedding methods.

Generalizes memory and forecasting capacities for nonlinear recurrent networks with dependent inputs.

Memory capacity of DAM scales exponentially with feature separation, unaffected by correlations.

Study characterizes memory capacity of quantum reservoirs using transmon qubits.

Enhances sequence memory capacity in neural networks.

Enhanced Hopfield model boosts memory retrieval capacity.

Nonlinear RNNs' memory capacity varies widely, making it impractical.

The memory capacity of linear echo state networks is accurately calculated using new numerical methods.

A new memory system handles non-stationary environments by self-sizing and retaining memories.

ADHD is being recognized as a diagnosis which persists into adulthood impacting economic, occupational, and educational outcomes. There is an increased need to accurately diagnose and recommend interventions for this population. One consideration is the development and implementation of reliable and valid outcome measu…

Derives an empirical capacity model for self-attention neural networks.

In this paper, the echo state network (ESN) memory capacity, which represents the amount of input data an ESN can store, is analyzed for a new type of deep ESNs. In particular, two deep ESN architectures are studied. First, a parallel deep ESN is proposed in which multiple reservoirs are connected in parallel allowing …

Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain memory differentiability, or by discounting the RNN representation learning towards encoding shorte…

Sparse neural encoding can store more memories as targets become sparser.

Sharp limits found for storing and retrieving input-output associations in linear associative memories.

gLSTM improves graph neural networks by increasing storage capacity to prevent over-squashing.

This paper describes an improvement in Deep Q-learning called Reverse Experience Replay (also RER) that solves the problem of sparse rewards and helps to deal with reward maximizing tasks by sampling transitions successively in reverse order. On tasks with enough experience for training and enough Experience Replay mem…

DMs emerge from DenseAMs, transitioning from memorization to generalization.

Sparse codes improve optimal control tasks with correlated inputs.

It has been known for a long time that the classical spherical perceptrons can be used as storage memories. Seminal work of Gardner, \cite{Gar88}, started an analytical study of perceptrons storage abilities. Many of the Gardner's predictions obtained through statistical mechanics tools have been rigorously justified. …

New neural networks model complex phenomena with fewer parameters.

The brain optimizes memory by forgetting what's predictable, improving generalization.

Extends DAMs to Gaussian distributions for efficient pattern storage and retrieval.

Study capacity constraints in continual learning with a simple model.

Transformers can store facts efficiently using associative memories.

The study revisits Hopfield's associative memory model and calculates its capacity for two specific pattern basins.

Linear memory stores associations up to a logarithmic scale, but listwise retrieval can handle a quadratic scale.

We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva's sparse distributed memory, it has a robust distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian updat…

Service-induced congestion in memory-constrained LLM serving

A variety of large-scale machine learning problems can be cast as instances of constrained submodular maximization. Existing approaches for distributed submodular maximization have a critical drawback: The capacity - number of instances that can fit in memory - must grow with the data set size. In practice, while one c…

Typical neural networks with external memory do not effectively separate capacity for episodic and working memory as is required for reasoning in humans. Applying knowledge gained from psychological studies, we designed a new model called Differentiable Working Memory (DWM) in order to specifically emulate human workin…

Stable Hadamard Memory improves reinforcement learning by efficiently managing memory.

We propose a new method to study the internal memory used by reinforcement learning policies. We estimate the amount of relevant past information by estimating mutual information between behavior histories and the current action of an agent. We perform this estimation in the passive setting, that is, we do not interven…

New framework captures long-term decision dependence in online learning.

An ideal cognitively-inspired memory system would compress and organize incoming items. The Kanerva Machine (Wu et al, 2018) is a Bayesian model that naturally implements online memory compression. However, the organization of the Kanerva Machine is limited by its use of a single Gaussian random matrix for storage. Her…

Following the recent work on capacity allocation, we formulate the conjecture that the shattering problem in deep neural networks can only be avoided if the capacity propagation through layers has a non-degenerate continuous limit when the number of layers tends to infinity. This allows us to study a number of commonly…

Recurrent neural networks (RNNs) have drawn interest from machine learning researchers because of their effectiveness at preserving past inputs for time-varying data processing tasks. To understand the success and limitations of RNNs, it is critical that we advance our analysis of their fundamental memory properties. W…

Muon spectral optimizer outperforms SGD in associative memory tasks.

Dense Associative Memories outperform classical networks in robustness and signal processing.

The paper analyzes the performance of delay-based reservoir computing using eigenvalue analysis.

LLMs can be tricked into recalling facts based on context clues.

PipeDream-2BW accelerates large model training by 20x with minimal memory usage.

Global Memory Augmentation (GMAT) improves Transformer performance on long documents.

We propose a generic algorithmic building block to accelerate training of machine learning models on heterogeneous compute systems. Our scheme allows to efficiently employ compute accelerators such as GPUs and FPGAs for the training of large-scale machine learning models, when the training data exceeds their memory cap…

The computation of convolution layers in deep neural networks typically rely on high performance routines that trade space for time by using additional memory (either for packing purposes or required as part of the algorithm) to improve performance. The problems with such an approach are two-fold. First, these routines…

STanHop predicts multivariate time series with memory-enhanced capabilities.

This study investigates self-organizing dynamics in a stochastic exponential DAM model using Temporal Complexity.