Enhanced Hopfield model boosts memory retrieval capacity.
problem Memory retrieval in modern Hopfield models with limited capacity.
method Introduces a learnable feature map transforming energy function into kernel space, minimizing separation loss for uniform memory distribution.
result Significant reduction in metastable states, enhancing memory capacity and retrieval accuracy.
Sparse Hopfield model improves memory retrieval with fewer connections.
problem Memory retrieval efficiency with fewer connections.
method Sparse extension of Hopfield model, derived from sparse entropic regularizer.
result Sparse Hopfield model achieves tighter error bounds and better performance.
A new associative memory uses Sinkhorn divergence for efficient pattern retrieval.
problem Efficiently retrieving patterns from large datasets of weighted point clouds.
method Derived retrieval dynamics as a SHK gradient flow, discretized for a deterministic algorithm.
result Proved basin invariance, geometric convergence, and robust recovery from perturbations.
Extends DAMs to Gaussian distributions for efficient pattern storage and retrieval.
problem Limited storage capacity and retrieval methods for non-vector pattern representations.
method Introduces a log-sum-exp energy function over Gaussian distributions, using optimal transport maps for retrieval dynamics.
result Proves exponential storage capacity and provides quantitative retrieval guarantees.
New cooperative dynamics enhances retrieval performance in neural networks.
problem Understanding emergent computational capabilities in disordered systems.
method Leveraging statistical mechanics, extended neural network architecture for hetero-associative memory.
result Layers trained with less informative datasets develop retrieval regions of the same amplitude, leading to optimal performance.
New neural networks model complex phenomena with fewer parameters.
problem Challenges in studying higher-order interactions in neural networks.
method Introducing curved neural networks using the maximum entropy principle.
result Curved neural networks accelerate memory retrieval and exhibit explosive phase transitions.
Efficient sparse modern Hopfield models are introduced for memory retrieval and learning tasks.
problem Efficient modern Hopfield models for memory retrieval and learning tasks.
method Nonparametric interpretation of Hopfield models as a regression problem.
result Sparse-structured modern Hopfield models with sub-quadratic complexity.
Linear memory stores associations up to a logarithmic scale, but listwise retrieval can handle a quadratic scale.
problem How many key-value associations can a linear memory store?
method Analyzed linear memory models for top-1 and listwise retrieval, proving phase transitions and developing asymptotic theories.
result Linear memory has a logarithmic capacity for top-1 retrieval and a quadratic capacity for listwise retrieval.
A central challenge faced by memory systems is the robust retrieval of a stored pattern in the presence of interference due to other stored patterns and noise. A theoretically well-founded solution to robust retrieval is given by attractor dynamics, which iteratively clean up patterns during recall. However, incorporat…
DMs emerge from DenseAMs, transitioning from memorization to generalization.
problem Hindered memory retrieval in DenseAMs due to spurious states.
method Examined diffusion models through the lens of DenseAMs, focusing on their generative process.
result Identified a critical phase in DMs transitioning from memorization to generalization.
Recent advances in spoken language technologies and the introduction of many customer facing products, have given rise to a wide customer reliance on smart personal assistants for many of their daily tasks. In this paper, we present a system to reduce users' cognitive load by extending personal assistants with long-ter…
In the domain of machine learning, Neural Memory Networks (NMNs) have recently achieved impressive results in a variety of application areas including visual question answering, trajectory prediction, object tracking, and language modelling. However, we observe that the attention based knowledge retrieval mechanisms us…
We study the problem of learning associative memory -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version. Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by…
Stochastic attention learns to retrieve and generate from memory without training.
problem Learning to retrieve and generate from memory efficiently.
method Langevin dynamics on modern Hopfield energy for stochastic attention.
result Stochastic attention can retrieve and generate from memory without training, matching gold standards.
LLMs can be tricked into recalling facts based on context clues.
problem Manipulation of LLMs' factual recall through context changes.
method Mathematical exploration of transformers' associative memory properties.
result Transformers use self-attention and value matrix for associative memory.
HabitatAgent offers a multi-agent system for transparent housing consultation.
problem Opaque reasoning and brittle multi-constraint handling in housing recommendation systems.
method HabitatAgent is a multi-agent architecture with specialized roles for memory, retrieval, generation, and validation.
result HabitatAgent achieves 95% accuracy in real user consultation scenarios, significantly outperforming a strong baseline.
Continual learning, the setting where a learning agent is faced with a never ending stream of data, continues to be a great challenge for modern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Me…
GENRE retrieves entities autoregressively, improving efficiency and accuracy.
problem Retrieving entities from queries efficiently and accurately.
method Autoregressive generation of entity names, reducing memory footprint and improving context encoding.
result Significantly improved performance on entity disambiguation, linking, and retrieval tasks.
We analyze computational limits of modern Hopfield models based on pattern norms.
problem Understanding the efficiency of modern Hopfield models from a fine-grained complexity perspective.
method Fine-grained complexity analysis and upper bound criterion for pattern norms.
result Below a specific norm threshold, efficient variants of modern Hopfield models exist.
PCA reduces language model embeddings to improve speed and memory efficiency.
problem High-dimensional embeddings in RAG models cause scalability issues.
method Used Principal Component Analysis (PCA) to compress embeddings.
result Reducing from 3,072 to 110 dimensions provides significant speedup and memory reduction.
A new framework for efficient sequence maps using Bayesian filtering and covariance.
problem Designing efficient recurrent sequence maps from explicit memory assumptions.
method Design-model framework, exact Bayesian filtering, query-dependent readout, linear-Gaussian instantiation.
result Improved robustness and retrieval performance across various benchmarks.
V-HMN integrates memory mechanisms for improved image recognition.
problem Limited interpretability and high data requirements of existing vision backbones.
method Brain-inspired hierarchical memory modules with iterative refinement.
result V-HMN achieves strong performance on image classification benchmarks.
Study on neural networks with non-normal interactions reveals unique spectral properties.
problem Understanding episodic memory encoding in the brain.
method Developed a neural network model with non-Hermitian couplings and applied random matrix theory.
result Spectral density of the model is non-uniform and can transition to chaos, providing computational benefits.
Deep learning typically requires training a very capable architecture using large datasets. However, many important learning problems demand an ability to draw valid inferences from small size datasets, and such problems pose a particular challenge for deep learning. In this regard, various researches on "meta-learning…
We design and study a Contextual Memory Tree (CMT), a learning memory controller that inserts new memories into an experience store of unbounded size. It is designed to efficiently query for memories from that store, supporting logarithmic time insertion and retrieval operations. Hence CMT can be integrated into existi…
RNNs struggle with in-context retrieval, while Transformers excel.
problem In-context retrieval capability of RNNs.
method Theoretical analysis and experimental techniques (CoT, RAG, Transformer layer).
result Enhancing RNNs with techniques improves their in-context retrieval capability, closing the representation gap with Transformers.
Language Models (LMs) are important components in several Natural Language Processing systems. Recurrent Neural Network LMs composed of LSTM units, especially those augmented with an external memory, have achieved state-of-the-art results. However, these models still struggle to process long sequences which are more li…
A new model for sequential memory using temporal predictive coding.
problem Forming accurate memory of sequential stimuli in the brain.
method Proposes a novel PC-based model called temporal predictive coding (tPC).
result Shows that tPC models can accurately memorize and retrieve sequential inputs.
Enhances hashing for fast retrieval with correlated bits.
problem Fast retrieval and small memory footprint for large-scale information retrieval.
method Employing Boltzmann machine distribution as variational posterior to model correlations among hash code bits.
result Significant performance gains achieved by effectively modeling correlations among hash code bits.
Research on interference has provided evidence that the formation of dependencies between non-adjacent words relies on a cue-based retrieval mechanism. Two different models can account for one of the main predictions of interference, i.e., a slowdown at a retrieval site, when several items share a feature associated wi…
A bridge between continuous signals and discrete Ising spins for associative memory.
problem Associative memory in continuous-signal-driven Ising spin systems.
method Multilayer Ising framework with PCA whitening and SimHash projection, coupled to pseudo-inverse memory couplings.
result Finite-size scaling of operational storage capacity with αc(N)=αc(∞)−cN−1/2, approaching αc(∞)≈0.50. A new Hopfield model reduces inefficiency in large transformer models.
problem Inefficiency in training large transformer-based models.
method Introduces an Outlier-Efficient Hopfield Model (OutEffHop) to improve model performance.
result Achieves up to 26% reduction in model output norms across four models.
We investigate the robustness properties of image recognition models equipped with two features inspired by human vision, an explicit episodic memory and a shape bias, at the ImageNet scale. As reported in previous work, we show that an explicit episodic memory improves the robustness of image recognition models agains…
Enhances sequence memory capacity in neural networks.
problem Limited sequence capacity in Hopfield-like neural networks.
method Introducing a nonlinear interaction term and a generalized pseudoinverse rule.
result Significantly increased sequence capacity with novel scaling laws.
Model user preferences for conversational LLMs using weak rewards.
problem Lack of persistent user models in conversational LLMs leading to repeated user restatements.
method Vector-Adapted Retrieval Scoring (VARS) framework that updates user vectors online from weak scalar rewards.
result Full VARS agent achieves strongest overall performance, matches strong Reflection baseline in task success, and reduces user effort.
A federated model learns shared archetypes from heterogeneous clients in continual learning.
problem Federated learning struggles with client heterogeneity and streaming distribution shifts.
method Clients encode their data as low-rank Hebbian operators, which are sent to a central server for aggregation and factorization into global archetypes.
result Improved global archetype reconstruction and associative retrieval in heterogeneous clients, drift, and novelty settings.
New findings show tool-augmented models can recall unlimited facts, outperforming purely memorized models.
problem Limitations of purely memorized models in recalling large amounts of factual information.
method Demonstrated the benefits of in-tool learning (external retrieval) over in-weight learning (memorization) for factual recall.
result Proved that tool-use enables unbounded factual recall via a simple and efficient circuit construction.
A modern Hopfield network improves deep learning with better memory and attention.
problem Improving memory and attention mechanisms in deep learning.
method Introducing a new Hopfield network with continuous states and a novel update rule.
result The new Hopfield network can store and retrieve patterns efficiently with low errors.
STanHop predicts multivariate time series with memory-enhanced capabilities.
problem Predicting multivariate time series with memory-enhanced capabilities.
method Sparse Tandem Hopfield Network (STanHop) with two external memory modules.
result STanHop outperforms dense Hopfield models in memory retrieval error.
Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued…
Sharp limits found for storing and retrieving input-output associations in linear associative memories.
problem Understanding the fundamental limits of storing and retrieving input-output associations in neural networks.
method Study of a minimal linear associative memory model, introducing a decoupled model and using statistical physics to characterize storage capacity.
result Linear associative memory can store up to 1/2 log(p) associations, providing a sharp statistical-physics characterization.
Recently, neuro-inspired episodic control (EC) methods have been developed to overcome the data-inefficiency of standard deep reinforcement learning approaches. Using non-/semi-parametric models to estimate the value function, they learn rapidly, retrieving cached values from similar past states. In realistic scenarios…
New analysis tightens memory capacity of Hopfield models using spherical codes.
problem Optimizing memory capacity in modern Hopfield models and Kernelized Hopfield Models.
method Connecting Hopfield models to spherical codes in information theory, establishing an optimal capacity bound and a sub-linear algorithm.
result First tight and optimal asymptotic memory capacity for modern Hopfield models, matching known lower bounds.
A new method learns from multi-modal sequences with external memory.
problem Learning new modes in a dynamic environment without prior knowledge.
method Maintains a neural episodic memory with a Dirichlet Process prior to store mode descriptors and transfers knowledge through retrieval.
result Performs continual learning favorably compared to mainstream approaches.
SAGE improves memory efficiency by selectively adding, merging, or ignoring new facts.
problem Efficiently managing new facts in agentic LLMs to avoid costly write-time reasoning.
method SAGE uses a von Mises-Fisher-based density estimator to score and route candidate facts.
result SAGE achieves the best average token-F1 on LoCoMo and reduces add-phase API cost by 3.4x on GPT-4o-mini.
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of mo…
Hydra boosts efficiency for long-context reasoning in resource-constrained settings.
problem Quadratic complexity of transformers limits long-context reasoning in resource-constrained systems.
method Hydra uses a modular architecture with adaptive routing between sparse global attention, mixture-of-experts, and dual memories.
result Hydra achieves significant throughput and accuracy improvements for long-context reasoning.
The Unlearning algorithm improves neural network performance in memory tasks.
problem Improving neural network performance in memory tasks.
method Simplified Unlearning algorithm, structured training data, and new regularization technique for Boltzmann Machines.
result The Unlearning rule outperforms traditional Boltzmann-Machine learning in learning hidden probability distributions.