Enhanced Hopfield model boosts memory retrieval capacity.
problem Memory retrieval in modern Hopfield models with limited capacity.
method Introduces a learnable feature map transforming energy function into kernel space, minimizing separation loss for uniform memory distribution.
result Significant reduction in metastable states, enhancing memory capacity and retrieval accuracy.
Generative memory model avoids vanishing gradients to robustly retrieve patterns.
problem Robust retrieval of stored patterns in the presence of interference and noise.
method Training a generative distributed memory without explicitly simulating attractor dynamics, using a likelihood-based Lyapunov function.
result The model converges to correct patterns upon iterative retrieval and achieves competitive performance as a memory model and a generative model.
Sparse Hopfield model improves memory retrieval with fewer connections.
problem Memory retrieval efficiency with fewer connections.
method Sparse extension of Hopfield model, derived from sparse entropic regularizer.
result Sparse Hopfield model achieves tighter error bounds and better performance.
A new associative memory uses Sinkhorn divergence for efficient pattern retrieval.
problem Efficiently retrieving patterns from large datasets of weighted point clouds.
method Derived retrieval dynamics as a SHK gradient flow, discretized for a deterministic algorithm.
result Proved basin invariance, geometric convergence, and robust recovery from perturbations.
Extends DAMs to Gaussian distributions for efficient pattern storage and retrieval.
problem Limited storage capacity and retrieval methods for non-vector pattern representations.
method Introduces a log-sum-exp energy function over Gaussian distributions, using optimal transport maps for retrieval dynamics.
result Proves exponential storage capacity and provides quantitative retrieval guarantees.
New cooperative dynamics enhances retrieval performance in neural networks.
problem Understanding emergent computational capabilities in disordered systems.
method Leveraging statistical mechanics, extended neural network architecture for hetero-associative memory.
result Layers trained with less informative datasets develop retrieval regions of the same amplitude, leading to optimal performance.
New neural networks model complex phenomena with fewer parameters.
problem Challenges in studying higher-order interactions in neural networks.
method Introducing curved neural networks using the maximum entropy principle.
result Curved neural networks accelerate memory retrieval and exhibit explosive phase transitions.
Efficient sparse modern Hopfield models are introduced for memory retrieval and learning tasks.
problem Efficient modern Hopfield models for memory retrieval and learning tasks.
method Nonparametric interpretation of Hopfield models as a regression problem.
result Sparse-structured modern Hopfield models with sub-quadratic complexity.
Linear memory stores associations up to a logarithmic scale, but listwise retrieval can handle a quadratic scale.
problem How many key-value associations can a linear memory store?
method Analyzed linear memory models for top-1 and listwise retrieval, proving phase transitions and developing asymptotic theories.
result Linear memory has a logarithmic capacity for top-1 retrieval and a quadratic capacity for listwise retrieval.
DMs emerge from DenseAMs, transitioning from memorization to generalization.
problem Hindered memory retrieval in DenseAMs due to spurious states.
method Examined diffusion models through the lens of DenseAMs, focusing on their generative process.
result Identified a critical phase in DMs transitioning from memorization to generalization.
System optimizes retrieval for personal assistants using reinforcement learning.
problem Needing a neural retrieval-based Q&A system for user memory.
method Direct optimization of F1-score using reinforcement learning.
result Improved retrieval performance on test sets.
Proposes a plastic neural memory model for better anomaly detection.
problem Static attention mechanisms limit NMNs in anomaly detection.
method Introduces dynamic connection weights for improved knowledge retrieval.
result Outperforms state-of-the-art in three medical anomaly detection tasks.
Meta-learning approach for fast and compressive energy-based memory models.
problem Learning associative memory with fast and compressive models for complex data.
method Meta-learning approach to energy-based memory models (EBMM) using arbitrary neural architectures.
result Demonstrated associative retrieval outperforming existing systems in reconstruction error and compression rate.
New method retrieves most interfered samples for continual learning.
problem Challenges in continual learning with online data streams.
method Controlled sampling of most interfered samples for replay.
result Consistent gains in performance and reduced forgetting.
Stochastic attention learns to retrieve and generate from memory without training.
problem Learning to retrieve and generate from memory efficiently.
method Langevin dynamics on modern Hopfield energy for stochastic attention.
result Stochastic attention can retrieve and generate from memory without training, matching gold standards.
Paper shows how LSTM can remember long sequences by attending to persisted information.
problem LSTMs struggle with long sequences due to fading information and bias towards recent data.
method The paper introduces a mechanism that allows LSTMs to attend to information in memory based on how long it was persisted by the gating mechanism.
result The method improves LSTM's ability to process long sequences by retrieving information proportionally to its persistence in memory.
LLMs can be tricked into recalling facts based on context clues.
problem Manipulation of LLMs' factual recall through context changes.
method Mathematical exploration of transformers' associative memory properties.
result Transformers use self-attention and value matrix for associative memory.
HabitatAgent offers a multi-agent system for transparent housing consultation.
problem Opaque reasoning and brittle multi-constraint handling in housing recommendation systems.
method HabitatAgent is a multi-agent architecture with specialized roles for memory, retrieval, generation, and validation.
result HabitatAgent achieves 95% accuracy in real user consultation scenarios, significantly outperforming a strong baseline.
GENRE retrieves entities autoregressively, improving efficiency and accuracy.
problem Retrieving entities from queries efficiently and accurately.
method Autoregressive generation of entity names, reducing memory footprint and improving context encoding.
result Significantly improved performance on entity disambiguation, linking, and retrieval tasks.
We analyze computational limits of modern Hopfield models based on pattern norms.
problem Understanding the efficiency of modern Hopfield models from a fine-grained complexity perspective.
method Fine-grained complexity analysis and upper bound criterion for pattern norms.
result Below a specific norm threshold, efficient variants of modern Hopfield models exist.
Efficiently stores and retrieves past states for faster learning in reinforcement learning.
problem Data inefficiency and memory limitations in reinforcement learning.
method Dynamic online k-means for state clustering and prioritization.
result Dynamic online k-means improves performance with smaller memory sizes.
PCA reduces language model embeddings to improve speed and memory efficiency.
problem High-dimensional embeddings in RAG models cause scalability issues.
method Used Principal Component Analysis (PCA) to compress embeddings.
result Reducing from 3,072 to 110 dimensions provides significant speedup and memory reduction.
A new framework for efficient sequence maps using Bayesian filtering and covariance.
problem Designing efficient recurrent sequence maps from explicit memory assumptions.
method Design-model framework, exact Bayesian filtering, query-dependent readout, linear-Gaussian instantiation.
result Improved robustness and retrieval performance across various benchmarks.
V-HMN integrates memory mechanisms for improved image recognition.
problem Limited interpretability and high data requirements of existing vision backbones.
method Brain-inspired hierarchical memory modules with iterative refinement.
result V-HMN achieves strong performance on image classification benchmarks.
Study on neural networks with non-normal interactions reveals unique spectral properties.
problem Understanding episodic memory encoding in the brain.
method Developed a neural network model with non-Hermitian couplings and applied random matrix theory.
result Spectral density of the model is non-uniform and can transition to chaos, providing computational benefits.
Deep learning typically requires training a very capable architecture using large datasets. However, many important learning problems demand an ability to draw valid inferences from small size datasets, and such problems pose a particular challenge for deep learning. In this regard, various researches on "meta-learning…
We design and study a Contextual Memory Tree (CMT), a learning memory controller that inserts new memories into an experience store of unbounded size. It is designed to efficiently query for memories from that store, supporting logarithmic time insertion and retrieval operations. Hence CMT can be integrated into existi…
RNNs struggle with in-context retrieval, while Transformers excel.
problem In-context retrieval capability of RNNs.
method Theoretical analysis and experimental techniques (CoT, RAG, Transformer layer).
result Enhancing RNNs with techniques improves their in-context retrieval capability, closing the representation gap with Transformers.
A new model for sequential memory using temporal predictive coding.
problem Forming accurate memory of sequential stimuli in the brain.
method Proposes a novel PC-based model called temporal predictive coding (tPC).
result Shows that tPC models can accurately memorize and retrieve sequential inputs.
Enhances hashing for fast retrieval with correlated bits.
problem Fast retrieval and small memory footprint for large-scale information retrieval.
method Employing Boltzmann machine distribution as variational posterior to model correlations among hash code bits.
result Significant performance gains achieved by effectively modeling correlations among hash code bits.
Research on interference has provided evidence that the formation of dependencies between non-adjacent words relies on a cue-based retrieval mechanism. Two different models can account for one of the main predictions of interference, i.e., a slowdown at a retrieval site, when several items share a feature associated wi…
A bridge between continuous signals and discrete Ising spins for associative memory.
problem Associative memory in continuous-signal-driven Ising spin systems.
method Multilayer Ising framework with PCA whitening and SimHash projection, coupled to pseudo-inverse memory couplings.
result Finite-size scaling of operational storage capacity with αc(N)=αc(∞)−cN−1/2, approaching αc(∞)≈0.50. A new Hopfield model reduces inefficiency in large transformer models.
problem Inefficiency in training large transformer-based models.
method Introduces an Outlier-Efficient Hopfield Model (OutEffHop) to improve model performance.
result Achieves up to 26% reduction in model output norms across four models.
A federated model learns shared archetypes from heterogeneous clients in continual learning.
problem Federated learning struggles with client heterogeneity and streaming distribution shifts.
method Clients encode their data as low-rank Hebbian operators, which are sent to a central server for aggregation and factorization into global archetypes.
result Improved global archetype reconstruction and associative retrieval in heterogeneous clients, drift, and novelty settings.
Enhances sequence memory capacity in neural networks.
problem Limited sequence capacity in Hopfield-like neural networks.
method Introducing a nonlinear interaction term and a generalized pseudoinverse rule.
result Significantly increased sequence capacity with novel scaling laws.
Model user preferences for conversational LLMs using weak rewards.
problem Lack of persistent user models in conversational LLMs leading to repeated user restatements.
method Vector-Adapted Retrieval Scoring (VARS) framework that updates user vectors online from weak scalar rewards.
result Full VARS agent achieves strongest overall performance, matches strong Reflection baseline in task success, and reduces user effort.
Enhanced image recognition models learn from human-like memory and shape biases.
problem Improving robustness of image recognition models against various perturbations.
method Integrating human-like episodic memory and shape bias features into image recognition models.
result Combining human-like features improves robustness against both adversarial and natural perturbations.
A modern Hopfield network improves deep learning with better memory and attention.
problem Improving memory and attention mechanisms in deep learning.
method Introducing a new Hopfield network with continuous states and a novel update rule.
result The new Hopfield network can store and retrieve patterns efficiently with low errors.
New findings show tool-augmented models can recall unlimited facts, outperforming purely memorized models.
problem Limitations of purely memorized models in recalling large amounts of factual information.
method Demonstrated the benefits of in-tool learning (external retrieval) over in-weight learning (memorization) for factual recall.
result Proved that tool-use enables unbounded factual recall via a simple and efficient circuit construction.
Deep neural networks can store and recall data efficiently.
problem Identifying computational mechanisms for memorization and retrieval of data.
method Training overparameterized autoencoders and sequence encoders using standard optimization methods.
result Overparameterized autoencoders and sequence encoders store and recall data efficiently as attractors.
STanHop predicts multivariate time series with memory-enhanced capabilities.
problem Predicting multivariate time series with memory-enhanced capabilities.
method Sparse Tandem Hopfield Network (STanHop) with two external memory modules.
result STanHop outperforms dense Hopfield models in memory retrieval error.
Sharp limits found for storing and retrieving input-output associations in linear associative memories.
problem Understanding the fundamental limits of storing and retrieving input-output associations in neural networks.
method Study of a minimal linear associative memory model, introducing a decoupled model and using statistical physics to characterize storage capacity.
result Linear associative memory can store up to 1/2 log(p) associations, providing a sharp statistical-physics characterization.
New analysis tightens memory capacity of Hopfield models using spherical codes.
problem Optimizing memory capacity in modern Hopfield models and Kernelized Hopfield Models.
method Connecting Hopfield models to spherical codes in information theory, establishing an optimal capacity bound and a sub-linear algorithm.
result First tight and optimal asymptotic memory capacity for modern Hopfield models, matching known lower bounds.
A new method learns from multi-modal sequences with external memory.
problem Learning new modes in a dynamic environment without prior knowledge.
method Maintains a neural episodic memory with a Dirichlet Process prior to store mode descriptors and transfers knowledge through retrieval.
result Performs continual learning favorably compared to mainstream approaches.
SAGE improves memory efficiency by selectively adding, merging, or ignoring new facts.
problem Efficiently managing new facts in agentic LLMs to avoid costly write-time reasoning.
method SAGE uses a von Mises-Fisher-based density estimator to score and route candidate facts.
result SAGE achieves the best average token-F1 on LoCoMo and reduces add-phase API cost by 3.4x on GPT-4o-mini.
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of mo…
Hydra boosts efficiency for long-context reasoning in resource-constrained settings.
problem Quadratic complexity of transformers limits long-context reasoning in resource-constrained systems.
method Hydra uses a modular architecture with adaptive routing between sparse global attention, mixture-of-experts, and dual memories.
result Hydra achieves significant throughput and accuracy improvements for long-context reasoning.
The Unlearning algorithm improves neural network performance in memory tasks.
problem Improving neural network performance in memory tasks.
method Simplified Unlearning algorithm, structured training data, and new regularization technique for Boltzmann Machines.
result The Unlearning rule outperforms traditional Boltzmann-Machine learning in learning hidden probability distributions.