Transformers can store facts efficiently using associative memories.
problem Understanding how transformers store and recall factual information.
method Proved linear scaling of storage capacities for linear and MLP associative memories, introduced a synthetic task, and analyzed gradient flow.
result Shallow transformers can achieve near optimal storage capacity for factual recall tasks using associative memories.
LLMs can be tricked into recalling facts based on context clues.
problem Manipulation of LLMs' factual recall through context changes.
method Mathematical exploration of transformers' associative memory properties.
result Transformers use self-attention and value matrix for associative memory.
This paper designs neural associative memories that can correct many adversarial errors.
problem Designing neural associative memories that can correct many adversarial errors.
method Mapping the learning phase and recall phase to dictionary learning with a square dictionary and iterative error correction in an expander code.
result The designed associative memories can store datasets with exp ( n ) \exp(n) exp ( n ) vectors and tolerate Ω ( n m p o l y l o g n ) Ω(\frac{n}{{
m polylog} n}) Ω ( m p o l y l o g n n ) adversarial errors. GeNet classifies metagenomic sequences with less memory and comparable recall to state-of-the-art methods.
problem Classifying metagenomic sequences from raw DNA sequences.
method Exploits hierarchical structure between labels for training, using deep representations.
result GeNet achieves competitive precision and good recall with less memory requirements.
Transformers recall from long distributions with statistical guarantees.
problem Designing Transformers that can recall from arbitrarily long, distributional contexts.
method Recast associative memory as probability measures, decomposing the task into recall and prediction.
result A shallow measure-theoretic Transformer learns the recall-and-predict map under spectral assumptions.
A new algorithm uses reservoir sampling to enhance a reinforcement learning agent's memory.
problem Efficiently maintaining and recalling past states for reinforcement learning.
method Reservoir sampling to maintain a fixed number of past states for an external memory.
result The method allows for efficient online computation of gradient estimates.
APL learns from surprising observations to quickly generalize from few examples.
problem Quickly generalize from limited data for intelligent systems.
method Approximates probability distributions by remembering surprising observations in an external memory module.
result APL performs as well as state-of-the-art baselines on few-shot classification benchmarks.
A new method for generating replay samples on the fly, optimizing for not forgetting.
problem Addressing the issue of forgetting in neural networks.
method Generates auxiliary samples on the fly using the model's implicit memory, specialized to each real training batch.
result Optimizing for not forgetting leads to more efficient and scalable generation of specialized samples.
New findings show tool-augmented models can recall unlimited facts, outperforming purely memorized models.
problem Limitations of purely memorized models in recalling large amounts of factual information.
method Demonstrated the benefits of in-tool learning (external retrieval) over in-weight learning (memorization) for factual recall.
result Proved that tool-use enables unbounded factual recall via a simple and efficient circuit construction.
Generative memory helps AI learn new tasks without forgetting old ones.
problem Catastrophic forgetting in lifelong reinforcement learning.
method Generative memory model for task-agnostic latent space.
result Generative memory can improve lifelong reinforcement learning performance.
Estimates temporal relationships in time-series data with latent variables.
problem Learning causal relationships between time series with latent memory.
method Developed an estimator for latent Markov processes with variable lags.
result Parameters can be learned consistently under genericity assumption.
Paper classifies brain signals using eigenvalues for 2D and 3D educational content questions.
problem Classifying brain signals for 2D and 3D educational content questions.
method Eigenvalues of covariance matrix used as features; KNN and SVM classifiers applied.
result No significant difference in learning, memory retention, and recall between 2D and 3D educational content.
Generative memory model avoids vanishing gradients to robustly retrieve patterns.
problem Robust retrieval of stored patterns in the presence of interference and noise.
method Training a generative distributed memory without explicitly simulating attractor dynamics, using a likelihood-based Lyapunov function.
result The model converges to correct patterns upon iterative retrieval and achieves competitive performance as a memory model and a generative model.
Study compares AI and static analysis for detecting buffer overflows.
problem Detecting buffer overflows in code.
method Developed s-bAbI to generate code samples, compared AI with static analysis tools.
result AI system requires extensive training data to match static analysis precision and recall.
Generative diffusion models mimic biological memory networks, encoding associative dynamics in deep neural weights.
problem Understanding long-term memory mechanisms in neuroscience and AI.
method Interpreting generative diffusion models as energy-based models and comparing them to Hopfield networks.
result Generative diffusion models can encode associative dynamics of Hopfield networks in deep neural weights.
Meta-learning agents forget tasks; this work creates a memory system to help them remember.
problem Meta-learning agents forget previously learned tasks when new ones appear.
method Developed a meta-learning architecture that combines LSTM and a differentiable neural episodic memory.
result Agents with the episodic LSTM architecture can solve tasks more effectively when they recur.
Enhances sequence memory capacity in neural networks.
problem Limited sequence capacity in Hopfield-like neural networks.
method Introducing a nonlinear interaction term and a generalized pseudoinverse rule.
result Significantly increased sequence capacity with novel scaling laws.
Deep neural networks can store and recall data efficiently.
problem Identifying computational mechanisms for memorization and retrieval of data.
method Training overparameterized autoencoders and sequence encoders using standard optimization methods.
result Overparameterized autoencoders and sequence encoders store and recall data efficiently as attractors.
Sharp limits found for storing and retrieving input-output associations in linear associative memories.
problem Understanding the fundamental limits of storing and retrieving input-output associations in neural networks.
method Study of a minimal linear associative memory model, introducing a decoupled model and using statistical physics to characterize storage capacity.
result Linear associative memory can store up to 1/2 log(p) associations, providing a sharp statistical-physics characterization.
Smooth calibration improves forecast reliability even with leaked information.
problem Improving forecast reliability with leaked information.
method Combining nearby forecasts to ensure smooth calibration, which can be guaranteed by deterministic procedures.
result Smooth calibration can be guaranteed by deterministic procedures even with leaked forecasts, and it yields uncoupled finite-memory dynamics in games.
Sublinear memory sketch finds nearest neighbors in streaming data.
problem Finding nearest neighbors in large datasets with limited memory.
method Combines LSH, online kernel density estimation, and compressed sensing to achieve sublinear memory.
result Achieves sublinear memory performance on stable queries, reporting nearest neighbors efficiently.
This paper uses deep learning to improve memory prefetching.
problem Improving memory performance in software optimizations.
method Relating prefetching strategies to n-gram models and using recurrent neural networks.
result Neural networks consistently outperform traditional prefetching strategies in benchmark tests.
Unified framework for sequence models using test-time regression.
problem Designing efficient sequence models with associative memory.
method Formalizing associative recall as regression over input tokens, deriving various sequence models.
result Clarifies the effectiveness of query-key normalization in softmax attention and offers new generalizations.
Extends RCNNs to handle hypercomplex-valued data.
problem Processing high-dimensional data.
method Develops mathematical background and conditions for hypercomplex-valued RCNNs.
result Hypercomplex-valued RCNNs can always settle at an equilibrium.
Face recall is a basic human cognitive process performed routinely, e.g., when meeting someone and determining if we have met that person before. Assisting a subject during face recall by suggesting candidate faces can be challenging. One of the reasons is that the search space - the face space - is quite large and lac…
A new framework for efficient sequence maps using Bayesian filtering and covariance.
problem Designing efficient recurrent sequence maps from explicit memory assumptions.
method Design-model framework, exact Bayesian filtering, query-dependent readout, linear-Gaussian instantiation.
result Improved robustness and retrieval performance across various benchmarks.
CLA improves investment accuracy by leveraging past knowledge.
problem Improving investment decisions through past knowledge integration.
method CLA uses an explicit memory structure and FFNN base model, incorporating change points and contextual similarity.
result CLA significantly outperforms FFNN base models in expected return forecasting.
A new memory system handles non-stationary environments by self-sizing and retaining memories.
problem Non-stationary environments where memories arrive over time and must be distinguished from noise.
method A self-sizing continual associative memory that generalizes Hopfield's network, handling adaptation and novelty.
result The memory system grows to the intrinsic memory demand of the environment and retains memories without forgetting.
RUM improves RNN's long-term memory by using unitary matrices.
problem Limited capacity of RNN to manipulate long-term memory.
method Proposes Rotational Unit of Memory (RUM) with unitary matrices.
result RUM learns long-term dependencies and improves state-of-the-art results.
Optimizes learning schedules for better memory retention.
problem Finding the best review schedule for spaced repetition.
method Flexible representation of spaced repetition using marked temporal point processes and optimal control for stochastic differential equations with jumps.
result Optimal reviewing schedule is the recall probability of content.
Enhances generative models with latent features and neighborhood memories.
problem Current generative models only use one of two components: learned features or instance recall.
method Proposes methods to integrate neighborhood information into a flow model's latent space.
result Empirically shows significant improvement over baselines on image datasets.
Real-time fuel leakage detection framework MOCPD improves accuracy.
problem Early detection of fuel leakage to prevent hazards and losses.
method Memory-based Online Change Point Detection (MOCPD) framework.
result MOCPD outperforms baseline methods in detection accuracy.
New methods assess neural connectivity to predict cognitive performance.
problem Predict cognitive performance using neural connectivity.
method Two novel approaches for characterizing functional network connectivity from EEG data.
result Fused AUC results of 0.63/0.58/0.61 for digit recall failure and 0.58/0.59/0.54 for sentence recall failure.
Task-conditioned hypernetworks help neural networks learn multiple tasks without forgetting.
problem Catastrophic forgetting in neural networks when sequentially trained on multiple tasks.
method Task-conditioned hypernetworks that generate target model weights based on task identity.
result Task-conditioned hypernetworks achieve state-of-the-art performance on CL benchmarks and retain long memories.
The paper analyzes the performance of delay-based reservoir computing using eigenvalue analysis.
problem Quantifying the performance of delay-based reservoir computing.
method Eigenvalue analysis of the dynamical system to predict reservoir computing performance.
result The performance of a reservoir computing system can be predicted by analyzing the small signal response and eigenvalue spectrum.
Muon spectral optimizer outperforms SGD in associative memory tasks.
problem Understanding the advantage of spectral optimizers in learning associative memory.
method Linear associative memory problem, Gaussian inputs and outputs, power law frequency distribution, thresholded gradient approximation.
result Muon significantly outperforms SGD in storage capacity and recovery rates.
New algorithm reduces online learning regret for bounded recall games.
problem Reducing regret in online learning with limited past information.
method Constructing a stationary bounded-recall algorithm with O ( 1 / M ) O(1/\sqrt{M}) O ( 1/ M ) regret. result Any low regret bounded-recall algorithm must be aware of past losses' order.
A bridge between continuous signals and discrete Ising spins for associative memory.
problem Associative memory in continuous-signal-driven Ising spin systems.
method Multilayer Ising framework with PCA whitening and SimHash projection, coupled to pseudo-inverse memory couplings.
result Finite-size scaling of operational storage capacity with α c ( N ) = α c ( ∞ ) − c N − 1 / 2 α_c(N)=α_c(\infty)-c\,N^{-1/2} α c ( N ) = α c ( ∞ ) − c N − 1/2 , approaching α c ( ∞ ) ≈ 0.50 α_c(\infty)\approx 0.50 α c ( ∞ ) ≈ 0.50 . LSTM model detects voice disorders with high accuracy.
problem Automated detection of voice disorders is challenging due to continuous audio data.
method Used Long Short Term Memory (LSTM) model for feature extraction and classification of voice disorders.
result 22% sensitivity, 97% specificity, 56% unweighted average recall.
LSTMs fail in financial tasks, CL improves performance.
problem Financial time-series analysis and inference with LSTMs.
method Continual Learning (CL) approach for financial decision making.
result CL outperforms LSTMs and FFNN in financial decision making.
This paper introduces a new method to train normalizing flows using precision-recall divergences.
problem Training generative models with mode dropping and low-quality samples.
method Introduces PR-divergences and proposes a novel generative model to minimize precision-recall trade-offs.
result Normalizing flows can be trained to achieve specific precision-recall trade-offs using PR-divergences.
Maximizes precision and recall at top by thresholding posterior probability.
problem Improving precision and recall at the top in information retrieval and binary classification.
method Thresholding the posterior probability of the positive class.
result Maximizes precision and recall at the top through thresholding.
The study examines how class imbalance affects precision-recall curves.
problem Understanding how precision changes with class imbalance ratios.
method Analyzes the relationship between precision, class imbalance ratio, and true/false positive rates.
result Predicts changes in precision-recall curves and other measures with class imbalance ratios.
Revises precision-recall curves for generative models.
problem Improves evaluation of generative models by distinguishing mode-collapse and quality issues.
method Generalizes PR curve formulation to arbitrary measures, exposes a bridge to error rates, proposes a new algorithm to approximate precision-recall curves.
result Demonstrates the interest of the new formulation over the original approach on multi-modal datasets.
Recommender system improves recall of omitted foods in online dietary surveys.
problem Improving accuracy of online dietary assessment surveys through recall assistance.
method Developed a recommender algorithm to remind respondents of omitted foods based on past survey data.
result The recommender system captures more omitted foods than hand-coded prompts, but with lower precision.
Predicts S&P 500 trends using machine learning models.
problem Market trend prediction for S&P 500 index.
method Feature engineering, machine learning models (Logistic Regression, Decision Trees, Random Forests, Neural Networks, KNN, XGBoost), data preprocessing, hyperparameter tuning, SMOTE.
result KNN for short-term predictions, XGBoost for long-term forecasts.
The study formalizes temporal precision and recall for anomaly detection in sequences.
problem Insufficient understanding of precision and recall in sequential anomaly detection.
method Formalized temporal precision and recall measures, developed time-tolerant confusion matrices, and demonstrated statistical significance.
result Precision and recall may overestimate performance with temporal tolerance.
FSNet improves online time series forecasting by balancing fast adaptation and old knowledge.
problem Online time series forecasting challenges in handling abrupt and recurring patterns.
method Inspired by CLS theory, FSNet uses a dynamic balance between fast adaptation and old knowledge retrieval.
result FSNet achieves robustness to both new and recurring patterns through dynamic balancing and associative memory.