Generative memory model avoids vanishing gradients to robustly retrieve patterns.
problem Robust retrieval of stored patterns in the presence of interference and noise.
method Training a generative distributed memory without explicitly simulating attractor dynamics, using a likelihood-based Lyapunov function.
result The model converges to correct patterns upon iterative retrieval and achieves competitive performance as a memory model and a generative model.
Improves memory robustness in RNNs for sequential data.
problem Improving memory robustness in RNNs for sequential data processing.
method Utilized various training protocols, datasets, and architectures to analyze hidden state dynamics and propose a regularization technique.
result Manipulating hidden state speeds improves memory robustness over time.
NeuZip compresses neural network weights to save memory during training and inference.
problem Memory constraints in neural network training and inference.
method Entropy-based dynamic weight compression.
result Significant reduction in memory usage without performance loss.
Generative diffusion models mimic biological memory networks, encoding associative dynamics in deep neural weights.
problem Understanding long-term memory mechanisms in neuroscience and AI.
method Interpreting generative diffusion models as energy-based models and comparing them to Hopfield networks.
result Generative diffusion models can encode associative dynamics of Hopfield networks in deep neural weights.
Meta-learning approach for fast and compressive energy-based memory models.
problem Learning associative memory with fast and compressive models for complex data.
method Meta-learning approach to energy-based memory models (EBMM) using arbitrary neural architectures.
result Demonstrated associative retrieval outperforming existing systems in reconstruction error and compression rate.
AUTR generates sentences using a dynamic memory and attention mechanism.
problem Generating coherent sentences without explicit training data.
method Recurrent neural network with dynamic attention and canvas memory.
result AUTR learns meaningful latent representations and achieves competitive performance.
A method models nonlinear dynamics from data using barycentric coordinates and memory.
problem Modeling complex dynamical systems from data.
method SPA for data projection, barycentric coordinates, delay-embedding theorem for memory.
result Stable models of chaotic dynamics and attractors are reproduced.
New method combines long-memory reservoirs for accurate dengue forecasting from short data.
problem Accurate dengue forecasting from short, noisy, non-stationary, and nonlinear data.
method Fractional ESN and Wavelet ESN frameworks integrating long-term memory.
result fESN and wESN outperform baselines in multiple dengue datasets and forecasting horizons.
New model forecasts long-memory series with time-varying parameters.
problem Forecasting long-memory series with dynamic parameters.
method Proposes a new long-memory model with a time-varying fractional parameter, driven by predictive likelihood score.
result Validated through Monte Carlo experiment and real data applications.
New insights link memory loss to system stability in dynamical systems.
problem Understanding the relationship between dynamics and computation, especially stability and memory loss.
method Analyzing driven dynamical systems responding to temporal inputs.
result Memory loss in driven systems leads to consistent responses to similar inputs and affects stability.
New algorithm reduces decision switching in dynamic environments.
problem Online learning with memory and non-stationary environments.
method Dynamic policy regret, novel ensemble approach, meta-base decomposition.
result Proves optimal dynamic policy regret for memory length, non-stationarity, and time horizon.
A new algorithm reduces memory usage for deep learning models.
problem Training deep learning models requires significant memory.
method Dynamic Tensor Rematerialization (DTR) is a greedy online algorithm that dynamically plans recomputations.
result DTR achieves comparable performance to optimal static checkpointing with only a small memory budget.
Study on memory effects in RNNs learning temporal data.
problem Understanding memory effects in RNNs for temporal data learning.
method Mathematical analysis of continuous-time linear RNNs, focusing on approximation and optimization dynamics.
result Long-term memory requires a large number of neurons and slows down training.
Learning and memory are intertwined in our brain and their relationship is at the core of several recent neural network models. In particular, the Attention-Gated MEmory Tagging model (AuGMEnT) is a reinforcement learning network with an emphasis on biological plausibility of memory dynamics and learning. We find that …
Area attention allows dynamic area-based attention in memory.
problem Fixed attention granularity limits model performance.
method Area attention dynamically determines area shape and size via learning.
result Area attention improves performance on neural machine translation and image captioning.
A new method learns from multi-modal sequences with external memory.
problem Learning new modes in a dynamic environment without prior knowledge.
method Maintains a neural episodic memory with a Dirichlet Process prior to store mode descriptors and transfers knowledge through retrieval.
result Performs continual learning favorably compared to mainstream approaches.
Method interprets LSTMs at the cell level for better understanding of their dynamics.
problem Understanding the dynamics of LSTMs at the cell level.
method A systematic pipeline for interpreting individual hidden state dynamics using response characterization methods.
result Identifies neurons with insightful dynamics and quantifies their impact on network performance.
Optimal linear contracts are possible even with memory in Gaussian settings.
problem Can optimal dynamic contracts be linear when agents control memory processes?
method Developed a methodology for non-Markovian and non-semimartingale settings, showed linear contracts are optimal for one-dimensional models.
result Linear contracts are optimal for one-dimensional models with memory, and for radial effort cost functions in higher dimensions.
Serenity optimizes neural network execution for edge devices by scheduling with optimal memory footprint.
problem Order of nodes in irregular neural networks affects memory footprint, complicating execution under resource constraints.
method Memory-aware compiler using dynamic programming and graph rewriting to find optimal schedules.
result Achieves optimal peak memory and further improves it with graph rewriting, reducing memory usage by 1.68x-1.86x compared to TensorFlow Lite.
A new method reduces memory usage in neural networks by 36-81%.
problem Minimizing memory usage in neural networks during backpropagation.
method Formalized as a graph theory problem, uses dynamic programming to minimize computational overhead.
result Reduces peak memory consumption by 36-81% on various networks.
Product Kanerva Machines dynamically combine smaller models for better memory organization.
problem Limited organization in the Kanerva Machine.
method Introducing Product Kanerva Machines that dynamically combine multiple smaller Kanerva Machines.
result Product Kanerva Machines can discover spatial tunings that approximately factorize simple images by object.
Neural ARFIMA model improves exchange rate forecasting for BRIC economies.
problem Forecasting exchange rates for emerging markets with long-term memory and nonlinear dynamics.
method Integrates ARFIMA for long-memory with neural networks for nonlinear approximation.
result NARFIMA model outperforms benchmarks in BRIC exchange rate forecasting.
Unified treatment of RC in stochastic and deterministic settings.
problem Understanding and generalizing reservoir computing in both deterministic and stochastic contexts.
method Investigation of state-space systems, analysis of fading memory and solution stability, introduction of stochastic echo states.
result Generality of fading memory and solution stability in state-space systems, even without the echo state property.
New method uses neural networks to create models with memory effects.
problem Accurately modeling memory effects in reduced models.
method Analogies between recurrent neural networks and Mori-Zwanzig formalism to develop reduced models with memory.
result The proposed method produces reduced models with good performance on short-term and long-term predictions.
Enhanced Hopfield model boosts memory retrieval capacity.
problem Memory retrieval in modern Hopfield models with limited capacity.
method Introduces a learnable feature map transforming energy function into kernel space, minimizing separation loss for uniform memory distribution.
result Significant reduction in metastable states, enhancing memory capacity and retrieval accuracy.
Deep neural networks with memory learn reduced equations from partial data.
problem Constructing governing equations for unknown dynamical systems from limited data.
method Formulate a discrete approximation of memory integrals, use deep neural networks to incorporate history terms.
result Deep neural networks can learn reduced equations with memory from partial data.
The paper examines how long-memory dynamics, rough-volatility, and persistence affect equity volatility forecasting.
problem The study investigates how long-memory dynamics, rough-volatility, and persistence impact equity volatility forecasting.
method The paper combines semiparametric long-memory estimation, rough-volatility diagnostics, and structured forecasting regressions.
result Persistence measures improve out-of-sample volatility forecasts, particularly during periods of elevated market volatility and in volatility-managed portfolio applications.
Dynamic memory prevents forgetting in continuous learning of medical images.
problem Catastrophic forgetting in machine learning models over time due to domain shifts.
method Dynamic memory to store and replay diverse training data subsets.
result Dynamic memory mitigates forgetting without knowing when shifts occur.
NEST optimizes deep learning training by placing devices efficiently across networks and memory.
problem Inefficient device placement in distributed deep learning leads to high communication and memory overhead.
method NEST uses network-, compute-, and memory-aware dynamic programming to optimize device placement.
result NEST achieves up to 2.43 times higher throughput and better memory efficiency.
The study examines how investor protection and past information affect stock returns and interest rates.
problem Empirical regularities related to investor protection and past information in asset pricing models.
method Developed a dynamic asset pricing model with a controlling shareholder and good/bad memory in budget dynamics.
result Good/bad memory of investors on historical market information affects stock returns and interest rates, strengthening investor protection in high ownership concentration.
A new memory replay mechanism improves reinforcement learning stability and speed.
problem Forgetting in reinforcement learning with continuous control.
method Augmented Memory Replay (AMR) that optimizes the replay of past experiences.
result AMR enhances stability and convergence speed of learning algorithms.
Experience replay is a key technique behind many recent advances in deep reinforcement learning. Allowing the agent to learn from earlier memories can speed up learning and break undesirable temporal correlations. Despite its wide-spread application, very little is understood about the properties of experience replay. …
This paper tackles federated incremental learning with dynamic memory allocation for improved model performance in non-IID data.
problem Catastrophic forgetting in federated healthcare systems with non-IID data.
method Dynamic memory allocation strategy based on data replay mechanism.
result Significant performance improvements in medical image datasets compared to baseline models.
Proposes a plastic neural memory model for better anomaly detection.
problem Static attention mechanisms limit NMNs in anomaly detection.
method Introduces dynamic connection weights for improved knowledge retrieval.
result Outperforms state-of-the-art in three medical anomaly detection tasks.
New neural operators model turbulence with memory and randomness.
problem Modeling turbulence in complex fluid dynamics with memory and randomness.
method Symmetrized activation functions, fractional derivatives, and stochastic noise.
result Theoretical guarantees for approximation quality in turbulent phenomena.
New model captures long-term memory effects in epidemic dynamics.
problem Identifying memory effects in disease progression and recovery.
method Physics-informed neural networks (PINN) with fractional SEIRD model.
result Fractional memory order α improves predictive performance over classical models. DCRNN improves LSTM for chaotic dynamical system forecasting.
problem Modeling chaotic dynamical systems with recurrent neural networks.
method DCRNN incorporates learnable skip-connections and a Lyapunov stability regularization term.
result DCRNN outperforms LSTM in 100 out of 100 experiments, reducing mean squared error by 80.0%.
Study adds memory effect to Solow-Swan model for more accurate economic growth modeling.
problem Inaccuracies in classical Solow-Swan model in capturing long-term dynamics.
method Introduced fractional calculus with Caputo derivative into Solow-Swan framework.
result Fractional-order model shows significant impact on capital accumulation and stability.
Neural SVEs model complex systems with memory, outperforming traditional methods.
problem Modeling systems with memory effects and irregular behavior.
method Introducing neural stochastic Volterra equations as a physics-inspired architecture.
result Neural SVEs outperform neural SDEs and DeepONets in various applications.
Improved recurrent neural networks learn long-term dependencies through multi-scale memory.
problem Capturing long-term dependencies in recurrent neural networks.
method Incremental training of a modular RNN architecture with multi-scale hidden states.
result Incremental training and multi-scale memory enhance RNNs' ability to learn long-term dependencies.
The paper explores how gradient descent trains associative memories, revealing oscillations and convergence issues.
problem Training dynamics of associative memories in overparameterized and underparameterized settings.
method Reduction to particle system dynamics, theory, and experiments.
result Oscillatory transitory regimes and benign loss spikes in overparameterized settings, suboptimal memorization in underparameterized settings.
This paper introduces a hierarchical associative memory model with multiple layers.
problem Limitations of traditional associative memory models with only one hidden layer.
method Develops a fully recurrent model with arbitrary layers, including locally connected ones, and a corresponding energy function.
result The model can dynamically assemble memories using weights from lower layers and higher layers' rules.
Unified framework for blending ML and mechanistic models in dynamical systems.
problem Learning dynamical systems from noisy, partially observed data.
method A unifying framework that combines mechanistic and machine learning approaches.
result Proves that hybrid models can learn memory-dependent model error.
Derivation of reduced order representations of dynamical systems requires the modeling of the truncated dynamics on the retained dynamics. In its most general form, this so-called closure model has to account for memory effects. In this work, we present a framework of operator inference to extract the governing dynamic…
RNN-HAR model improves VaR forecasting with long-memory and non-linear dynamics.
problem Efficiently forecasting Value at Risk (VaR) with long-memory and non-linear realized volatility.
method Loss-based generalized Bayesian inference with Sequential Monte Carlo for model estimation and prediction.
result RNN-HAR model consistently outperforms other VaR forecasting models.
Enhances option pricing with fractional order Black-Scholes-Merton model.
problem Improving precision and authenticity of option pricing.
method Integrates fractional order Black-Scholes-Merton with neural networks.
result Improves accuracy in capturing complex diffusion dynamics and memory effects.
Stable Hadamard Memory improves reinforcement learning by efficiently managing memory.
problem Memory models struggle in partially observable reinforcement learning environments.
method Introduces a novel memory model using the Hadamard product for efficient memory management and updates.
result Significantly outperforms state-of-the-art memory-based methods on challenging benchmarks.
The study analyzes prediction errors in systems with memory kernels, providing bounds and stability results.
problem Prediction errors in stochastic dynamical systems with memory kernels.
method Analysis of generalized Langevin equations (GLEs) with Volterra equations, integrating synchronized noise coupling and weighted norms.
result Prediction discrepancies decay at a rate determined by the memory kernel's decay, quantitatively bounded by kernel estimation errors.