We augment recurrent neural networks with an external memory mechanism that builds upon recent progress in metalearning. We conceptualize this memory as a rapidly adaptable function that we parameterize as a deep neural network. Reading from the neural memory function amounts to pushing an input (the key vector) throug…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Memory Function”
Recurrent neural networks can learn complex transduction problems that require maintaining and actively exploiting a memory of their inputs. Such models traditionally consider memory and input-output functionalities indissolubly entangled. We introduce a novel recurrent architecture based on the conceptual separation b…
A new memory replay mechanism improves reinforcement learning stability and speed.
problem Forgetting in reinforcement learning with continuous control.
method Augmented Memory Replay (AMR) that optimizes the replay of past experiences.
result AMR enhances stability and convergence speed of learning algorithms.
Enhanced Hopfield model boosts memory retrieval capacity.
problem Memory retrieval in modern Hopfield models with limited capacity.
method Introduces a learnable feature map transforming energy function into kernel space, minimizing separation loss for uniform memory distribution.
result Significant reduction in metastable states, enhancing memory capacity and retrieval accuracy.
A general market model with memory is considered in terms of stochastic functional differential equations. We aim at representation formulae for the sensitivity analysis of the dependence of option prices on the memory. This implies a generalization of the concept of delta.
Memory-constrained algorithms need superlinear memory for efficient convex optimization.
problem Efficiently minimizing convex functions with limited memory.
method Analyzing first-order algorithms with superlinear memory constraints.
result Superlinear memory is necessary for optimal performance in convex optimization.
Novel method learns memory kernels in Langevin equations.
problem Estimating memory kernels in Langevin equations.
method Regularized Prony method for correlation functions, followed by regression over Sobolev norm-based loss function with RKHS regularization.
result Method outperforms other regression estimators in exponentially weighted L^2 space.
Study on memory effects in RNNs learning temporal data.
problem Understanding memory effects in RNNs for temporal data learning.
method Mathematical analysis of continuous-time linear RNNs, focusing on approximation and optimization dynamics.
result Long-term memory requires a large number of neurons and slows down training.
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of mo…
We note that known methods achieving the optimal oracle complexity for first order convex optimization require quadratic memory, and ask whether this is necessary, and more broadly seek to characterize the minimax number of first order queries required to optimize a convex Lipschitz function subject to a memory constra…
Typical neural networks with external memory do not effectively separate capacity for episodic and working memory as is required for reasoning in humans. Applying knowledge gained from psychological studies, we designed a new model called Differentiable Working Memory (DWM) in order to specifically emulate human workin…
Hierarchical Memory Networksstat.ML
Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is often addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which …
New memory models improve LSSVM's generalization and reduce time cost.
problem Improving LSSVM's generalization performance and reducing training time.
method Introducing MIMM and WIMM models that modify LSSVM's memory influence mechanism.
result MIMM and WIMM models outperform LSSVM in generalization and time efficiency.
Generalizes memory and forecasting capacities for nonlinear recurrent networks with dependent inputs.
problem Understanding memory and forecasting capabilities in networks with dependent inputs.
method Formulated bounds for memory and forecasting capacities in terms of network size and input properties.
result Proved that memory capacity for linear recurrent networks with independent inputs is given by the rank of the controllability matrix.
This paper introduces a hierarchical associative memory model with multiple layers.
problem Limitations of traditional associative memory models with only one hidden layer.
method Develops a fully recurrent model with arbitrary layers, including locally connected ones, and a corresponding energy function.
result The model can dynamically assemble memories using weights from lower layers and higher layers' rules.
This work analyzes and optimizes memory and compute costs of learned optimizers.
problem High memory and compute costs of learned optimizers.
method Identified and quantified design features of learned and hand-designed optimizers, constructed a more efficient learned optimizer.
result A learned optimizer that is faster and more memory efficient than previous work.
New memory-query tradeoffs for convex optimization algorithms.
problem Optimizing memory usage for convex optimization algorithms.
method Analyzing randomized first-order algorithms for minimizing convex functions.
result Cutting plane methods are optimal in terms of memory and query complexity.
A new model for sequential memory using temporal predictive coding.
problem Forming accurate memory of sequential stimuli in the brain.
method Proposes a novel PC-based model called temporal predictive coding (tPC).
result Shows that tPC models can accurately memorize and retrieve sequential inputs.
Study identifies three quantization regimes for ReLU networks.
problem Approximation of Lipschitz functions by ReLU networks with finite-precision weights.
method Established through nonasymptotic tight lower and upper bounds on minimax approximation error.
result Memory-optimality achieved in proper quantization regime for deep networks.
Generative diffusion models mimic biological memory networks, encoding associative dynamics in deep neural weights.
problem Understanding long-term memory mechanisms in neuroscience and AI.
method Interpreting generative diffusion models as energy-based models and comparing them to Hopfield networks.
result Generative diffusion models can encode associative dynamics of Hopfield networks in deep neural weights.
A technique identifies memoryless algorithms approximating memory-dependent optimization methods.
problem Understanding how memory in optimization algorithms affects loss and generalization.
method Introducing a general technique to replace past iterates with the current one and adding a correction term.
result Lion does not have the same implicit anti-regularization as AdamW, explaining its better generalization performance.
Sparse Hopfield model improves memory retrieval with fewer connections.
problem Memory retrieval efficiency with fewer connections.
method Sparse extension of Hopfield model, derived from sparse entropic regularizer.
result Sparse Hopfield model achieves tighter error bounds and better performance.
Memory Organization for Energy-Efficient Learning and Inference in Digital Neuromorphic Acceleratorscs.NE
Improves energy efficiency of neuromorphic hardware by optimizing memory organization and encoding schemes.
problem Energy inefficiency in neuromorphic hardware, especially in digital accelerators.
method Synthesized controller and memory for different encoding schemes, introduced functional encoding for structured connectivity.
result Functional encoding offers a 58% reduction in energy for weight updates in convolutional layers.
Heavy-tail driven by memoryq-fin.ST
We propose a stochastic process driven by memory effect with novel distributions including both exponential and leptokurtic heavy-tailed distributions. A class of distribution is analytically derived from the continuum limit of the discrete binary process with the renormalized auto-correlation and the closed form momen…
Energy Transformercs.LG
Energy Transformer integrates attention, energy models, and associative memory.
problem Lack of clear theoretical foundations in attention mechanisms and straightforward design of energy functions in energy-based models.
method Proposes Energy Transformer, a sequence of attention layers with a specifically engineered energy function.
result Obtained strong results on graph anomaly detection and classification tasks.
A new theory explains large associative memory with biological plausibility.
problem Large associative memory in neurobiology and machine learning.
method Microscopic theory with hidden neurons and two-body interactions.
result Valid model of large associative memory with biological plausibility.
A central challenge faced by memory systems is the robust retrieval of a stored pattern in the presence of interference due to other stored patterns and noise. A theoretically well-founded solution to robust retrieval is given by attractor dynamics, which iteratively clean up patterns during recall. However, incorporat…
Despite their attractiveness, popular perception is that techniques for nonparametric function approximation do not scale to streaming data due to an intractable growth in the amount of storage they require. To solve this problem in a memory-affordable way, we propose an online technique based on functional stochastic …
Brownian marketsq-fin.ST
Financial market dynamics is rigorously studied via the exact generalized Langevin equation. Assuming market Brownian self-similarity, the market return rate memory and autocorrelation functions are derived, which exhibit an oscillatory-decaying behavior with a long-time tail, similar to empirical observations. Individ…
New neural operators model turbulence with memory and randomness.
problem Modeling turbulence in complex fluid dynamics with memory and randomness.
method Symmetrized activation functions, fractional derivatives, and stochastic noise.
result Theoretical guarantees for approximation quality in turbulent phenomena.
Deep learning typically requires training a very capable architecture using large datasets. However, many important learning problems demand an ability to draw valid inferences from small size datasets, and such problems pose a particular challenge for deep learning. In this regard, various researches on "meta-learning…
We obtain option pricing formulas for stock price models in which the drift and volatility terms are functionals of a continuous history of the stock prices. That is, the stock dynamics follows a nonlinear stochastic functional differential equation. A model with full memory is obtained via approximation through a stoc…
Tensor models decode human perception and memory using SPO triples.
problem Understanding implicit and explicit perception and memory in the brain.
method Tensor models with SPO triples, dual representations, and four layers.
result Semantic memory is crucial for explicit perception and declarative memories.
Meta-learning approach for fast and compressive energy-based memory models.
problem Learning associative memory with fast and compressive models for complex data.
method Meta-learning approach to energy-based memory models (EBMM) using arbitrary neural architectures.
result Demonstrated associative retrieval outperforming existing systems in reconstruction error and compression rate.
Develops hyperparameter transfer methods for Dense Associative Memories.
problem Challenges in transferring hyperparameters for DenseAMs due to unique architecture and activation functions.
method Derives explicit prescriptions for hyperparameter transfer from small to large models.
result Excellent agreement between theoretical and empirical results.
HiPPO framework optimizes memory compression for sequential data.
problem Incremental representation of cumulative history in sequential data.
method Optimal polynomial projections for online function approximation.
result HiPPO-LegS achieves state-of-the-art accuracy on MNIST.
The long memory of the efficient marketcond-mat.other
For the London Stock Exchange we demonstrate that the signs of orders obey a long-memory process. The autocorrelation function decays roughly as with , corresponding to a Hurst exponent . This implies that the signs of future orders are quite predictable from the signs of past orde…
V-HMN integrates memory mechanisms for improved image recognition.
problem Limited interpretability and high data requirements of existing vision backbones.
method Brain-inspired hierarchical memory modules with iterative refinement.
result V-HMN achieves strong performance on image classification benchmarks.
Quadratic memory is essential for optimal convex optimization queries.
problem Optimal query complexity for convex optimization and feasibility problems.
method Lower bounds on query complexity for convex optimization and feasibility problems.
result Center-of-mass algorithms are Pareto-optimal for both convex optimization and feasibility problems.
Survey on statistical inference under memory constraints.
problem Effect of memory limitations on statistical inference performance.
method Review of state-of-the-art in several canonical problems.
result Identification of fundamental building blocks and useful techniques.
New matrix approximation method using RBF components for better memory efficiency.
problem Efficiently approximate any real matrix without being symmetric or positive definite.
method Formulate as an optimization problem with gradient descent methods.
result Significantly reduces memory usage for various matrix types.
A method for learning complex functions from data with reduced memory usage.
problem Learning highly nonlinear, multivariate functions from examples.
method Transforming function learning into tensor reconstruction, incrementally building tensors from rank-one terms.
result Efficient gradient-based algorithm with linear time complexity in sample size and tensor dimensions.
MEM learns set functions from permutation-invariant data.
problem Learning from sets of instances with labels only on sets, not instances.
method Memory-based Exchangeable Model (MEM) with self-attention mechanism.
result Achieved 84.84% accuracy on lung cancer classification.
The origin of the long-range memory in the non-equilibrium systems is still an open problem as the phenomenon can be reproduced using models based on Markov processes. In these cases a notion of spurious memory is introduced. A good example of Markov processes with spurious memory is stochastic process driven by a non-…
On the interplay between short and long term memory in the power-law cross-correlations settingstat.ME
We focus on emergence of the power-law cross-correlations from processes with both short and long term memory properties. In the case of correlated error-terms, the power-law decay of the cross-correlation function comes automatically with the characteristics of separate processes. Bivariate Hurst exponent is then equa…
Memory capacity of DAM scales exponentially with feature separation, unaffected by correlations.
problem Understanding how feature correlations impact DAM's capacity.
method Developed an empirical framework to analyze DAM's capacity under varying feature correlations and pattern separations.
result Memory capacity scales exponentially with feature separation, unaffected by correlations.
We consider the problem of performing linear regression over a stream of -dimensional examples, and show that any algorithm that uses a subquadratic amount of memory exhibits a slower rate of convergence than can be achieved without memory constraints. Specifically, consider a sequence of labeled examples $(a_1,b_1)…
New algorithms for constrained online optimization with memory and predictions.
problem Control of constrained dynamical systems and scheduling with reconfiguration budgets.
method Proposed algorithms achieving sublinear regret and constraint violation under time-varying constraints, both with and without predictions.
result First algorithms achieving sublinear regret and constraint violation in constrained online optimization with memory.