Deep neural networks with memory learn reduced equations from partial data.
problem Constructing governing equations for unknown dynamical systems from limited data.
method Formulate a discrete approximation of memory integrals, use deep neural networks to incorporate history terms.
result Deep neural networks can learn reduced equations with memory from partial data.
Research on long-range memory in financial and social systems using various models.
problem Understanding the nature of long-range memory in socioeconomic systems.
method Various Markov processes including point processes, stochastic differential equations, and agent-based models.
result New estimators of self-similarity and long-range memory for non-Gaussian systems are needed.
GPU (graphics processing unit) has been used for many data-intensive applications. Among them, deep learning systems are one of the most important consumer systems for GPU nowadays. As deep learning applications impose deeper and larger models in order to achieve higher accuracy, memory management becomes an important …
Enhanced financial trading system using multi-agent LLMs with layered memory.
problem Inefficient prioritization of tasks in LLMs due to their memory processing.
method Introducing a multi-agent framework with layered memories and inter-agent debate.
result Superior automated trading accuracy and decision robustness.
Memory-efficient learning for large-scale imaging systems.
problem Memory limitations in GPUs for real-world large-scale inverse problems.
method Exploits reversibility of network layers to enable data-driven design.
result Demonstrated on small-scale and large-scale real-world systems.
Paper proposes a GPU-based system for training massive deep learning models in ads systems.
problem Training massive deep learning models with terabyte-scale parameters in ads systems.
method Hierarchical GPU parameter server with 3-layer storage (GPU High-Bandwidth Memory, CPU main memory, SSD).
result 4-node hierarchical GPU parameter server trains a model 2X faster than a 150-node in-memory system.
The mathematical model of a linear system with the short memory about own stochastic behavior is proposed. It is assumed that the system is under a continual influence of independent stochastic impulses. In a short memory approximation the expression of the stochastic process is found. An application of the model propo…
We propose two optimization techniques to minimize memory usage and computation while meeting system timing constraints for real-time classification in wearable systems. Our method derives a hierarchical classifier structure for Support Vector Machine (SVM) in order to reduce the amount of computations, based on the pr…
SDA method reduces memory and time for assimilating noisy geophysical data.
problem Challenges in identifying state trajectories of high-dimensional geophysical systems.
method Score-based data assimilation with modified score network architecture.
result Promising results for a two-layer quasi-geostrophic model.
Algorithm learns linear systems from partial observations with near-optimal rate.
problem Identifying linear dynamical systems from partial observations, especially those with long-term memory.
method Multi-scale low-rank approximation using SVD on Hankel matrices of increasing sizes, combined with Fourier domain concentration bounds.
result Near-optimal rate of $\widetilde O\left(\sqrt\frac{d}{T}
ight)$ in H2 error, with logarithmic dependence on memory length. Generative diffusion models mimic biological memory networks, encoding associative dynamics in deep neural weights.
problem Understanding long-term memory mechanisms in neuroscience and AI.
method Interpreting generative diffusion models as energy-based models and comparing them to Hopfield networks.
result Generative diffusion models can encode associative dynamics of Hopfield networks in deep neural weights.
Comprehending complex systems by simplifying and highlighting important dynamical patterns requires modeling and mapping higher-order network flows. However, complex systems come in many forms and demand a range of representations, including memory and multilayer networks, which in turn call for versatile community-det…
Memory-Augmented Meta-Optimization improves cold-start recommendation.
problem Cold-start problem in recommender systems for new users or items.
method Memory-Augmented Meta-Optimization approach with personalized and task-specific memories.
result Significant improvement in cold-start recommendation performance on multiple datasets.
PSI-LinUCB improves scalability for large recommender systems.
problem Efficiently training and inferring for large action spaces in recommender systems.
method Represent inverse design matrix as diagonal + low-rank correction, derive stable rank-1 and batched updates, use projector-splitting integrator.
result Demonstrated effectiveness on recommender system datasets, achieving scalable training and inference.
Memory is an important aspect of intelligence and plays a role in many deep reinforcement learning models. However, little progress has been made in understanding when specific memory systems help more than others and how well they generalize. The field also has yet to see a prevalent consistent and rigorous approach f…
A central challenge faced by memory systems is the robust retrieval of a stored pattern in the presence of interference due to other stored patterns and noise. A theoretically well-founded solution to robust retrieval is given by attractor dynamics, which iteratively clean up patterns during recall. However, incorporat…
The study analyzes prediction errors in systems with memory kernels, providing bounds and stability results.
problem Prediction errors in stochastic dynamical systems with memory kernels.
method Analysis of generalized Langevin equations (GLEs) with Volterra equations, integrating synchronized noise coupling and weighted norms.
result Prediction discrepancies decay at a rate determined by the memory kernel's decay, quantitatively bounded by kernel estimation errors.
We propose a generic algorithmic building block to accelerate training of machine learning models on heterogeneous compute systems. Our scheme allows to efficiently employ compute accelerators such as GPUs and FPGAs for the training of large-scale machine learning models, when the training data exceeds their memory cap…
Modeling financial markets with memory using fractional calculus and Brownian motion.
problem Capturing memory effects in financial markets using stochastic models.
method Fractional Langevin equation with colored noise generated by fractional Brownian motion.
result Anomalous marginal glass phase observed in some regions of the system.
Service-induced congestion in memory-constrained LLM serving
problem Service-induced congestion in memory-constrained large language model (LLM) serving
method Developing a discrete-time dynamical model of memory-constrained LLM inference
result The system converges to a unique worst-case limit cycle that is asymptotically stable outside a Lebesgue-measure-zero exact-capture set, with throughput losses as large as 50%.
LEM efficiently models long-term sequences with gradients.
problem Learning long-term sequential dependencies in data.
method Gradient-based multiscale ordinary differential equations system.
result LEM outperforms state-of-the-art models in various tasks.
Unified treatment of RC in stochastic and deterministic settings.
problem Understanding and generalizing reservoir computing in both deterministic and stochastic contexts.
method Investigation of state-space systems, analysis of fading memory and solution stability, introduction of stochastic echo states.
result Generality of fading memory and solution stability in state-space systems, even without the echo state property.
TASO optimizes CNN models for memory-constrained devices.
problem Large CNN models are too resource-intensive for mobile devices.
method Integer Linear Programming (ILP) for selecting operations and workspace allocation.
result 8x speedup and 2.2x reduction in memory usage compared to greedy algorithms.
Our research is focused on understanding and applying biological memory transfers to new AI systems that can fundamentally improve their performance, throughout their fielded lifetime experience. We leverage current understanding of biological memory transfer to arrive at AI algorithms for memory consolidation and repl…
Efficient algorithm predicts unknown linear systems with long-term memory.
problem Predicting unknown and partially observed linear dynamical systems with long-term memory.
method Bounding the generalized Kolmogorov width of the Kalman filter model using spectral methods and conducting tight convex relaxation.
result Competes with Kalman filter in hindsight with only logarithmic regret.
A new memory system handles non-stationary environments by self-sizing and retaining memories.
problem Non-stationary environments where memories arrive over time and must be distinguished from noise.
method A self-sizing continual associative memory that generalizes Hopfield's network, handling adaptation and novelty.
result The memory system grows to the intrinsic memory demand of the environment and retains memories without forgetting.
We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva's sparse distributed memory, it has a robust distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian updat…
Study characterizes memory capacity of quantum reservoirs using transmon qubits.
problem Understanding the memory capacity of quantum reservoirs built with transmon qubits.
method Characterized memory capacity of quantum reservoirs using transmon qubits from IBM, focusing on NMSE and topology complexity.
result Found a peak in memory capacity for configurations with n-1 self-loops, suggesting optimal design for forecasting tasks.
Quantum systems with scrambling improve temporal information processing, but scaling requires exponential overhead.
problem Scalability and memory retention of quantum reservoirs in temporal information processing.
method Examined a quantum reservoir processing framework with scrambling reservoirs modeled by high-order unitary designs, analyzed in noiseless and noisy settings.
result Memory retention improves exponentially with reservoir size but worsens with reservoir iterations, requiring exponential shot overhead for scaling.
Unified framework for blending ML and mechanistic models in dynamical systems.
problem Learning dynamical systems from noisy, partially observed data.
method A unifying framework that combines mechanistic and machine learning approaches.
result Proves that hybrid models can learn memory-dependent model error.
The origin of the long-range memory in the non-equilibrium systems is still an open problem as the phenomenon can be reproduced using models based on Markov processes. In these cases a notion of spurious memory is introduced. A good example of Markov processes with spurious memory is stochastic process driven by a non-…
New cooperative dynamics enhances retrieval performance in neural networks.
problem Understanding emergent computational capabilities in disordered systems.
method Leveraging statistical mechanics, extended neural network architecture for hetero-associative memory.
result Layers trained with less informative datasets develop retrieval regions of the same amplitude, leading to optimal performance.
Neural SVEs model complex systems with memory, outperforming traditional methods.
problem Modeling systems with memory effects and irregular behavior.
method Introducing neural stochastic Volterra equations as a physics-inspired architecture.
result Neural SVEs outperform neural SDEs and DeepONets in various applications.
Trojans can be inserted into deep neural networks without altering training data.
problem Inserting malicious code into deep learning models without changing training data.
method Patches model parameters in memory to achieve desired behavior on specific inputs.
result Efficient patches can induce trojan behavior with minimal risk of detection.
We design and study a Contextual Memory Tree (CMT), a learning memory controller that inserts new memories into an experience store of unbounded size. It is designed to efficiently query for memories from that store, supporting logarithmic time insertion and retrieval operations. Hence CMT can be integrated into existi…
Quantum memory limits set by relativity theory.
problem Quantum memory efficiency and relativity constraints.
method Relativistic quantum field theory and Lieb-Robinson bounds.
result Quantum memory capacity is limited by fundamental physics.
We study the problem of learning associative memory -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version. Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by…
We discuss memory models which are based on tensor decompositions using latent representations of entities and events. We show how episodic memory and semantic memory can be realized and discuss how new memory traces can be generated from sensory input: Existing memories are the basis for perception and new memories ar…
TMNs model brain memory systems for continual learning.
problem Catastrophic forgetting in neural networks.
method Triple Network architecture of GANs, incorporating brain-inspired algorithms.
result New state-of-the-art performance on class-incremental learning benchmarks.
Because deep neural networks (DNNs) rely on a large number of parameters and computations, their implementation in energy-constrained systems is challenging. In this paper, we investigate the solution of reducing the supply voltage of the memories used in the system, which results in bit-cell faults. We explore the rob…
This paper proposes a hardware-oriented dropout algorithm, which is efficient for field programmable gate array (FPGA) implementation. In deep neural networks (DNNs), overfitting occurs when networks are overtrained and adapt too well to training data. Consequently, they fail in predicting unseen data used as test data…
New memory allocation scheme improves image generation performance.
problem Improving episodic and semantic memory representation in neural networks.
method Developed a hierarchical latent variable model with differentiable, locally block allocated latent memory.
result Improved conditional likelihood values on various datasets.
Improved accuracy in neural networks using CTF memory with RPU.
problem Challenges in achieving high accuracy in neural networks using in-memory computing.
method Exploring the trade-off between conductance change range and linearity in CTF memory for RPU-based matrix operations.
result Achieved accuracy of 97.9% on MNIST dataset, 89.1% and 70.5% on CIFAR-10 and CIFAR-100 datasets.
We introduce for the first time the utilization of Long short-term memory (LSTM) neural network architectures for the compensation of fiber nonlinearities in digital coherent systems. We conduct numerical simulations considering either C-band or O-band transmission systems for single channel and multi-channel 16-QAM mo…
New magnetic memory effects found in gravitational waves and memory.
problem Understanding new effects in gravitational waves and memory.
method Analyzing asymptotically-flat spacetimes with slow decay.
result Diverging magnetic memory sourced by curvature and neutrino cloud.
Deep learning predicts baseball home runs with better accuracy.
problem Predicting player performance in baseball.
method Long Short-Term Memory (LSTM) neural network.
result LSTM outperformed other models in predicting home runs.
Agents learn and control complex mechanical systems through shared memories.
problem Controlling multi-joint dynamical systems.
method Coupled autoregressive active inference agents using Bayesian filtering and minimizing expected free energy.
result Demonstrated learning and control of a double mass-spring-damper system.
Delay-SDE-net models time series with memory and uncertainty, outperforming other models.
problem Accurately modeling time series with memory and uncertainty.
method Stochastic delay differential equations (SDDEs) neural network model with aleatoric and epistemic uncertainty.
result The Delay-SDE-net consistently outperforms other models in predicting time series values and uncertainties.