New neural operators model turbulence with memory and randomness.
problem Modeling turbulence in complex fluid dynamics with memory and randomness.
method Symmetrized activation functions, fractional derivatives, and stochastic noise.
result Theoretical guarantees for approximation quality in turbulent phenomena.
We design and study a Contextual Memory Tree (CMT), a learning memory controller that inserts new memories into an experience store of unbounded size. It is designed to efficiently query for memories from that store, supporting logarithmic time insertion and retrieval operations. Hence CMT can be integrated into existi…
New algorithm improves PPS for multi-object matching.
problem Efficiently synchronize partial permutations for multi-object matching.
method Proposed CEMP-Partial algorithm for partial permutation synchronization (PPS). Uses sparse matrix operations and nonconvex weighted projected power method.
result Proves CEMP-Partial can exactly classify corrupted and clean partial permutations under adversarial corruption.
Designing deep learning models for highly-constrained hardware would allow imbuing many edge devices with intelligence. Microcontrollers (MCUs) are an attractive platform for building smart devices due to their low cost, wide availability, and modest power usage. However, they lack the computational resources to run ne…
Regression learns Mori-Zwanzig operators for dynamical systems.
problem Learning Mori-Zwanzig operators for complex dynamical systems.
method Statistical regression to extract Markov and memory operators.
result Regression models improve learning of memory-dependent corrections.
Recent empirical results on long-term dependency tasks have shown that neural networks augmented with an external memory can learn the long-term dependency tasks more easily and achieve better generalization than vanilla recurrent neural networks (RNN). We suggest that memory augmented neural networks can reduce the ef…
Heretofore, neural networks with external memory are restricted to single memory with lossy representations of memory interactions. A rich representation of relationships between memory pieces urges a high-order and segregated relational memory. In this paper, we propose to separate the storage of individual experience…
NEST optimizes deep learning training by placing devices efficiently across networks and memory.
problem Inefficient device placement in distributed deep learning leads to high communication and memory overhead.
method NEST uses network-, compute-, and memory-aware dynamic programming to optimize device placement.
result NEST achieves up to 2.43 times higher throughput and better memory efficiency.
Develops a new deep learning formulation using Mori-Zwanzig formalism.
problem Improves deep learning by introducing a new concept of memory.
method Uses Mori-Zwanzig formalism to propagate quantities of interest through neural networks.
result Rigorously transforms deep networks into shallow ones using decay property of memory operator.
A new algorithm reduces memory usage for deep learning models.
problem Training deep learning models requires significant memory.
method Dynamic Tensor Rematerialization (DTR) is a greedy online algorithm that dynamically plans recomputations.
result DTR achieves comparable performance to optimal static checkpointing with only a small memory budget.
Model combines long-term and short-term memory using conceptors.
problem Transfer between long-term and short-term memory.
method Recurrent neural network with gated reservoir for short-term memory and conceptors for long-term memory.
result Standard operations on conceptors allow combining long-term memories and describing their effect on short-term memory.
A new memory system handles non-stationary environments by self-sizing and retaining memories.
problem Non-stationary environments where memories arrive over time and must be distinguished from noise.
method A self-sizing continual associative memory that generalizes Hopfield's network, handling adaptation and novelty.
result The memory system grows to the intrinsic memory demand of the environment and retains memories without forgetting.
Derives operational-time variance kernel for reaction boundaries in financial markets.
problem Separating components in volatility models to better understand market dynamics.
method Derives a variance kernel for a latent-order-book reaction boundary, separating structural boundary cumulant, clock projection, and pricing-measure choice.
result Operational variance has a closed asymptotic form for long-memory forcing, with effective signed-forcing intensity and resilience.
In recent years, we have witnessed an increased interest in temporal modeling of patient records from large scale Electronic Health Records (EHR). While simpler RNN models have been used for such problems, memory networks, which in other domains were found to generalize well, are underutilized. Traditional memory netwo…
HOPE improves SSMs for long-memory tasks with robust initialization and training.
problem Improving state-space models for long-memory tasks with robust initialization and training.
method Developed a new parameterization scheme called HOPE using Hankel operators and Markov parameters.
result HOPE improves SSMs' performance on Long-Range Arena tasks and demonstrates non-decaying memory.
Derives variance kernel for reaction boundary in financial models.
problem Separating components in financial volatility models.
method Operational-time variance kernel, damped Abel response kernel, closed asymptotic form.
result Operational variance has a closed asymptotic form involving various parameters.
POET enables large neural network training on tiny devices with reduced energy.
problem Training large neural networks on memory-limited edge devices.
method Jointly optimizes rematerialization and paging for memory reduction, formulating an MILP for energy-efficient training.
result POET trains ResNet-18 and BERT within Cortex-M memory constraints, outperforming current methods in energy efficiency.
TASO optimizes CNN models for memory-constrained devices.
problem Large CNN models are too resource-intensive for mobile devices.
method Integer Linear Programming (ILP) for selecting operations and workspace allocation.
result 8x speedup and 2.2x reduction in memory usage compared to greedy algorithms.
We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read}, compose and write operations. NSE c…
HS-FNO models non-Markovian PDEs by learning history and future states.
problem Non-Markovian dynamics where future states depend on past history.
method History-Space Fourier Neural Operator (HS-FNO) for delay and memory-driven PDEs.
result HS-FNO achieves lowest aggregate errors across various PDE families.
Efficient memory layer improves graph neural networks for graph classification and regression.
problem Efficiently learning node representations and graph coarsening for arbitrary graph topology.
method Introduces a memory layer for GNNs that learns node representations and graph coarsening, and two new networks: MemGNN and GMN.
result Proposed models achieve state-of-the-art results in graph classification and regression benchmarks.
XLA compiler extension improves memory efficiency for machine learning.
problem Memory constraints limit the scalability of memory-intensive machine learning algorithms.
method Developed an XLA compiler extension that adjusts algorithm data-flow representation to fit memory limits.
result k-nearest neighbour and sparse Gaussian process regression can be run at larger scales.
A bridge between continuous signals and discrete Ising spins for associative memory.
problem Associative memory in continuous-signal-driven Ising spin systems.
method Multilayer Ising framework with PCA whitening and SimHash projection, coupled to pseudo-inverse memory couplings.
result Finite-size scaling of operational storage capacity with αc(N)=αc(∞)−cN−1/2, approaching αc(∞)≈0.50. Cyclic Data Parallelism reduces memory usage and balances gradient communications.
problem Training large deep learning models requires efficient parallelism to scale.
method Cyclic Data Parallelism shifts micro-batches from simultaneous to sequential execution, balancing memory and gradient communications.
result Cyclic Data Parallelism reduces total memory usage and balances gradient communications.
Introduces Exponentially Weighted Signature for better path representation.
problem Uniform treatment of historical information in signatures.
method Generalizes EFM signature to bounded linear operators, enabling contextualised temporal weighting.
result EWS is the unique solution to a linear controlled differential equation and generalizes state-space models.
Improved accuracy in neural networks using CTF memory with RPU.
problem Challenges in achieving high accuracy in neural networks using in-memory computing.
method Exploring the trade-off between conductance change range and linearity in CTF memory for RPU-based matrix operations.
result Achieved accuracy of 97.9% on MNIST dataset, 89.1% and 70.5% on CIFAR-10 and CIFAR-100 datasets.
This study examines memory effects in S&P500 market correlations using Langevin models.
problem The neglect of memory effects in market correlations for optimal portfolio selection.
method Fit a generalised Langevin equation (GLE) to S&P500 market correlation data.
result Memory effects in market correlations significantly improve forecasting accuracy and suggest a hidden slow time scale.
For successful deployment of deep neural networks on highly--resource-constrained devices (hearing aids, earbuds, wearables), we must simplify the types of operations and the memory/power resources used during inference. Completely avoiding inference-time floating-point operations is one of the simplest ways to design …
SGQuant reduces GNN memory usage without significant accuracy loss.
problem High memory consumption in GNNs limits their applicability on memory-constrained devices.
method Proposes a specialized GNN quantization scheme (SGQuant) with a quantization algorithm, fine-tuning scheme, and multi-granularity strategy.
result SGQuant reduces GNN memory footprint from 4.25x to 31.9x with minimal accuracy loss.
Neural SVEs model complex systems with memory, outperforming traditional methods.
problem Modeling systems with memory effects and irregular behavior.
method Introducing neural stochastic Volterra equations as a physics-inspired architecture.
result Neural SVEs outperform neural SDEs and DeepONets in various applications.
Efficiently discovers Bayesian network structure with reduced memory usage.
problem Bayesian network structure discovery is NP-hard and memory-intensive.
method Progressively leveled scoring approach with hierarchical computation.
result Achieved processing of a 28-variable Bayesian network using only memory.
Long and short memory in economic processes is usually described by the so-called discrete fractional differencing and fractional integration. We prove that the discrete fractional differencing and integration are the Grunwald-Letnikov fractional differences of non-integer order d. Equations of ARIMA(p,d,q) and ARFIMA(…
In this report we review memory-based meta-learning as a tool for building sample-efficient strategies that learn from past experience to adapt to any task within a target class. Our goal is to equip the reader with the conceptual foundations of this tool for building new, scalable agents that operate on broad domains.…
The implementation of Deep Convolutional Neural Networks (ConvNets) on tiny end-nodes with limited non-volatile memory space calls for smart compression strategies capable of shrinking the footprint yet preserving predictive accuracy. There exist a number of strategies for this purpose, from those that play with the to…
Derivation of reduced order representations of dynamical systems requires the modeling of the truncated dynamics on the retained dynamics. In its most general form, this so-called closure model has to account for memory effects. In this work, we present a framework of operator inference to extract the governing dynamic…
We analyse perception and memory, using mathematical models for knowledge graphs and tensors, to gain insights into the corresponding functionalities of the human mind. Our discussion is based on the concept of propositional sentences consisting of \textit{subject-predicate-object} (SPO) triples for expressing elementa…
Both the human brain and artificial learning agents operating in real-world or comparably complex environments are faced with the challenge of online model selection. In principle this challenge can be overcome: hierarchical Bayesian inference provides a principled method for model selection and it converges on the sam…
A novel online GP model captures long-term memory in sequential data.
problem Capturing long-term memory in sequential data online.
method Integrates HiPPO framework into interdomain GP, leveraging time-varying orthogonal projections as inducing variables.
result OHSVGP outperforms existing online GP methods in predictive performance, long-term memory preservation, and computational efficiency.
This paper optimizes SMPC for neural network inference, reducing memory and time.
problem Memory and time constraints in secure neural network inference.
method Implemented ABY2.0 protocol, optimized memory usage, and used a helper node.
result MNIST inference reduced from 8.03 GB RAM and 200s to 0.2 GB RAM and 32s.
While accelerators such as GPUs have limited memory, deep neural networks are becoming larger and will not fit with the memory limitation of accelerators for training. We propose an approach to tackle this problem by rewriting the computational graph of a neural network, in which swap-out and swap-in operations are ins…
This paper investigates the role of sparsity in Reservoir Computing networks.
problem Designing efficient Recurrent Neural Networks (RNNs) with hidden recurrent layers.
method Empirical investigation of sparsity in input-reservoir connections and recurrent connections.
result Sparsity, particularly in input-reservoir connections, enhances the network's temporal memory and dimensionality.
State of the art deep learning models have made steady progress in the fields of computer vision and natural language processing, at the expense of growing model sizes and computational complexity. Deploying these models on low power and mobile devices poses a challenge due to their limited compute capabilities and str…
New algorithm reduces memory usage for multi-pass bandit problems.
problem Memory-efficient multi-pass bandit algorithms for large action spaces.
method Develops a B-pass algorithm with O(1) memory that achieves optimal regret. result Sharp memory-regret trade-off: O(1) memory suffices for Θ(T1/2) regret in B passes. This paper reviews methods to create compact neural networks for IoT applications.
problem Complex deep neural networks are costly and slow, hindering real-world deployment.
method Automatic synthesis of compact, accurate DNN/LSTM models.
result Compact neural networks reduce energy consumption, memory, and inference time.
Linear algebra approach for parallel deep learning models.
problem Training large DNNs in distributed environments.
method Linear algebraic approach to model parallelism.
result Manual development of backward operators for gradient-based training.
This paper presents a novel end-to-end methodology for enabling the deployment of low-error deep networks on microcontrollers. To fit the memory and computational limitations of resource-constrained edge-devices, we exploit mixed low-bitwidth compression, featuring 8, 4 or 2-bit uniform quantization, and we model the i…
New graph coarsening method preserves GNN message-passing signals.
problem Preserving GNN message-passing on coarsened graphs.
method Proposed a new oriented message-passing operation for coarsened graphs.
result Improved node classification results compared to naive methods.
PySAD offers a unified Python framework for efficient streaming anomaly detection.
problem Efficient anomaly detection in streaming data with strict constraints.
method Unified architecture with 17+ streaming algorithms, specialized components, and support for multiple learning paradigms.
result PySAD enables real-time processing with bounded memory and is compatible with other Python frameworks.