We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva's sparse distributed memory, it has a robust distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian updat…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Distributed Memory”
DAM with MRL improves relational reasoning in MANNs.
problem Limited performance of associative memory networks on complex relational reasoning tasks.
method Distributed Associative Memory architecture with Memory Refreshing Loss.
result Enhanced relation reasoning performance of MANNs on long temporal sequence data.
Heavy-tail driven by memoryq-fin.ST
We propose a stochastic process driven by memory effect with novel distributions including both exponential and leptokurtic heavy-tailed distributions. A class of distribution is analytically derived from the continuum limit of the discrete binary process with the renormalized auto-correlation and the closed form momen…
Research on long-range memory in financial and social systems using various models.
problem Understanding the nature of long-range memory in socioeconomic systems.
method Various Markov processes including point processes, stochastic differential equations, and agent-based models.
result New estimators of self-similarity and long-range memory for non-Gaussian systems are needed.
NEST optimizes deep learning training by placing devices efficiently across networks and memory.
problem Inefficient device placement in distributed deep learning leads to high communication and memory overhead.
method NEST uses network-, compute-, and memory-aware dynamic programming to optimize device placement.
result NEST achieves up to 2.43 times higher throughput and better memory efficiency.
In distributed reinforcement learning, it is common to exchange the experience memory of each agent and thereby collectively train their local models. The experience memory, however, contains all the preceding state observations and their corresponding policies of the host agent, which may violate the privacy of the ag…
Extends DAMs to Gaussian distributions for efficient pattern storage and retrieval.
problem Limited storage capacity and retrieval methods for non-vector pattern representations.
method Introduces a log-sum-exp energy function over Gaussian distributions, using optimal transport maps for retrieval dynamics.
result Proves exponential storage capacity and provides quantitative retrieval guarantees.
We propose a stochastic process driven by the memory effect with novel distributions which include both exponential and leptokurtic heavy-tailed distributions. A class of the distributions is analytically derived from the continuum limit of the discrete binary process with the renormalized auto-correlation. The moment …
We investigate the probability distribution of the volatility return intervals for the Chinese stock market. We rescale both the probability distribution and the volatility return intervals as to obtain a uniform scaling curve for different threshold value . The scali…
Enhanced Hopfield model boosts memory retrieval capacity.
problem Memory retrieval in modern Hopfield models with limited capacity.
method Introduces a learnable feature map transforming energy function into kernel space, minimizing separation loss for uniform memory distribution.
result Significant reduction in metastable states, enhancing memory capacity and retrieval accuracy.
We study distribution testing with communication and memory constraints in the following computational models: (1) The {\em one-pass streaming model} where the goal is to minimize the sample complexity of the protocol subject to a memory constraint, and (2) A {\em distributed model} where the data samples reside at mul…
Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoTstat.ML
Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. Th…
We study the problem of identifying correlations in multivariate data, under information constraints: Either on the amount of memory that can be used by the algorithm, or the amount of communication when the data is distributed across several machines. We prove a tight trade-off between the memory/communication complex…
When training large machine learning models with many variables or parameters, a single machine is often inadequate since the model may be too large to fit in memory, while training can take a long time even with stochastic updates. A natural recourse is to turn to distributed cluster computing, in order to harness add…
Communication-Efficient, 2D Parallel Stochastic Gradient Descent for Distributed-Memory Optimizationcs.DC
HybridSGD improves SGD performance by balancing computation and communication.
problem Limited scalability and performance of SGD due to communication costs.
method 2D parallel SGD method (HybridSGD) that trades off between 1D -step SGD and 1D Federated SGD (FedAvg).
result HybridSGD achieves better convergence than FedAvg at similar processor scales and up to 121x speedup over FedAvg.
mGRN improves multivariate time series prediction by managing marginal and joint memories.
problem Extracting dependencies in multivariate sequential data with strong serial and cross-sectional dependencies.
method Developed a novel recurrent network architecture, Memory-Gated Recurrent Networks (mGRN), with gates for marginal and joint memories.
result mGRN consistently outperforms state-of-the-art architectures on various public datasets.
Survey on statistical inference under memory constraints.
problem Effect of memory limitations on statistical inference performance.
method Review of state-of-the-art in several canonical problems.
result Identification of fundamental building blocks and useful techniques.
Memory-based models can learn to approximate Bayes-optimal predictors for non-stationary data.
problem Learning from non-stationary data with unobserved switching points.
method Memory-based neural models, including Transformers, LSTMs, and RNNs, trained to minimize log loss.
result Memory-based models can accurately approximate known Bayes-optimal algorithms and perform Bayesian inference over latent switching points.
Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memorycs.NE
New memory allocation scheme improves image generation performance.
problem Improving episodic and semantic memory representation in neural networks.
method Developed a hierarchical latent variable model with differentiable, locally block allocated latent memory.
result Improved conditional likelihood values on various datasets.
This paper tackles federated incremental learning with dynamic memory allocation for improved model performance in non-IID data.
problem Catastrophic forgetting in federated healthcare systems with non-IID data.
method Dynamic memory allocation strategy based on data replay mechanism.
result Significant performance improvements in medical image datasets compared to baseline models.
Gradient boosted decision trees (GBDT) is the leading algorithm for many commercial and academic data applications. We give a deep analysis of this algorithm, especially the histogram technique, which is a basis for the regulized distribution with compact support. We present three new modifications. 1) Share memory tec…
Bayesian histograms achieve optimal distribution estimation with minimal memory usage.
problem Efficiently estimating distributions with minimal memory footprint.
method Bayesian histograms for distribution estimation under Wasserstein distance.
result Bayesian histograms require fewer bins to achieve minimax optimality, reducing memory usage by a polynomial factor.
Distributed machine learning (ML) can bring more computational resources to bear than single-machine learning, thus enabling reductions in training time. Distributed learning partitions models and data over many machines, allowing model and dataset sizes beyond the available compute power and memory of a single machine…
Clapping reduces memory usage in distributed optimization by reusing data samples.
problem Significant communication overhead and impractical memory overhead in pipeline-parallel distributed optimization.
method Lazy sampling strategy to reuse data samples across steps, supporting convergence without unbiased gradient assumptions.
result Clapping achieves convergence in few-epoch or online training regimes without sample-size memory overhead.
Distributed learning with random features and gradient descent improves performance and reduces memory usage.
problem Improving generalization in decentralized learning with limited memory.
method Distributed Gradient Descent with Random Features and Implicit Regularization.
result High probability bounds on predictive performance with optimal statistical rates.
The Unlearning algorithm improves neural network performance in memory tasks.
problem Improving neural network performance in memory tasks.
method Simplified Unlearning algorithm, structured training data, and new regularization technique for Boltzmann Machines.
result The Unlearning rule outperforms traditional Boltzmann-Machine learning in learning hidden probability distributions.
Paper proposes a GPU-based system for training massive deep learning models in ads systems.
problem Training massive deep learning models with terabyte-scale parameters in ads systems.
method Hierarchical GPU parameter server with 3-layer storage (GPU High-Bandwidth Memory, CPU main memory, SSD).
result 4-node hierarchical GPU parameter server trains a model 2X faster than a 150-node in-memory system.
MIP framework improves urban flow prediction by adapting to distribution shifts.
problem Distribution shifts in urban flow data make prediction models unreliable.
method Memory-enhanced Invariant Prompt learning with learnable memory bank.
result MIP ensures robust predictions by focusing on invariant features.
A theory for long-memory in supply and demandcond-mat.other
Recent empirical studies have demonstrated long-memory in the signs of orders to buy or sell in financial markets [2, 19]. We show how this can be caused by delays in market clearing. Under the common practice of order splitting, large orders are broken up into pieces and executed incrementally. If the size of such lar…
Combinatorial dimensions play an important role in the theory of machine learning. For example, VC dimension characterizes PAC learning, SQ dimension characterizes weak learning with statistical queries, and Littlestone dimension characterizes online learning. In this paper we aim to develop combinatorial dimensions th…
Convolutional Neural Networks (CNN) are being actively explored for safety-critical applications such as autonomous vehicles and aerospace, where it is essential to ensure the reliability of inference results in the presence of possible memory faults. Traditional methods such as error correction codes (ECC) and Triple …
A new method uses burst and inter-burst duration to test long-range memory in financial markets.
problem Varying results from long-range memory estimators in financial markets.
method Burst and inter-burst duration statistical analysis of limit order book data.
result The new method provides a more reliable evaluation of the Hurst exponent.
Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as they process data, requiring very low learning rates. If the training distribution shifts, the network is slow to adapt, and when it does adapt…
A central challenge faced by memory systems is the robust retrieval of a stored pattern in the presence of interference due to other stored patterns and noise. A theoretically well-founded solution to robust retrieval is given by attractor dynamics, which iteratively clean up patterns during recall. However, incorporat…
PipeDream-2BW accelerates large model training by 20x with minimal memory usage.
problem Training large models requires memory beyond single accelerator capacity.
method Pipeline parallelism, weight gradient coalescing, double buffering.
result Accelerates large model training by up to 20x.
We propose two optimization techniques to minimize memory usage and computation while meeting system timing constraints for real-time classification in wearable systems. Our method derives a hierarchical classifier structure for Support Vector Machine (SVM) in order to reduce the amount of computations, based on the pr…
We propose a general method to obtain approximation of the first passage time distribution for the birth-death processes. We rely on the general properties of birth-death processes, Keilson's theorem and the concept of Riemann sum to obtain closed-form expressions. We apply the method to the three selected birth-death …
Much sequential data exhibits highly non-uniform information distribution. This cannot be correctly modeled by traditional Long Short-Term Memory (LSTM). To address that, recent works have extended LSTM by adding more activations between adjacent inputs. However, the approaches often use a fixed depth, which is at the …
Severe constraints on memory and computation characterizing the Internet-of-Things (IoT) units may prevent the execution of Deep Learning (DL)-based solutions, which typically demand large memory and high processing load. In order to support a real-time execution of the considered DL model at the IoT unit level, DL sol…
The ability to generalize quickly from few observations is crucial for intelligent systems. In this paper we introduce APL, an algorithm that approximates probability distributions by remembering the most surprising observations it has encountered. These past observations are recalled from an external memory module and…
Unified treatment of RC in stochastic and deterministic settings.
problem Understanding and generalizing reservoir computing in both deterministic and stochastic contexts.
method Investigation of state-space systems, analysis of fading memory and solution stability, introduction of stochastic echo states.
result Generality of fading memory and solution stability in state-space systems, even without the echo state property.
Sequential learning of tasks using gradient descent leads to an unremitting decline in the accuracy of tasks for which training data is no longer available, termed catastrophic forgetting. Generative models have been explored as a means to approximate the distribution of old tasks and bypass storage of real data. Here …
Variational Bayesian Sequence-to-Sequence Networks for Memory-Efficient Sign Language Translationstat.ML
A new memory-efficient sign language translation model reduces weight usage.
problem Memory constraints in real-time sign language translation.
method Variational Bayesian sequence-to-sequence network with Gaussian posterior and Indian Buffet Process prior.
result The proposed model achieves substantial weight compression without compromising performance.
Earlier we proposed the stochastic point process model, which reproduces a variety of self-affine time series exhibiting power spectral density S(f) scaling as power of the frequency f and derived a stochastic differential equation with the same long range memory properties. Here we present a stochastic differential eq…
New methods improve memory efficiency for sampling from complex distributions.
problem Sampling from complex unnormalized distributions over discrete domains.
method Two novel training methods for discrete diffusion samplers.
result Achieve state-of-the-art results in unsupervised combinatorial optimization.
New method combines long-memory reservoirs for accurate dengue forecasting from short data.
problem Accurate dengue forecasting from short, noisy, non-stationary, and nonlinear data.
method Fractional ESN and Wavelet ESN frameworks integrating long-term memory.
result fESN and wESN outperform baselines in multiple dengue datasets and forecasting horizons.
On non-markovian nature of stock tradingcond-mat.other
Using a relationship between the moments of the probability distribution of times between the two consecutive trades (intertrade time distribution) and the moments of the distribution of a daily number of trades we show, that the underlying point process generating times of the trades is an essentially non-markovian lo…
Anomalous diffusion and long-range memory in the scaled voter modelcond-mat.stat-mech
Analyzed a generalized voter model with power-law herding intensity, revealing anomalous diffusion and long-range memory.
problem Anomalous diffusion and long-range memory in a generalized voter model.
method Derived analytical expressions for moments and first passage time distribution, confirmed numerically.
result The model exhibits long-range memory indicators despite being a Markov model.