arXiv research

New memory-query tradeoffs for convex optimization algorithms.

Memory-constrained algorithms need superlinear memory for efficient convex optimization.

A technique identifies memoryless algorithms approximating memory-dependent optimization methods.

New analysis tightens memory capacity of Hopfield models using spherical codes.

This work analyzes and optimizes memory and compute costs of learned optimizers.

Quadratic memory is essential for optimal convex optimization queries.

State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption. This has created a recent demand for memory-efficient optimizers. To this end, we investigate the limits and performance tradeoffs of memory-efficient adaptively preconditioned gradient methods.…

Paper proposes efficient optimizers for large language models with fast convergence and low memory usage.

We note that known methods achieving the optimal oracle complexity for first order convex optimization require quadratic memory, and ask whether this is necessary, and more broadly seek to characterize the minimax number of first order queries required to optimize a convex Lipschitz function subject to a memory constra…

Memory-Augmented Meta-Optimization improves cold-start recommendation.

Training machine learning models on mobile devices has the potential of improving both privacy and accuracy of the models. However, one of the major obstacles to achieving this goal is the memory limitation of mobile devices. Reducing training memory enables models with high-dimensional weight matrices, like automatic …

TASO optimizes CNN models for memory-constrained devices.

New algorithms for constrained online optimization with memory and predictions.

Study on memory effects in RNNs learning temporal data.

Second-order optimizers retain residual information after data deletion, affecting machine unlearning.

Serenity optimizes neural network execution for edge devices by scheduling with optimal memory footprint.

LDAdam optimizes large models with low memory by adapting to lower-dimensional subspaces.

Optimal CL requires perfect memory and is NP-hard.

Optimal linear contracts are possible even with memory in Gaussian settings.

Memory-efficient algorithm reduces variance in off-policy RL.

Sketchy reduces memory and compute requirements for adaptive regularization in deep learning.

AlphaGrad optimizes memory usage in RL algorithms by normalizing gradients.

New algorithms solve convex optimization problems with limited memory.

This paper presents competitive algorithms for a novel class of online optimization problems with memory. We consider a setting where the learner seeks to minimize the sum of a hitting cost and a switching cost that depends on the previous $p$ decisions. This setting generalizes Smoothed Online Convex Optimization. The…

A new memory-efficient Adam variant reduces second moments when feasible.

OpTorch optimizes deep learning for resource-limited environments.

Deep Neural Networks(DNNs) require huge GPU memory when training on modern image/video databases. Unfortunately, the GPU memory is physically finite, which limits the image resolutions and batch sizes that could be used in training for better DNN performance. Unlike solutions that require physically upgrade GPUs, the G…

NEST optimizes deep learning training by placing devices efficiently across networks and memory.

Memory-based models can learn to approximate Bayes-optimal predictors for non-stationary data.

This study examines memory effects in S&P500 market correlations using Langevin models.

We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva's sparse distributed memory, it has a robust distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian updat…

New algorithms achieve optimal regret in sliding window model with limited memory.

We propose two optimization techniques to minimize memory usage and computation while meeting system timing constraints for real-time classification in wearable systems. Our method derives a hierarchical classifier structure for Support Vector Machine (SVM) in order to reduce the amount of computations, based on the pr…

POET enables large neural network training on tiny devices with reduced energy.

SMGD trains low-bit neural networks with memory constraints.

We present Memory Augmented Policy Optimization (MAPO), a simple and novel way to leverage a memory buffer of promising trajectories to reduce the variance of policy gradient estimate. MAPO is applicable to deterministic environments with discrete actions, such as structured prediction and combinatorial optimization ta…

Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize t…

This paper optimizes SMPC for neural network inference, reducing memory and time.

Efficiently discovers Bayesian network structure with reduced memory usage.

New method improves FO-BLO convergence without increasing memory or time complexity.

New framework captures long-term decision dependence in online learning.

Gradient descent is efficient for solving feasibility problems with minimal memory and queries.

In this paper, the Kyle model of insider trading is extended by characterizing the trading volume with long memory and allowing the noise trading volatility to follow a general stochastic process. Under this newly revised model, the equilibrium conditions are determined, with which the optimal insider trading strategy,…

HiPPO framework optimizes memory compression for sequential data.

A new method reduces memory requirements for sorting high-dimensional data.

Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling. However, these methods maintain second-order statistics for each parameter, thus introducing significant memory overheads that restrict the size of the model b…

The paper studies scaling laws for associative memory mechanisms.

The key to a Transformer model is the self-attention mechanism, which allows the model to analyze an entire sequence in a computationally efficient manner. Recent work has suggested the possibility that general attention mechanisms used by RNNs could be replaced by active-memory mechanisms. In this work, we evaluate wh…