arXiv research

Memory-efficient algorithm reduces variance in off-policy RL.

New algorithm improves PPS for multi-object matching.

SketchOGD improves memory efficiency for continual learning.

Incremental versions of batch algorithms are often desired, for increased time efficiency in the streaming data setting, or increased memory efficiency in general. In this paper we present a novel algorithm for incremental kernel PCA, based on rank one updates to the eigendecomposition of the kernel matrix, which is mo…

XLA compiler extension improves memory efficiency for machine learning.

New algorithm learns stable LDSs with lower error and better control performance.

Memory-efficient learning for large-scale imaging systems.

SINGD improves KFAC for memory-efficiency and stability in low-precision training.

A new memory-efficient sign language translation model reduces weight usage.

In this paper, we study the generalization properties of online learning based stochastic methods for supervised learning problems where the loss function is dependent on more than one training sample (e.g., metric learning, ranking). We present a generic decoupling technique that enables us to provide Rademacher compl…

New algorithms delete user data from machine learning models efficiently.

We study the statistical and computational aspects of kernel principal component analysis using random Fourier features and show that under mild assumptions, $O(\sqrt{n} \log n)$ features suffices to achieve $O(1/ε^2)$ sample complexity. Furthermore, we give a memory efficient streaming algorithm based on classical Oja…

We present a novel algorithm, Westfall-Young light, for detecting patterns, such as itemsets and subgraphs, which are statistically significantly enriched in one of two classes. Our method corrects rigorously for multiple hypothesis testing and correlations between patterns through the Westfall-Young permutation proced…

New fair PCA method using streaming algorithms with statistical guarantees.

A new algorithm reduces memory usage for long token attention in streaming applications.

A new algorithm reduces memory and computational needs for reinforcement learning.

Improved memory efficiency for deep neural networks.

Lion optimizer performs well in training AI models with memory efficiency.

Study proposes memory-efficient backpropagation for linear layers in neural networks.

AlphaGrad optimizes memory usage in RL algorithms by normalizing gradients.

A new RNN model tackles long-time dependencies with fast, invertible, and memory-efficient hidden states.

NeuZip compresses neural network weights to save memory during training and inference.

Paper proposes efficient optimizers for large language models with fast convergence and low memory usage.

We introduce a new sequential Monte Carlo algorithm we call the particle cascade. The particle cascade is an asynchronous, anytime alternative to traditional particle filtering algorithms. It uses no barrier synchronizations which leads to improved particle throughput and memory efficiency. It is an anytime algorithm i…

Efficient sampling reduces memory usage for Minimax distance analysis.

DPZero fine-tunes large models privately without backpropagation.

PipeDream-2BW accelerates large model training by 20x with minimal memory usage.

Stochastic variational inference for collapsed models has recently been successfully applied to large scale topic modelling. In this paper, we propose a stochastic collapsed variational inference algorithm for hidden Markov models, in a sequential data setting. Given a collapsed hidden Markov Model, we break its long M…

Quantum-assisted VAE improves similarity search in high-dimensional datasets.

The class of Gaussian Process (GP) methods for Temporal Difference learning has shown promise for data-efficient model-free Reinforcement Learning. In this paper, we consider a recent variant of the GP-SARSA algorithm, called Sparse Pseudo-input Gaussian Process SARSA (SPGP-SARSA), and derive recursive formulas for its…

We show that a simple modification of the 1-nearest neighbor classifier yields a strongly Bayes consistent learner. Prior to this work, the only strongly Bayes consistent proximity-based method was the k-nearest neighbor classifier, for k growing appropriately with sample size. We will argue that a margin-regularized 1…

Enhances neural architecture search efficiency and prevents performance collapse.

A new memory-efficient Adam variant reduces second moments when feasible.

Improves memory efficiency for meta-learning with large images.

In supervised machine learning, an agent is typically trained once and then deployed. While this works well for static settings, robots often operate in changing environments and must quickly learn new things from data streams. In this paradigm, known as streaming learning, a learner is trained online, in a single pass…

State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption. This has created a recent demand for memory-efficient optimizers. To this end, we investigate the limits and performance tradeoffs of memory-efficient adaptively preconditioned gradient methods.…

In this paper, we resolve many of the key algorithmic questions regarding robustness, memory efficiency, and differential privacy of tensor decomposition. We propose simple variants of the tensor power method which enjoy these strong properties. We present the first guarantees for online tensor power method which has a…

We show that fundamental learning tasks, such as finding an approximate linear separator or linear regression, require memory at least \emph{quadratic} in the dimension, in a natural streaming setting. This implies that such problems cannot be solved (at least in this setting) by scalable memory-efficient streaming alg…

New methods improve memory efficiency for sampling from complex distributions.

Exponential smoothers are a simple and memory efficient way to compute running averages of time series. Here we define and describe practical properties of exponential smoothers for signals observed at constant and variable intervals.

New CUSUM method detects changes in Hawkes networks efficiently.

A traditional artificial neural network (ANN) is normally trained slowly by a gradient descent algorithm, such as the backpropagation algorithm, since a large number of hyperparameters of the ANN need to be fine-tuned with many training epochs. Since a large number of hyperparameters of a deep neural network, such as a…

Bayesian histograms achieve optimal distribution estimation with minimal memory usage.

PCA reduces language model embeddings to improve speed and memory efficiency.

Metagenomic studies have increasingly utilized sequencing technologies in order to analyze DNA fragments found in environmental samples.One important step in this analysis is the taxonomic classification of the DNA fragments. Conventional read classification methods require large databases and vast amounts of memory to…

CICLAD efficiently mines frequent closed itemsets from data streams with minimal memory usage.

The adjoint sensitivity method scalably computes gradients of solutions to ordinary differential equations. We generalize this method to stochastic differential equations, allowing time-efficient and constant-memory computation of gradients with high-order adaptive solvers. Specifically, we derive a stochastic differen…

New algorithm speeds up cluster-based compressive sensing tasks.