arXiv research

PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.

Flora uses random projections to achieve high-rank updates with low memory usage.

We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on …

New biased compression methods lead to faster convergence in distributed learning.

A new method improves communication efficiency in distributed learning.

We consider lossless compression based on statistical data modeling followed by prediction-based encoding, where an accurate statistical model for the input data leads to substantial improvements in compression. We propose DZip, a general-purpose compressor for sequential data that exploits the well-known modeling capa…

New methods improve distributed optimization on non-iid data.

Paper designs a lossy compression method for lossless prediction.

The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main te…

Estimates rate-distortion function for large datasets using neural networks.

Paper improves communication in distributed optimization, reducing worker-to-server data exchanges.

New method reduces distributed non-convex optimization rounds and bits.

Paper improves compressed SGD to reach second-order stationary points.

CSER improves SGD efficiency by resetting errors and partial synchronization.

We present a new similarity measure based on information theoretic measures which is superior than Normalized Compression Distance for clustering problems and inherits the useful properties of conditional Kolmogorov complexity. We show that Normalized Compression Dictionary Size and Normalized Compression Dictionary En…

An algorithm tackles low-rank linear bandit problems with improved regret bounds.

Review of algorithms for linear system approximations.

Algorithm recovers multiple low-rank matrices from unlabeled data.

EF21 improves convergence in distributed machine learning models.

New algorithm learns low-rank matrices with linear number of samples.

Paper establishes limits for accurately estimating low-rank matrices from noisy, non-linear data.

Novel method for efficient low-rank matrix estimation and bandit algorithms.

Paper introduces G-LowTESTR for efficient tensor bandits.

Matrix completion works well for smooth non-linear structures, even without low-rank assumptions.

In this paper, we present NESTA, a specialized Neural engine that significantly accelerates the computation of convolution layers in a deep convolutional neural network, while reducing the computational energy. NESTA reformats Convolutions into $3 \times 3$ batches and uses a hierarchy of Hamming Weight Compressors to …

This work provides closed-form solutions and minimum achievable errors for a large class of low-rank approximation problems in Hilbert spaces. The proposed theorem generalizes to the case of bounded linear operators the previous results obtained in the finite dimensional case for the Frobenius norm. The theorem provide…

We study the problem of estimating low-rank matrices from linear measurements (a.k.a., matrix sensing) through nonconvex optimization. We propose an efficient stochastic variance reduced gradient descent algorithm to solve a nonconvex optimization problem of matrix sensing. Our algorithm is applicable to both noisy and…

Paper optimizes industrial refrigeration using adaptive exploration.

LoLCATs improves linearized LLM quality with less memory and compute.

Solving symmetric positive definite linear problems is a fundamental computational task in machine learning. The exact solution, famously, is cubicly expensive in the size of the matrix. To alleviate this problem, several linear-time approximations, such as spectral and inducing-point methods, have been suggested and a…

Paper develops methods for non-quadratic loss low-rank matrix recovery.

Gradient descent achieves exact linear convergence rate for symmetric matrix completion.

New findings show DNC is not optimal for deep models, revealing a low-rank bias.

Develops efficient method for updating models with small data changes.

Gradient descent solves asymmetric low-rank matrix sensing without balancing.

Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote …

We develop latent variable models for Bayesian learning based low-rank matrix completion and reconstruction from linear measurements. For under-determined systems, the developed methods are shown to reconstruct low-rank matrices when neither the rank nor the noise power is known a-priori. We derive relations between th…

Unified approach tackles high-dimensional tensor bandits with convex optimization and weakly decomposable regularizers.

We propose a unified framework to solve general low-rank plus sparse matrix recovery problems based on matrix factorization, which covers a broad family of objective functions satisfying the restricted strong convexity and smoothness conditions. Based on projected gradient descent and the double thresholding operator, …

Unified framework for statistical inference of low-rank tensors.

Faster GW alignment for incomparable point clouds via low-rank couplings.

FLAMBE tackles RL in low rank MDPs by learning features.

We develop a communication-efficient distributed learning algorithm that is robust against Byzantine worker machines. We propose and analyze a distributed gradient-descent algorithm that performs a simple thresholding based on gradient norms to mitigate Byzantine failures. We show the (statistical) error-rate of our al…

Paper improves sample complexity for reward-free RL in low-rank MDPs.

ScaledGD algorithm estimates low-rank tensors efficiently from corrupted data.

Study algebraic invariants from lightning self-attention models.

New algorithms solve dense linear systems with low-rank structure efficiently.

High-dimensional representations often have a lower dimensional underlying structure. This is particularly the case in many decision making settings. For example, when the representation of actions is generated from a deep neural network, it is reasonable to expect a low-rank structure whereas conventional structures l…