arXiv research

Paper improves compressed SGD to reach second-order stationary points.

A new method improves communication efficiency in distributed learning.

We consider lossless compression based on statistical data modeling followed by prediction-based encoding, where an accurate statistical model for the input data leads to substantial improvements in compression. We propose DZip, a general-purpose compressor for sequential data that exploits the well-known modeling capa…

PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.

Paper designs a lossy compression method for lossless prediction.

New biased compression methods lead to faster convergence in distributed learning.

New methods improve distributed optimization on non-iid data.

Estimates rate-distortion function for large datasets using neural networks.

Paper improves communication in distributed optimization, reducing worker-to-server data exchanges.

New method reduces distributed non-convex optimization rounds and bits.

CSER improves SGD efficiency by resetting errors and partial synchronization.

We present a new similarity measure based on information theoretic measures which is superior than Normalized Compression Distance for clustering problems and inherits the useful properties of conditional Kolmogorov complexity. We show that Normalized Compression Dictionary Size and Normalized Compression Dictionary En…

In this paper, we present NESTA, a specialized Neural engine that significantly accelerates the computation of convolution layers in a deep convolutional neural network, while reducing the computational energy. NESTA reformats Convolutions into $3 \times 3$ batches and uses a hierarchy of Hamming Weight Compressors to …

Paper optimizes industrial refrigeration using adaptive exploration.

Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote …

We develop a communication-efficient distributed learning algorithm that is robust against Byzantine worker machines. We propose and analyze a distributed gradient-descent algorithm that performs a simple thresholding based on gradient norms to mitigate Byzantine failures. We show the (statistical) error-rate of our al…

A novel deep probabilistic model for dynamic systems forecasting.

Paper analyzes and compares ELF algorithms for federated learning.

The family of image visibility graphs (IVGs) have been recently introduced as simple algorithms by which scalar fields can be mapped into graphs. Here we explore the usefulness of such operator in the scenario of image processing and image classification. We demonstrate that the link architecture of the image visibilit…

TopoFisher learns topological summaries by maximizing Fisher information, improving parameter efficiency and inference quality.

EF21 improves convergence in distributed machine learning models.

We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on …

Flora uses random projections to achieve high-rank updates with low memory usage.

Bayesian method for estimating quantile sets efficiently.

Neural NCD reveals LLMs don't compress well for classification.

Global-QSGD accelerates distributed training by up to 3.51%.

This paper explores the potential of Lagrangian duality for learning applications that feature complex constraints. Such constraints arise in many science and engineering domains, where the task amounts to learning optimization problems which must be solved repeatedly and include hard physical and operational constrain…

Safe-EF improves federated learning for non-smooth, constrained optimization.

New algorithm improves distributed SGD with random sparsification for better convergence and generalization.

Proposes Stochastic-Sign SGD for federated learning with theoretical guarantees.

With the rapid growth of data, distributed momentum stochastic gradient descent~(DMSGD) has been widely used in distributed learning, especially for training large-scale deep models. Due to the latency and limited bandwidth of the network, communication has become the bottleneck of distributed learning. Communication c…

LLMs compress financial texts, but distort decision-making.

The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main te…

New compression schemes save communication in distributed mean estimation.

EControl improves fast distributed optimization with compression and error control.

SA-PEF improves federated learning efficiency by correcting gradient mismatches.

In order to mitigate the high communication cost in distributed and federated learning, various vector compression schemes, such as quantization, sparsification and dithering, have become very popular. In designing a compression method, one aims to communicate as few bits as possible, which minimizes the cost per commu…

New methods reduce communication in distributed training for variational inequalities.

A new method learns graph compression from data.

A framework infers causal direction from symbolic sequences using compression measures.

MARINA-P improves non-smooth federated optimization with adaptive stepsizes.

Two algorithms improve federated learning efficiency and resilience.

Midicoth compresses online probability estimates by correcting prior smoothing biases.

This book presents a methodology and philosophy of empirical science based on large scale lossless data compression. In this view a theory is scientific if it can be used to build a data compression program, and it is valuable if it can compress a standard benchmark database to a small size, taking into account the len…