A new method compresses conditional distributions of labelled data.
problem No existing method directly compresses the conditional distribution of labelled data.
method Introduce Average Maximum Conditional Mean Discrepancy (AMCMD), derive a closed form estimator, and extend Kernel Herding (KH) to Average Conditional Kernel Herding (ACKH).
result Directly compressing conditional distributions outperforms joint distribution compression and greedy selection.
BDC compresses both sample size and dimensionality of large datasets.
problem Large datasets in both sample size and dimensionality.
method Two-stage framework using Decoded MMD, Reconstruction MMD, and Encoded MMD.
result BDC achieves comparable or superior performance with lower cost and higher compression rates.
Paper introduces a new adaptive gradient method with gradient compression for distributed training.
problem Communication overhead in distributed machine learning systems.
method Adaptive gradient method with gradient compression, scalable system BytePS-Compress.
result Convergence rate of O ( 1 / T ) \mathcal{O}(1/\sqrt{T}) O ( 1/ T ) for non-convex problems. Distributed sensors compress and send features to a fusion center for linear regression.
problem Efficiently compress and transmit features from distributed sensors to a fusion center under varying communication constraints.
method Designs a distributed and adaptive feature compression scheme using optimal quantizers and simple adaptive strategies.
result Demonstrates improved inference performance through simulated experiments.
Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suff…
New uncertainty principle limits compression in distributed learning, suggesting optimal methods.
problem Minimizing communication cost while maintaining message quality in distributed learning.
method Formalized uncertainty principle for compression operators, introduced Kashin compression.
result Explicit formula for Kashin compression's variance bound, dimension independent.
Compress++ speeds up distribution compression to near-linear time.
problem Accurately summarize a probability distribution using a small number of points efficiently.
method Introduces Compress++, a meta-procedure to speed up any thinning algorithm.
result Achieves n \sqrt{n} n points with O ( log n / n ) \mathcal{O}(\sqrt{\log n/n}) O ( log n / n ) integration error in O ( n log 3 n ) \mathcal{O}(n \log^3 n) O ( n log 3 n ) time and O ( n log 2 n ) \mathcal{O}( \sqrt{n} \log^2 n ) O ( n log 2 n ) space. This work compresses heavy-tailed weight matrices for tighter generalization bounds.
problem Empirical evidence linking heavy-tailed weight matrices to test set accuracy but lack of formal relationship with generalization bounds.
method Utilized the compression framework to show that heavy-tailed matrices can be compressed, resulting in sparse weight matrices.
result Demonstrated a non-vacuous generalization bound for compressed networks with heavy-tailed weight matrices.
Profile entropy measures learnability and compressibility of discrete distributions.
problem Understanding the learnability and compressibility of discrete distributions.
method Investigates profile entropy, showing its role in estimation, inference, and compression.
result Profile entropy is a fundamental measure unifying estimation, inference, and compression.
Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed gradient methods operating with staled and compressed gradients. Non-asymptotic bo…
New biased compression methods lead to faster convergence in distributed learning.
problem Improving convergence rates in distributed learning with biased compression.
method Study of three classes of biased compression operators in distributed learning.
result Biased compressors can lead to linear convergence rates in both single node and distributed settings.
A new method learns graph compression from data.
problem Graphs lack ordering, making conventional compression algorithms ineffective.
method Partition and Code framework: decompose, learn, encode.
result PnC achieves compression gains that grow with graph size.
New algorithm reduces communication in distributed learning by sharing compressed beliefs.
problem Efficiently learning from private data in a distributed setting with large hypothesis sets.
method Proposes a belief update rule for distributed cooperative learning with compressed (sparse or quantized) beliefs.
result Beliefs converge almost surely to optimal hypotheses with a linear concentration rate.
Artemis framework improves distributed learning with bidirectional compression and partial participation.
problem Learning in distributed or federated settings with communication constraints and device partial participation.
method Artemis framework using bidirectional compression, memory mechanism, and Polyak-Ruppert averaging.
result Fast rates of convergence (linear up to a threshold) under weak assumptions on stochastic gradients.
COMP-AMS optimizes distributed training with compressed gradients, achieving similar accuracy with less communication.
problem Efficiently training large-scale models in distributed environments with reduced communication costs.
method Distributed optimization framework using gradient averaging and adaptive AMSGrad, with gradient compression and error feedback.
result COMP-AMS achieves the same convergence rate and linear speedup as standard AMSGrad with less communication.
New compression schemes save communication in distributed mean estimation.
problem Efficiently compressing vectors for mean estimation in a limited communication setting.
method Collaborative compression schemes that exploit vector similarities without requiring known correlations.
result Analysis shows varying error types with vector similarity.
This letter proposes a sparse diffusion steepest-descent algorithm for one bit compressed sensing in wireless sensor networks. The approach exploits the diffusion strategy from distributed learning in the one bit compressed sensing framework. To estimate a common sparse vector cooperatively from only the sign of measur…
In this paper, we derive Hybrid, Bayesian and Marginalized Cramér-Rao lower bounds (HCRB, BCRB and MCRB) for the single and multiple measurement vector Sparse Bayesian Learning (SBL) problem of estimating compressible vectors and their prior distribution parameters. We assume the unknown vector to be drawn from a compr…
Paper improves communication in distributed optimization, reducing worker-to-server data exchanges.
problem Efficiency in server-to-worker communication in distributed optimization.
method MARINA-P, a novel downlink compression method using correlated compressors; M3, combining MARINA-P with uplink compression.
result MARINA-P achieves provably superior server-to-worker communication complexity with increasing number of workers.
Dirichlet pruning compresses neural networks by removing unimportant units.
problem Compressing large neural network models without sacrificing performance.
method Assigns Dirichlet distribution over network layers' units and uses variational inference to estimate parameters.
result Achieves state-of-the-art compression performance on larger architectures like VGG and ResNet.
Deep Neural Networks (DNNs) have recently been achieving state-of-the-art performance on a variety of computer vision related tasks. However, their computational cost limits their ability to be implemented in embedded systems with restricted resources or strict latency constraints. Model compression has therefore been …
Novel compression method preserves privacy while reducing communication costs.
problem Reducing communication costs in differential privacy mechanisms.
method Poisson private representation (PPR) for compressing and simulating local randomizers.
result Achieves compression within a logarithmic gap from theoretical lower bound.
New method reduces communication costs in distributed nonconvex optimization.
problem Large communication costs between central server and local workers in distributed learning.
method Communication-compressed AMSGrad for distributed nonconvex optimization.
result Converges to first-order stationary point with same iteration complexity as vanilla AMSGrad.
DeepCMC compresses CSI for massive MIMO systems, reducing overhead and improving performance.
problem High CSI overhead in massive MIMO systems limits spectral efficiency.
method Deep learning-based fully convolutional neural network with residual layers and entropy coding.
result DeepCMC outperforms state-of-the-art schemes in CSI reconstruction quality for the same compression rate.
Modern deep learning models are often trained in parallel over a collection of distributed machines to reduce training time. In such settings, communication of model updates among machines becomes a significant performance bottleneck and various lossy update compression techniques have been proposed to alleviate this p…
We propose and study the problem of distribution-preserving lossy compression. Motivated by recent advances in extreme image compression which allow to maintain artifact-free reconstructions even at very low bitrates, we propose to optimize the rate-distortion tradeoff under the constraint that the reconstructed sample…
CTT compresses samples to test distributions near-linearly, outperforming existing methods.
problem Efficiently testing distributions with high power and near-linear runtime.
method Sample compression followed by permutation testing.
result CTT achieves near-linear runtime while maintaining high statistical power.
New techniques improve distributed training with compressed gradients.
problem Gradient mismatch problem in local error feedback.
method Step-ahead error feedback and error averaging techniques.
result Our methods handle gradient mismatch and train faster than full-precision training.
Unified analysis of federated learning with compression for various data distributions.
problem Communication overhead in federated learning with unreliable or limited communication.
method Periodic compressed communication and local gradient tracking schemes.
result Sharp convergence rates for various objective functions and data distributions.
Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote …
Paper improves MIRACLE for faster, more robust neural network compression.
problem Efficiently compressing neural networks while maintaining performance.
method Introduces Mean-KL parameterization to constrain compression cost.
result Mean-KL parameterization leads to twice as fast convergence and more robust compression.
New method reduces FL communication cost by leveraging server's side information.
problem High communication cost in federated learning.
method Exploits server's side information to compress client updates.
result Up to 82 times smaller bitrate with higher accuracy.
Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. Th…
A new deep learning framework for efficient IoT data compression and inference.
problem Limited bandwidth and power resources in IoT sensors.
method Co-designed deep learning framework that maximizes sensing goal accuracy.
result Superior performance compared to benchmark models.
We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on …
Unified framework for distributed compressed SGD under ( L 0 , L 1 ) (L_0, L_1) ( L 0 , L 1 ) -smoothness.
problem Understanding the joint effect of batch noise, adaptivity, and compression in distributed stochastic optimization.
method Developed a unified theoretical framework using SDEs that incorporate curvature-dependent terms.
result Normalizing updates in DCSGD stabilizes convergence, with normalization degree determined by noise structure and landscape regularity.
CTE improves explanation estimation with less data and faster computation.
problem Inefficient and inaccurate explanation estimation in machine learning models.
method Distribution compression through kernel thinning to reduce sample size.
result CTE significantly improves accuracy and stability of explanation estimation.
New techniques save bits in image compression with upsampling.
problem Lack of context dependence in current image compression methods with upsampling.
method Simple, inexpensive techniques exploiting context to predict Laplace distribution parameters.
result Average savings of 0.645 bits per difference, up to 1.489 bits.
Unified strategy for efficient data compression and model estimation.
problem Limited interactive exploration and data interaction in linear model development and deployment.
method Conditionally sufficient statistics for optimal data compression and estimation of linear models.
result Linear models can be estimated from compressed data without loss of parameters or covariances.
Clapping reduces memory usage in distributed optimization by reusing data samples.
problem Significant communication overhead and impractical memory overhead in pipeline-parallel distributed optimization.
method Lazy sampling strategy to reuse data samples across steps, supporting convergence without unbiased gradient assumptions.
result Clapping achieves convergence in few-epoch or online training regimes without sample-size memory overhead.
EControl improves fast distributed optimization with compression and error control.
problem Stable convergence issues in distributed training with compression.
method Proposes EControl to regulate error compensation and prove fast convergence.
result Proves fast convergence for EControl in various convex settings without additional assumptions.
New compression methods handle biased input sequences for more accurate posterior summaries.
problem Handling biased input sequences for accurate posterior summaries.
method Stein kernel thinning, low-rank SKT, Stein recombination, Stein Cholesky.
result Achieves accurate posterior summaries with biased input sequences.
This paper connects Wasserstein distances to MMD norms for compressive statistical learning.
problem Comparing and controlling Wasserstein distances between probability distributions.
method Establishing conditions under which Wasserstein distances can be controlled by MMD norms.
result Introducing Wasserstein regularity for compressive statistical learning.
Paper improves compressed SGD to reach second-order stationary points.
problem Efficiently reaching second-order stationary points in distributed machine learning.
method Gradient compression and RandomK compressor to improve convergence to second-order stationary points.
result Compressed SGD reaches second-order stationary points with improved communication efficiency.
NeLLoC improves image compression with parallel decoding.
problem Image compression with OOD generalization.
method Local autoregressive model with parallel decoding.
result Significant gains in compression runtime.
IntSGD compresses SGD gradients without floats, converging as SGD.
problem Efficiently compressing stochastic gradients in distributed SGD.
method Adaptive integer compression of gradients, estimating scaling adaptively.
result IntSGD matches SGD's iteration complexity for convex and non-convex functions.
New findings show learnable distributions remain learnable even with noisy or adversarial perturbations.
problem Learning from perturbed samples in high-dimensional spaces.
method Developed a perturbation-quantization framework to analyze additive noise and adversarial corruption models.
result Sample compressible families remain learnable even under noisy or adversarial perturbations.
Network agents solve adaptive regression problems with compressed signals.
problem Solving regression problems in networks with communication constraints.
method ACTC diffusion strategy with randomized differential compression.
result Optimized resource allocation improves performance.