Gradient descent with compressed iterates reduces model size for federated learning.
problem Efficiently compressing model sizes in federated learning.
method Gradient descent with iterates compressed using lossy randomized techniques.
result Improves model compression and aggregation in federated learning.
Paper proposes IIQ for compressing embedding vectors.
problem Memory issues in representing large vocabularies.
method Isotropic iterative quantization (IIQ) for binary compression.
result More than 30x compression ratio with comparable performance.
Iterative AutoML improves ASR model compression by 5x without WER degradation.
problem Challenges in achieving high compression levels without degrading ASR performance.
method Iterative AutoML-based Low Rank Factorization (LRF) approach.
result Achieved over 5x compression without WER degradation.
Recurrent iterated function systems (RIFSs) are improvements of iterated function systems (IFSs) using elements of the theory of Marcovian stochastic processes which can produce more natural looking images. We construct new RIFSs consisting substantially of a vertical contraction factor function and nonlinear transform…
The paper uses information geometry to analyze model compression techniques, focusing on operator factorization.
problem Efficiently compressing deep learning models for resource-constrained devices.
method Information geometry applied to model compression, focusing on operator factorization.
result Iterative methods are crucial for fine-tuning models, especially when compression ratios are fixed.
The low-rank tensor approximation is very promising for the compression of deep neural networks. We propose a new simple and efficient iterative approach, which alternates low-rank factorization with a smart rank selection and fine-tuning. We demonstrate the efficiency of our method comparing to non-iterative ones. Our…
Efficiently compress pretrained models using RSI for improved predictive accuracy.
problem Efficiently compressing large pretrained models for practical deployment.
method Randomized subspace iteration (RSI) for low-rank approximation of pretrained models.
result RSI achieves near-optimal approximation quality and outperforms RSVD in predictive accuracy.
New method improves image compression using bits-back coding.
problem Lossy image compression with deep latent variable models.
method Iterative inference, stochastic annealing, bits-back coding.
result New state-of-the-art performance on lossy image compression.
Model compression has gained a lot of attention due to its ability to reduce hardware resource requirements significantly while maintaining accuracy of DNNs. Model compression is especially useful for memory-intensive recurrent neural networks because smaller memory footprint is crucial not only for reducing storage re…
Multiple gossip steps improve decentralized optimization convergence.
problem Efficiently optimizing large-scale machine learning models with limited communication.
method Integrates multiple gossip steps between gradient descent iterations in compressed decentralized optimization.
result Convergence to within ε of the optimal value for smooth non-convex objectives.
PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.
problem Communication bottleneck in decentralized deep learning models.
method Low-rank linear compressors applied on model differences using power iteration steps.
result Asymptotically independent of network and compression, faster convergence, and comparable performance to tuned compression algorithms.
Proposes a link between randomness and compression in deep learning.
problem Improving efficiency in deep learning training.
method Introduces a novel tomographic compression framework called Dual Tomographic Compression (DTC).
result Demonstrates high correlation between learning performance and Gibbs entropy over compression ratios.
We adopt data structure in the form of cover trees and iteratively apply approximate nearest neighbour (ANN) searches for fast compressed sensing reconstruction of signals living on discrete smooth manifolds. Levering on the recent stability results for the inexact Iterative Projected Gradient (IPG) algorithm and by us…
New algorithm speeds up cluster-based compressive sensing tasks.
problem Efficiently solving multiple compressive sensing tasks with shared information.
method Combines Monte Carlo sampling with iterative linear solvers to avoid explicit covariance matrix computation.
result Up to thousands of times faster and orders of magnitude more memory-efficient compared to existing methods.
Algorithm estimates principal eigenvector with adaptive sensing, improving over non-adaptive methods.
problem Estimating principal eigenvector with limited scalar measurements.
method Compressed variant of Oja's algorithm using two adaptive measurements per sample.
result Convergence rate of O(λ1λ2d2/(Δ2t)) after t iterations, matching information-theoretic lower bound. New SGD variant makes neural networks compressible without assumptions.
problem Improving neural network compressibility without strong assumptions.
method Introducing heavy-tailed noise to SGD iterates.
result Compressible outputs with high probability for any compression rate.
Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed gradient methods operating with staled and compressed gradients. Non-asymptotic bo…
Simple iterative method reduces deep network size significantly.
problem Overparameterization of deep neural networks in compute-limited systems.
method Hybrid approach combining single shot pruning and Lottery-Ticket methods.
result State-of-the-art compression achieved with improved test accuracy and compression ratio.
Existing high-performance deep learning models require very intensive computing. For this reason, it is difficult to embed a deep learning model into a system with limited resources. In this paper, we propose the novel idea of the network compression as a method to solve this limitation. The principle of this idea is t…
Paper proposes a hybrid model-based and data-driven approach for one-bit compressive autoencoding.
problem Designing efficient one-bit compressive autoencoding models for complex systems.
method Hybrid model-based and data-driven methodology for one-bit sparse signal recovery.
result Significant improvement in one-bit compressive autoencoding compared to state-of-the-art algorithms.
New method reduces PDE model parameters by 30% with sparsity.
problem Redundant parameters in neural network projections.
method Bregman iterations for sparsity, POD compression, bias propagation.
result 30% fewer parameters with similar accuracy.
We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on …
Binary Iterative Hard Thresholding converges with optimal number of 1-bit measurements.
problem Recovering sparse signals from 1-bit compressed measurements.
method Binary Iterative Hard Thresholding (BIHT) algorithm.
result BIHT converges with only O(k/ε) measurements, optimal for recovery.
LASER compresses recursive model activations by exploiting their low-dimensional structure.
problem Understanding and optimizing the geometric structure of recursive reasoning trajectories.
method Dynamic low-rank basis tracking via matrix-free subspace tracking with a fidelity-triggered reset mechanism.
result Recursive activations occupy a linear, low-dimensional subspace that can be compressed efficiently.
Paper proposes a framework and algorithm for model compression in neural networks.
problem Training neural networks with model compression techniques suffers from accuracy loss and convergence issues.
method Holistic framework based on nonconvex optimization, using NN-BCD algorithm with closed-form iteration scheme.
result The proposed algorithm globally converges to a critical point at a rate of O(1/k).
IFR-Net improves MRI images with faster, better detail recovery.
problem Fine structure loss in high-speed MRI scans.
method Iterative feature refinement network with trainable parameters and CNN-based inversion blocks.
result Preserves image details and structural information with faster reconstruction.
HSQ reduces communication costs in federated learning.
problem High cost of communicating gradients in federated learning.
method Hyper-sphere quantization (HSQ) framework for efficient gradient compression.
result HSQ achieves O(logd) per-iteration communication cost, significantly reducing costs without compromising accuracy. Paper analyzes BIHT for noisy 1-bit CS, improving results with up to τ-fraction of incorrect measurements.
problem Estimating sparse vectors from noisy sign measurements in 1-bit compressed sensing.
method Binary Iterative Hard Thresholding (BIHT) algorithm, using Gaussian matrices and high-dimensional geometry analysis.
result BIHT provides estimates within ε+τ error with τ-fraction of incorrect measurements, maintaining universality of measurements.
Study proposes a new method for MRI image reconstruction using denoising autoencoders and undecimated wavelet transforms.
problem Efficient MRI image reconstruction using under-sampled data.
method Undecimated wavelet transform, denoising autoencoder, proximal gradient algorithm.
result The proposed method enhances MRI image reconstruction efficiency and robustness.
Unified strategy for efficient data compression and model estimation.
problem Limited interactive exploration and data interaction in linear model development and deployment.
method Conditionally sufficient statistics for optimal data compression and estimation of linear models.
result Linear models can be estimated from compressed data without loss of parameters or covariances.
Improved neural network reconstruction from sparse measurements with theoretical guarantees.
problem Improving neural network performance in sparse signal reconstruction from few measurements.
method Combining iterative reconstruction algorithms with neural networks, analyzing generalization properties, and deriving a generalization bound.
result Theoretical guarantees for neural network reconstruction from compressive linear measurements, with generalization error scaling logarithmically in the number of layers and linearly in the number of measurements.
IntSGD compresses SGD gradients without floats, converging as SGD.
problem Efficiently compressing stochastic gradients in distributed SGD.
method Adaptive integer compression of gradients, estimating scaling adaptively.
result IntSGD matches SGD's iteration complexity for convex and non-convex functions.
Compressing convolutional neural networks (CNNs) is essential for transferring the success of CNNs to a wide variety of applications to mobile devices. In contrast to directly recognizing subtle weights or filters as redundant in a given CNN, this paper presents an evolutionary method to automatically eliminate redunda…
Recovering matrices from compressive and grossly corrupted observations is a fundamental problem in robust statistics, with rich applications in computer vision and machine learning. In theory, under certain conditions, this problem can be solved in polynomial time via a natural convex relaxation, known as Compressive …
Computational Fluid Dynamics (CFD) is a hugely important subject with applications in almost every engineering field, however, fluid simulations are extremely computationally and memory demanding. Towards this end, we present Lat-Net, a method for compressing both the computation time and memory usage of Lattice Boltzm…
Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network qu…
Proposes using equivariant generative models for compressed sensing with unknown orientations.
problem Recovering signals with unknown orientations from underdetermined systems of linear measurements.
method Equivariant variational autoencoder as a generative prior for compressed sensing.
result Signals with unknown orientations can be recovered using iterative gradient descent on the latent space of equivariant models.
Layer fusion reduces deep neural network layers with minimal loss in accuracy.
problem Model compression to reduce neural network size and computation.
method Fusion of similar layers to reduce model size with minimal performance loss.
result Deep networks can be compressed up to 3.33x with minimal accuracy loss.
Modern scientific instruments produce vast amounts of data, which can overwhelm the processing ability of computer systems. Lossy compression of data is an intriguing solution, but comes with its own drawbacks, such as potential signal loss, and the need for careful optimization of the compression ratio. In this work, …
Paper proposes a compression principle for neural networks using Bayesian optimization.
problem Finding methods for making generalizable predictions in machine learning.
method Compression principle and Bayesian optimization approach.
result Optimal predictive models minimize total compressed message length of data and model definition.
Paper proposes compressive ICA algorithms for ICA model.
problem Efficiently solving ICA model with reduced memory and computational complexity.
method Compressive learning approach to ICA model, proving existence of compressive ICA scheme, proposing two algorithms (IPG and ASD).
result Proposed algorithms achieve substantial memory gains over well-known ICA algorithms.
We present a filter correlation based model compression approach for deep convolutional neural networks. Our approach iteratively identifies pairs of filters with the largest pairwise correlations and drops one of the filters from each such pair. However, instead of discarding one of the filters from each such pair naï…
In today's data driven world, storing, processing, and gleaning insights from large-scale data are major challenges. Data compression is often required in order to store large amounts of high-dimensional data, and thus, efficient inference methods for analyzing compressed data are necessary. Building on a recently desi…
We consider a decomposition method for compressive streaming data in the context of online compressive Robust Principle Component Analysis (RPCA). The proposed decomposition solves an n-ℓ1 cluster-weighted minimization to decompose a sequence of frames (or vectors), into sparse and low-rank components, from com…
Neurally Augmented ALISTA improves sparse reconstruction performance.
problem Improving sparse reconstruction performance with theoretical guarantees and empirical improvements.
method Integrates an LSTM network to compute adaptive step sizes and thresholds for each target vector during reconstruction.
result Empirical performance is further improved, especially as compression ratios become more challenging.
Diffusion models improve image compression at low bit-rates.
problem Efficiently compressing images at very low bit-rates.
method Encoding into an embedding, using diffusion models to refine the embedding iteratively.
result Realistic reconstructions can be generated at extremely low bit-rates.
In this paper we show that the computational complexity of the Iterative Thresholding and K-residual-Means (ITKrM) algorithm for dictionary learning can be significantly reduced by using dimensionality-reduction techniques based on the Johnson-Lindenstrauss lemma. The dimensionality reduction is efficiently carried out…
This paper proposes a new method for efficient data compression using Bayesian neural networks.
problem Efficient compression of data represented as functions mapping coordinates to signal values.
method Overfitting variational Bayesian neural networks to the data and compressing an approximate posterior weight sample using relative entropy coding.
result Our method achieves strong performance on image and audio compression while retaining simplicity.