Proposes a method to train neural networks directly on compressed text data.
problem Training neural networks on compressed text data without decompression.
method Introduces composer modules to encode symbols from grammar compression rules into vector representations.
result Demonstrates that the proposed method can achieve both memory and computational efficiency while maintaining moderate performance.
COIN++ compresses multiple data types efficiently.
problem Handling diverse data modalities in neural compression.
method Implicit neural representations and modulations quantization.
result Significant compression gains with reduced encoding time.
Can textual data be compressed intelligently without losing accuracy in evaluating sentiment? In this study, we propose a novel evolutionary compression algorithm, PARSEC (PARts-of-Speech for sEntiment Compression), which makes use of Parts-of-Speech tags to compress text in a way that sacrifices minimal classification…
Model compression is eminently suited for deploying deep learning on IoT-devices. However, existing model compression techniques rely on access to the original or some alternate dataset. In this paper, we address the model compression problem when no real data is available, e.g., when data is private. To this end, we p…
Paper proposes SCALLION and SCAFCOM for compressed FL with reduced communication.
problem Reducing communication overhead in Federated Learning with data heterogeneity and partial participation.
method Revisit and simplify stochastic controlled averaging, proposing SCALLION and SCAFCOM for unbiased and biased compression.
result SCALLION and SCAFCOM outperform existing methods in communication and computation complexities.
L21 SNF compresses mixed-sign data robustly.
problem Compression of mixed-sign data with high fidelity.
method Regularized L21 Semi-NonNegative Matrix Factorization (L21 SNF).
result Rigorous proof of convergence and use-case advantages demonstrated.
A new method compresses conditional distributions of labelled data.
problem No existing method directly compresses the conditional distribution of labelled data.
method Introduce Average Maximum Conditional Mean Discrepancy (AMCMD), derive a closed form estimator, and extend Kernel Herding (KH) to Average Conditional Kernel Herding (ACKH).
result Directly compressing conditional distributions outperforms joint distribution compression and greedy selection.
Deep learning accelerators efficiently train over vast and growing amounts of data, placing a newfound burden on commodity networks and storage devices. A common approach to conserve bandwidth involves resizing or compressing data prior to training. We introduce Progressive Compressed Records (PCRs), a data format that…
Data compression is a popular technique for improving the efficiency of data processing workloads such as SQL queries and more recently, machine learning (ML) with classical batch gradient methods. But the efficacy of such ideas for mini-batch stochastic gradient descent (MGD), arguably the workhorse algorithm of moder…
Unified strategy for efficient data compression and model estimation.
problem Limited interactive exploration and data interaction in linear model development and deployment.
method Conditionally sufficient statistics for optimal data compression and estimation of linear models.
result Linear models can be estimated from compressed data without loss of parameters or covariances.
This paper speeds up SVC clustering by compressing data while preserving key properties.
problem Efficiently clustering large-scale real-world data sets.
method Spectrum-preserving data compression for fast support vector clustering.
result Achieved 100X and 115X speedups on real-world data sets while maintaining clustering quality.
BDC compresses both sample size and dimensionality of large datasets.
problem Large datasets in both sample size and dimensionality.
method Two-stage framework using Decoded MMD, Reconstruction MMD, and Encoded MMD.
result BDC achieves comparable or superior performance with lower cost and higher compression rates.
One of the biggest issues in deep learning theory is the generalization ability of networks with huge model size. The classical learning theory suggests that overparameterized models cause overfitting. However, practically used large deep models avoid overfitting, which is not well explained by the classical approaches…
What learning algorithms can be run directly on compressively-sensed data? In this work, we consider the question of accurately and efficiently computing low-rank matrix or tensor factorizations given data compressed via random projections. We examine the approach of first performing factorization in the compressed dom…
Compressed sensing in MRI enables high subsampling factors while maintaining diagnostic image quality. This technique enables shortened scan durations and/or improved image resolution. Further, compressed sensing can increase the diagnostic information and value from each scan performed. Overall, compressed sensing has…
LEAD algorithm speeds up decentralized optimization with compression.
problem Slow convergence and stability issues in decentralized optimization with compression.
method Proposes the first linearly convergent decentralized algorithm with compression.
result First consensus error bound for coupled dynamics of primal and dual updates.
New framework compresses and recovers scientific data efficiently.
problem Efficiently managing and recovering from large scientific datasets.
method Grounded in learning exponential families, preserves uncertainty and supports trade-offs.
result Preserves physical features and quantities of interest in compressed representations.
Improves compression of neural networks for embedded systems.
problem High computational cost and data labeling issues in DNNs.
method Domain Adaptation Regularization for Spectral Pruning.
result Our method outperforms existing methods by a large margin for high compression rates.
Distributed sensors compress and send features to a fusion center for linear regression.
problem Efficiently compress and transmit features from distributed sensors to a fusion center under varying communication constraints.
method Designs a distributed and adaptive feature compression scheme using optimal quantizers and simple adaptive strategies.
result Demonstrates improved inference performance through simulated experiments.
Compression affects deep networks differently, impacting underrepresented data points.
problem Disparate impact of compression on different classes and images.
method Analysis of deep neural network pruning and quantization effects.
result Compression disproportionately impacts model performance on underrepresented data points.
Motivated by the ever-increasing demands for limited communication bandwidth and low-power consumption, we propose a new methodology, named joint Variational Autoencoders with Bernoulli mixture models (VAB), for performing clustering in the compressed data domain. The idea is to reduce the data dimension by Variational…
New methods compress models without real data, reducing accuracy loss.
problem Compression requires real data, which is often unavailable or sensitive.
method Synthetic data generation from trained models for calibration and fine-tuning.
result Best method shows negligible accuracy loss compared to original training set.
Real-world data typically contain repeated and periodic patterns. This suggests that they can be effectively represented and compressed using only a few coefficients of an appropriate basis (e.g., Fourier, Wavelets, etc.). However, distance estimation when the data are represented using different sets of coefficients i…
Autoencoders fail to capture sparse structure in 1-bit data compression.
problem Proving the performance of shallow autoencoders on sparse data compression.
method Gradient descent analysis and approximate message passing.
result Gradient descent minimizer for sparse data is the identity (up to permutation) above critical sparsity.
Recent research has studied the role of sparsity in high dimensional regression and signal reconstruction, establishing theoretical limits for recovering sparse models from sparse data. This line of work shows that ℓ1-regularized least squares regression can accurately estimate a sparse linear model from n nois…
DACE estimates covariance from compressed data, improving accuracy.
problem Estimating covariance from large, distributed data.
method Data-aware weighted sampling for unbiased estimation.
result DACE provides more accurate covariance estimation under compression.
New method compresses large sample data for faster discriminant analysis.
problem Large sample sizes in discriminant analysis increase computational burden.
method Proposes a new compression approach for reducing training samples.
result Significant computational gains and superior predictive ability compared to random sub-sampling.
A new method learns graph compression from data.
problem Graphs lack ordering, making conventional compression algorithms ineffective.
method Partition and Code framework: decompose, learn, encode.
result PnC achieves compression gains that grow with graph size.
Designing a data sharing mechanism without sacrificing too much privacy can be considered as a game between data holders and malicious attackers. This paper describes a compressive adversarial privacy framework that captures the trade-off between the data privacy and utility. We characterize the optimal data releasing …
Principal component analysis, dictionary learning, and auto-encoders are all unsupervised methods for learning representations from a large amount of training data. In all these methods, the higher the dimensions of the input data, the longer it takes to learn. We introduce a class of neural networks, termed RandNet, f…
Lossless compression methods shorten the expected representation size of data without loss of information, using a statistical model. Flow-based models are attractive in this setting because they admit exact likelihood optimization, which is equivalent to minimizing the expected number of bits per message. However, con…
New video compression method outperforms traditional approaches.
problem Efficient video compression in low latency mode.
method Feedback Recurrent Autoencoder network architecture.
result State of the art MS-SSIM/rate performance on UVG dataset.
Lossless compression of deep neural networks using NTK and RMT.
problem Compressing large-scale deep neural networks for low-power devices.
method High-dimensional neural tangent kernel approach.
result Asymptotic spectral equivalence between NTK matrices of wide DNNs enables lossless compression.
This paper proposes a new method for efficient data compression using Bayesian neural networks.
problem Efficient compression of data represented as functions mapping coordinates to signal values.
method Overfitting variational Bayesian neural networks to the data and compressing an approximate posterior weight sample using relative entropy coding.
result Our method achieves strong performance on image and audio compression while retaining simplicity.
Paper proposes a compression principle for neural networks using Bayesian optimization.
problem Finding methods for making generalizable predictions in machine learning.
method Compression principle and Bayesian optimization approach.
result Optimal predictive models minimize total compressed message length of data and model definition.
Adaptive Quantization Modules enable online continual compression of non-i.i.d data streams.
problem Learning to compress and store a dataset from a non-i.i.d data stream, only observing each sample once.
method Discrete auto-encoders and Adaptive Quantization Modules (AQM) to control compression ability.
result Significant gains on continual learning benchmarks with AQM replacing episodic memory.
Lossy compression of statistical data using quantum annealing.
problem Efficiently compressing statistical floating-point data.
method Representation learning with binary variables, classical optimization of basis vectors, quantum annealing for coefficients, bias correction.
result Quantum annealing shows promising results with 3.5x better compression than neural-network autoencoders.
This paper develops a method to estimate the rate-distortion function for general data sources.
problem Estimating the rate-distortion function for general data sources.
method Develops an algorithm for sandwiching the R-D function of a general (not necessarily discrete) source using i.i.d. data samples.
result Estimates R-D sandwich bounds for various data sources, including natural images, indicating potential for improving compression methods.
The usage of deep generative models for image compression has led to impressive performance gains over classical codecs while neural video compression is still in its infancy. Here, we propose an end-to-end, deep generative modeling approach to compress temporal sequences with a focus on video. Our approach builds upon…
New taxonomy and evaluation of neural network compression methods.
problem Efficiency of deep neural networks in real-world applications.
method Categorization and evaluation of tensor factorization and probabilistic compression methods.
result SVD and probabilistic compression methods are complementary and give the best results.
We present a new similarity measure based on information theoretic measures which is superior than Normalized Compression Distance for clustering problems and inherits the useful properties of conditional Kolmogorov complexity. We show that Normalized Compression Dictionary Size and Normalized Compression Dictionary En…
Data compression speeds up machine learning loss calculations.
problem Computational demand in calculating mean squared error for large datasets.
method Use rank-1 lattices to compress data, assigning weights based on original data and responses.
result Our QMC data compression algorithms can lead to arbitrary high convergence rates for smooth functions.
New compression theory justifies model pruning for neural networks.
problem Improving neural network performance with reduced model size.
method Information-theoretic rate-distortion theory applied to NN compression.
result Pruning improves model performance on CIFAR-10 and ImageNet datasets.
Wavelets help compress neural networks efficiently.
problem Efficiently compressing linear layers in neural networks.
method Learnable wavelet transforms to compress RNNs.
result Wavelet compressed RNNs have fewer parameters and perform competitively.
Few-shot network compression improves accuracy with minimal data.
problem High estimation errors from original network during inference.
method Cross distillation, layer-wise knowledge distillation approach.
result Cross distillation significantly improves student network's accuracy with few training instances.
CSDM integrates compressed sensing into diffusion models for faster data generation.
problem Efficiently generating synthetic data in high-dimensional spaces.
method Integrating compressed sensing into diffusion models (CSDM) to reduce dimensionality and accelerate inference.
result Achieves provably faster convergence and better latent space dimension selection.
New method preserves privacy while improving machine learning accuracy.
problem Privacy-preserving machine learning for daily data.
method Compressive Privacy and multi-kernel method.
result Improved utility classification accuracy with privacy preservation.
DZip improves general-purpose lossless compression using neural networks.
problem Improving general-purpose lossless compression methods.
method Hybrid architecture combining adaptive and semi-adaptive training of neural networks for prediction followed by arithmetic coding.
result Significant compression improvements on various real and synthetic datasets.