arXiv research

ActNN reduces neural network training memory by 2-bit quantization.

The method of random projections has become a standard tool for machine learning, data mining, and search with massive data at Web scale. The effective use of random projections requires efficient coding schemes for quantizing (real-valued) projected data into integers. In this paper, we focus on a simple 2-bit coding …

Quantized-TinyLLaVA reduces communication costs in split learning for multimodal models.

Improved 2-bit covariance estimator with reduced operator norm error and no tuning needed.

Autoencoder-based geometric shaping is proposed that includes optimizing bit mappings. Up to 0.2 bits/QAM symbol gain in GMI is achieved for a variety of data rates and in the presence of transceiver impairments. The gains can be harvested with standard binary FEC at no cost w.r.t. conventional BICM.

Maximizing trading volume in online learning framework between traders.

Adversarial examples are perturbed inputs that are designed (from a deep learning network's (DLN) parameter gradients) to mislead the DLN during test time. Intuitively, constraining the dimensionality of inputs or parameters of a network reduces the 'space' in which adversarial examples exist. Guided by this intuition,…

This paper presents a novel end-to-end methodology for enabling the deployment of low-error deep networks on microcontrollers. To fit the memory and computational limitations of resource-constrained edge-devices, we exploit mixed low-bitwidth compression, featuring 8, 4 or 2-bit uniform quantization, and we model the i…

Our work specifies the fundamental cost of using secure aggregation in federated learning.

Low-bit training framework reduces energy consumption in CNNs.

Implementing large-scale deep neural networks with high computational complexity on low-cost IoT devices may inevitably be constrained by limited computation resource, making the devices hard to respond in real-time. This disjunction makes the state-of-art deep learning algorithms, i.e. CNN (Convolutional Neural Networ…

A trend towards energy-efficiency, security and privacy has led to a recent focus on deploying DNNs on microcontrollers. However, limits on compute and memory resources restrict the size and the complexity of the ML models deployable in these systems. Computation-In-Memory architectures based on resistive nonvolatile m…

Deep Convolutional Neural Networks (DCNNs) are currently popular in human activity recognition applications. However, in the face of modern artificial intelligence sensor-based games, many research achievements cannot be practically applied on portable devices. DCNNs are typically resource-intensive and too large to be…

We study the following generalized matrix rank estimation problem: given an $n \times n$ matrix and a constant $c \geq 0$, estimate the number of eigenvalues that are greater than $c$. In the distributed setting, the matrix of interest is the sum of $m$ matrices held by separate machines. We show that any deterministic…

New memory-query tradeoffs for convex optimization algorithms.

Knowledge distillation (KD) is a very popular method for model size reduction. Recently, the technique is exploited for quantized deep neural networks (QDNNs) training as a way to restore the performance sacrificed by word-length reduction. KD, however, employs additional hyper-parameters, such as temperature, coeffici…

Flexible framework compresses models using LC algorithm.

Reduces multiclass and regression compression schemes to binary ones.

Proposes a link between randomness and compression in deep learning.

Paper introduces a new adaptive gradient method with gradient compression for distributed training.

Galen algorithm compresses neural networks for specific hardware with reduced latency.

The (isothermic) compressibility of lattice knots can be examined as a model of the effects of topology and geometry on the compressibility of ring polymers. In this paper, the compressibility of minimal length lattice knots in the simple cubic, face centered cubic and body centered cubic lattices are determined. Our r…

HiLLoC compresses large images losslessly using VAEs.

Neural NCD reveals LLMs don't compress well for classification.

BDC compresses both sample size and dimensionality of large datasets.

DoCoFL compresses model updates for cross-device federated learning.

Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quan…

New taxonomy and evaluation of neural network compression methods.

Proposes a method to train neural networks directly on compressed text data.

Study shows LLC correlates with neural network compressibility.

Paper proposes SCALLION and SCAFCOM for compressed FL with reduced communication.

New uncertainty principle limits compression in distributed learning, suggesting optimal methods.

One of the biggest issues in deep learning theory is the generalization ability of networks with huge model size. The classical learning theory suggests that overparameterized models cause overfitting. However, practically used large deep models avoid overfitting, which is not well explained by the classical approaches…

Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size.As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This paper introduces a method to compress RNNs for resource constrained e…

Proposes a new linearity-based neural network compression method.

Randomized matrix compression techniques, such as the Johnson-Lindenstrauss transform, have emerged as an effective and practical way for solving large-scale problems efficiently. With a focus on computational efficiency, however, forsaking solutions quality and accuracy becomes the trade-off. In this paper, we investi…

Can textual data be compressed intelligently without losing accuracy in evaluating sentiment? In this study, we propose a novel evolutionary compression algorithm, PARSEC (PARts-of-Speech for sEntiment Compression), which makes use of Parts-of-Speech tags to compress text in a way that sacrifices minimal classification…

Unified framework compresses GANs up to 47x with minimal quality loss.

New compression theory justifies model pruning for neural networks.

SHVC improves image compression with fewer parameters.

Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model com…

COIN++ compresses multiple data types efficiently.

Compressive learning is a framework where (so far unsupervised) learning tasks use not the entire dataset but a compressed summary (sketch) of it. We propose a compressive learning classification method, and a novel sketch function for images.

PCRs compress data for deep learning, reducing training time.

Paper introduces adversarial lossy compression for video artifacts reduction.

LLMs compress financial texts, but distort decision-making.

A new method, REC, compresses images by encoding their latent representations efficiently.

Wavelets help compress neural networks efficiently.