Quantization-aware training can recover accuracy lost by post-training quantization.
problem Post-training quantization (PTQ) can fail sharply at aggressive bitwidths.
method A unified geometric framework that explains PTQ failure and QAT recovery.
result QAT has a useful bias that steers iterates back into the basin.
Paper develops a statistical framework for quantized training of deep neural networks.
problem Lack of theoretical understanding of gradient quantization in FQT.
method Presented a statistical framework for analyzing FQT algorithms, viewing quantized gradient as a stochastic estimator of QAT gradient.
result Developed two novel gradient quantizers with smaller variance than existing per-tensor quantizer.
BatchNorm helps train quantized networks by avoiding gradient explosion.
problem Training quantized neural networks is difficult due to gradient issues.
method Investigated the impact of BatchNorm on both full-precision and quantized networks.
result BatchNorm avoids gradient explosion in quantized networks, contrary to expectations.
Optimal gradient quantization reduces communication costs in distributed deep learning.
problem High communication costs in distributed training of deep neural networks.
method Deduced optimal gradient quantization conditions for binary and multi-level quantization, developed novel schemes for dynamic quantization levels.
result Demonstrated superior performance of proposed quantization schemes on CIFAR and ImageNet datasets.
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
Proposes a robust neural network quantization method.
problem Training model's dependency on specific quantization methods.
method Intrinsic robustness to various quantization processes.
result Single model capable of operating at various bit-widths and policies.
We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks…
New method trains quantized neural networks to global optimality.
problem Training optimal quantized neural networks is intractable due to combinatorial non-convex optimization.
method Convex optimization strategy using hidden convexity, semidefinite lifting, and Grothendieck's identity.
result Quantized NN problems can be solved to global optimality in polynomial-time.
We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-deman…
Quantized Adam reduces communication cost in deep learning training.
problem Reducing communication cost in distributed deep learning training.
method Gradient and weight quantization with error feedback in Adam.
result Proposed methods converge to first-order stationary points.
This paper introduces a differentiable, scalable quantization method for neural networks.
problem Previous quantization methods lacked differentiability and scalability.
method The approach is differentiable and scalable, using bit-shifting and logarithmic quantization.
result The method achieves comparable accuracy to state-of-the-art approaches with less training time and lower inference cost.
Neural network quantization procedure is the necessary step for porting of neural networks to mobile devices. Quantization allows accelerating the inference, reducing memory consumption and model size. It can be performed without fine-tuning using calibration procedure (calculation of parameters necessary for quantizat…
Post-training quantization method using multiple low-precision points achieves higher precision for critical weights.
problem Discretizing pre-trained deep neural networks without re-training.
method Multipoint quantization with efficient greedy selection and adaptive point number.
result Outperforms state-of-the-art methods on ImageNet classification and PASCAL VOC object detection.
Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towa…
Unified finetuning of all quantization degrees of freedom achieves state-of-the-art 4-bit quantization.
problem Achieving high accuracy in quantized neural networks while maintaining speed and resource constraints.
method Quantization-aware finetuning (QFT) that jointly optimizes all quantization degrees of freedom.
result 4-bit weight quantization results on-par with state-of-the-art (SoTA) within PTQ constraints.
The deep layers of modern neural networks extract a rather rich set of features as an input propagates through the network. This paper sets out to harvest these rich intermediate representations for quantization with minimal accuracy loss while significantly reducing the memory footprint and compute intensity of the DN…
Deep quantization of neural networks (below eight bits) offers significant promise in reducing their compute and storage cost. Albeit alluring, without special techniques for training and optimization, deep quantization results in significant accuracy loss. To further mitigate this loss, we propose a novel sinusoidal r…
Deep neural network (DNN) quantization converting floating-point (FP) data in the network to integers (INT) is an effective way to shrink the model size for memory saving and simplify the operations for compute acceleration. Recently, researches on DNN quantization develop from inference to training, laying a foundatio…
Proposes QEP to mitigate quantization error propagation in layer-wise post-training quantization.
problem Growth of quantization errors across layers degrades performance, especially in low-bit regimes.
method Quantization Error Propagation (QEP) framework that explicitly propagates and compensates for quantization errors.
result QEP-enhanced layer-wise PTQ achieves substantially higher accuracy, especially in low-bit regimes.
Neural network models are resource hungry. It is difficult to deploy such deep networks on devices with limited resources, like smart wearables, cellphones, drones, and autonomous vehicles. Low bit quantization such as binary and ternary quantization is a common approach to alleviate this resource requirements. Ternary…
We extend quantization-aware training to extreme model compression.
problem Maximizing model accuracy with minimal model size.
method Quantize a random subset of weights during training, allowing unbiased gradients through other weights.
result Established new state-of-the-art compromises between accuracy and model size.
Low bit-width integer weights and activations are very important for efficient inference, especially with respect to lower power consumption. We propose Monte Carlo methods to quantize the weights and activations of pre-trained neural networks without any re-training. By performing importance sampling we obtain quantiz…
AdaRound improves post-training quantization of neural networks.
problem Improving the accuracy of quantized weights in neural networks.
method Adaptive rounding mechanism that adapts to data and task loss.
result AdaRound outperforms rounding-to-nearest and achieves state-of-the-art performance.
SPEQ improves quantized neural networks by stochastic precision sharing and cosine similarity loss.
problem Improving quantized deep neural networks for edge devices.
method SPEQ combines stochastic precision sharing and cosine similarity loss for knowledge distillation.
result SPEQ outperforms existing methods in various tasks.
Recent research implies that training and inference of deep neural networks (DNN) can be computed with low precision numerical representations of the training/test data, weights and gradients without a general loss in accuracy. The benefit of such compact representations is twofold: they allow a significant reduction o…
Discretizing multi-dimensional data distributions is a fundamental step of modern indexing methods. State-of-the-art techniques learn parameters of quantizers on training data for optimal performance, thus adapting quantizers to the data. In this work, we propose to reverse this paradigm and adapt the data to the quant…
Global-QSGD accelerates distributed training by up to 3.51%.
problem High communication overhead in distributed deep learning.
method Allreduce-compatible gradient quantization with theoretical guarantees.
result Global-QSGD accelerates distributed training by up to 3.51%.
AdaQuantFL reduces communication in federated learning by adaptively quantizing model updates.
problem Efficient communication of model updates in federated learning with high-dimensional models and limited bandwidth.
method AdaQuantFL uses adaptive quantization to reduce the number of bits for model updates while maintaining low error floor.
result AdaQuantFL converges in fewer communicated bits compared to fixed quantization levels, with minimal impact on accuracy.
Degree-Quant improves GNN efficiency by quantizing them without losing accuracy.
problem Efficiency of graph neural networks at inference time.
method Architecturally-agnostic method, Degree-Quant, for quantizing GNNs.
result Degree-Quant trained models perform as well as full-precision models and achieve up to 26% gains.
FrostNet improves INT8 quantization efficiency in mobile networks.
problem The importance of network architecture for optimal INT8 quantization.
method Quantization-aware training (QAT) with StatAssist and GradBoost, hardware-aware NAS.
result FrostNets achieve higher recognition accuracy with comparable latency when quantized.
Meta learning optimizes neural network quantization for efficient inference.
problem Uniform bitwidth quantization is sub-optimal for neural network compression.
method Meta learning to automatically generate hybrid quantization policies.
result Meta learning outperforms uniform quantization and RL approaches.
Based on the notion of information bottleneck (IB), we formulate a quantization problem called "IB quantization". We show that IB quantization is equivalent to learning based on the IB principle. Under this equivalence, the standard neural network models can be viewed as scalar (single sample) IB quantizers. It is know…
Designing a deep neural network (DNN) with good generalization capability is a complex process especially when the weights are severely quantized. Model averaging is a promising approach for achieving the good generalization capability of DNNs, especially when the loss surface for training contains many sharp minima. W…
Paper shows how to integrate quantization into neural compression models.
problem Integrating quantization into neural compression models.
method Integrates uniform noise channel at test time using universal quantization.
result Eliminates mismatch between training and test phases while maintaining differentiability.
This work studies scaling laws for low-precision training in high-dimensional linear regression.
problem Optimizing trade-off between model quality and training costs in high-dimensional linear regression.
method Theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression framework, analyzing multiplicative and additive quantization.
result Multiplicative quantization maintains full-precision model size, while additive quantization reduces effective model size.
S2D selectively decays large singular values to improve quantization of neural activations.
problem Large activation outliers in transformer models cause accuracy drops during quantization.
method Selective Spectral Decay (S2D) that surgically regularizes only the largest singular values. result Significantly reduces activation outliers and produces well-conditioned representations.
Study quantization effects on high-dimensional linear regression learning.
problem Understanding quantization's impact on learning high-dimensional linear regression models.
method Analyzes stochastic gradient descent for high-dimensional linear regression under various quantization targets.
result Establishes precise bounds on excess risk for different quantization schemes.
SMGD trains low-bit neural networks with memory constraints.
problem Training large neural networks with limited memory.
method Stochastic Markov Gradient Descent (SMGD).
result Encouraging numerical results and theoretical guarantees.
Differentiable model compression adds noise to parameters during training.
problem Model compression for efficient deployment.
method Adding pseudo quantization noise to model parameters during training.
result Compresses model by more than 8x on ImageNet with 0.3% accuracy loss.
Quantizes neural networks using frame theory for improved accuracy.
problem Improving neural network efficiency and accuracy through quantization.
method Sigma-Delta (ΣΔ) quantization with finite unit-norm tight frames. result Error bound between original and quantized neural networks derived.
Continuous representations have been widely adopted in recommender systems where a large number of entities are represented using embedding vectors. As the cardinality of the entities increases, the embedding components can easily contain millions of parameters and become the bottleneck in both storage and inference du…
UVeQFed tackles FL model compression over limited channels.
problem Efficiently transmitting trained models over rate-constrained channels.
method Universal vector quantization for FL (UVeQFed).
result UVeQFed minimizes distortion and converges to optimal model.
A new gradient quantization scheme improves communication efficiency in distributed training.
problem Efficiently compressing gradients for parallel training of large models.
method Proposes a new gradient quantization scheme with theoretical guarantees and empirical performance.
result The new scheme matches and exceeds the performance of existing methods.
Post-training quantization saves resources for neural networks.
problem Implementing neural networks in resource-constrained hardware.
method Generalized post-training quantization method (GPFQ) with modifications for sparsity and error analysis.
result Error decays linearly with over-parametrization, showing minor loss of accuracy.
One of the most significant bottleneck in training large scale machine learning models on parameter server (PS) is the communication overhead, because it needs to frequently exchange the model gradients between the workers and servers during the training iterations. Gradient quantization has been proposed as an effecti…
AskewSGD optimizes quantized neural networks with interval-constrained optimization.
problem Training deep neural networks with quantized weights.
method Formulates QNN training as smoothed interval-constrained optimization, proposes AskewSGD for solving each subproblem.
result AskewSGD avoids projections and allows infeasible iterates, performs better than state-of-the-art methods.
Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. …
Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves…