Meta learning optimizes neural network quantization for efficient inference.
problem Uniform bitwidth quantization is sub-optimal for neural network compression.
method Meta learning to automatically generate hybrid quantization policies.
result Meta learning outperforms uniform quantization and RL approaches.
Study quantization effects on high-dimensional linear regression learning.
problem Understanding quantization's impact on learning high-dimensional linear regression models.
method Analyzes stochastic gradient descent for high-dimensional linear regression under various quantization targets.
result Establishes precise bounds on excess risk for different quantization schemes.
This paper introduces a differentiable, scalable quantization method for neural networks.
problem Previous quantization methods lacked differentiability and scalability.
method The approach is differentiable and scalable, using bit-shifting and logarithmic quantization.
result The method achieves comparable accuracy to state-of-the-art approaches with less training time and lower inference cost.
This study analyzes quantization in deep learning models using statistical physics methods.
problem The computational resource requirements for large-scale data analysis models.
method Typical case analysis from statistical physics, specifically the replica method.
result Optimal quantization width minimizes error and delays overfitting.
Based on the notion of information bottleneck (IB), we formulate a quantization problem called "IB quantization". We show that IB quantization is equivalent to learning based on the IB principle. Under this equivalence, the standard neural network models can be viewed as scalar (single sample) IB quantizers. It is know…
Optimal gradient quantization reduces communication costs in distributed deep learning.
problem High communication costs in distributed training of deep neural networks.
method Deduced optimal gradient quantization conditions for binary and multi-level quantization, developed novel schemes for dynamic quantization levels.
result Demonstrated superior performance of proposed quantization schemes on CIFAR and ImageNet datasets.
In order to quantize the gate parameters of the LSTM (Long Short-Term Memory) neural network model with almost no recognition performance degraded, a new quantization method named Quantization Loss Re-Learn Method is proposed in this paper. The method does lossy quantization on gate parameters during training iteration…
Quantized Adam reduces communication cost in deep learning training.
problem Reducing communication cost in distributed deep learning training.
method Gradient and weight quantization with error feedback in Adam.
result Proposed methods converge to first-order stationary points.
A scalable approach to learning from probability measures using quantization.
problem Efficiently comparing and manipulating large sets of probability measures.
method Quantization of probability measures to a fixed support, followed by optimal transport computations.
result Consistency and convergence guarantees for quantized measures in various OT-based tasks.
This paper evaluates quantization techniques for deep learning inference.
problem Reducing the size and improving inference of deep neural networks.
method Review and empirical evaluation of quantization parameters for various neural network models.
result 8-bit quantization maintains accuracy within 1% of floating-point models.
Due to its efficiency and ease to implement, stochastic gradient descent (SGD) has been widely used in machine learning. In particular, SGD is one of the most popular optimization methods for distributed learning. Recently, quantized SGD (QSGD), which adopts quantization to reduce the communication cost in SGD-based di…
This study optimizes quantized neural networks by considering model architecture and quantization types.
problem Optimizing quantized neural networks for low-power, high-throughput applications.
method Holistic approach including training methods and quantization-friendly architecture design.
result Deeper models are more sensitive to activation quantization, while wider models improve resilience to both weight and activation quantization.
AdaQuantFL reduces communication in federated learning by adaptively quantizing model updates.
problem Efficient communication of model updates in federated learning with high-dimensional models and limited bandwidth.
method AdaQuantFL uses adaptive quantization to reduce the number of bits for model updates while maintaining low error floor.
result AdaQuantFL converges in fewer communicated bits compared to fixed quantization levels, with minimal impact on accuracy.
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the …
Improved deep learning model deployment on tiny MCUs with mixed-precision quantization.
problem Memory limitations prevent accurate deployment of DNN models on tiny MCUs.
method Automated mixed-precision quantization using Reinforcement Learning for MCU constraints.
result Mixed-precision models achieve high accuracy with uniform quantization policies.
New method quantizes neural networks for mobile devices.
problem High computational and memory costs of deep neural networks.
method Formulates quantization as a differentiable function.
result Quantization networks outperform state-of-the-art methods.
New quantization methods improve accuracy of Random Fourier Features.
problem Improving accuracy of Random Fourier Features for machine learning.
method Sigma-Delta and distributed noise-shaping quantization methods for 1-bit and low bit-depth quantization.
result Quantized RFFs allow high accuracy approximation of underlying kernels with polynomial error decay.
New insights into quantized neural networks reveal learning dynamics and generalization errors.
problem Understanding the impact of quantization hyperparameters on learning dynamics in high-dimensional models.
method Theoretical analysis and fixed-point analysis of STE dynamics in quantized models.
result STE training in quantized models converges to a plateau followed by a sharp drop in generalization error, influenced by quantization range.
SQWA improves low-precision DNNs with model averaging and quantization.
problem Designing good generalization DNNs with quantized weights.
method Floating-point model training, direct quantization, multiple low-precision models, weight averaging, re-quantization, fine-tuning, loss visualization.
result State-of-the-art results for 2-bit QDNNs on CIFAR-100 and ImageNet datasets.
One of the most significant bottleneck in training large scale machine learning models on parameter server (PS) is the communication overhead, because it needs to frequently exchange the model gradients between the workers and servers during the training iterations. Gradient quantization has been proposed as an effecti…
A new method quantizes conditional probability measures using deep learning.
problem Quantizing conditional probability measures efficiently.
method DCMQ method using Huber-energy kernel and deep neural network.
result Promising results on various examples.
Quantized-TinyLLaVA reduces communication costs in split learning for multimodal models.
problem High communication costs in split learning for multimodal models.
method Integrates a compression module that quantizes intermediate features into discrete representations before transmission.
result Achieves an approximate 87.5% reduction in communication overhead with 2-bit quantization.
There is a mismatch between the standard theoretical analyses of statistical machine learning and how learning is used in practice. The foundational assumption supporting the theory is that we can represent features and models using real-valued parameters. In practice, however, we do not use real numbers at any point d…
Autoencoders and their variations provide unsupervised models for learning low-dimensional representations for downstream tasks. Without proper regularization, autoencoder models are susceptible to the overfitting problem and the so-called posterior collapse phenomenon. In this paper, we introduce a quantization-based …
We consider the problem of deep neural net compression by quantization: given a large, reference net, we want to quantize its real-valued weights using a codebook with K entries so that the training loss of the quantized net is minimal. The codebook can be optimally learned jointly with the net, or fixed, as for bina…
Study quantized models' privacy against membership inference attacks.
problem Privacy risk in quantized machine learning models.
method Proposed a new MIS indicator for post-training quantization procedures, minimizing empirical loss.
result Demonstrated effectiveness of new MIS indicator in assessing and ranking privacy risk.
Q-GADMM reduces communication in decentralized ML by quantizing model updates.
problem Reducing communication in decentralized ML while maintaining accuracy.
method Quantized group ADMM (Q-GADMM) with adaptive quantization.
result Q-GADMM achieves similar accuracy and convergence to non-quantized methods with less communication.
This paper explores the problem of learning transforms for image compression via autoencoders. Usually, the rate-distortion performances of image compression are tuned by varying the quantization step size. In the case of autoen-coders, this in principle would require learning one transform per rate-distortion point at…
A new framework for neural network classification using vector quantization.
problem Learning a neural network classifier under the IB principle.
method Aggregated Learning framework, combining vector quantization and variational techniques.
result The effectiveness of Aggregated Learning verified through experiments.
This paper provides theoretical foundations for using quantized actions in behavior cloning.
problem Applying autoregressive models to continuous control requires discretizing actions through quantization, which is poorly understood.
method The paper analyzes quantization error propagation and statistical sample complexity, and proposes model-based augmentation.
result Behavior cloning with quantized actions achieves optimal sample complexity, matching existing lower bounds.
Adversarial examples have emerged as a significant threat to machine learning algorithms, especially to the convolutional neural networks (CNNs). In this paper, we propose two quantization-based defense mechanisms, Constant Quantization (CQ) and Trainable Quantization (TQ), to increase the robustness of CNNs against ad…
Paper proposes BQNs for efficient Bayesian quantized neural networks.
problem Learning with well-calibrated uncertainty in neural networks.
method Bayesian quantized networks (BQNs) with efficient algorithms for learning and prediction without sampling.
result BQNs achieve lower predictive errors and better-calibrated uncertainties than E-QNN with less than 20% negative log-likelihood.
Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves…
Efficient deep neural network (DNN) inference on mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precision networks achieve better performance than networks with homogeneous bitwidth for the same size constraint. Since choosing the optimal bitwi…
APoT quantization improves neural network efficiency and accuracy.
problem Efficiently quantizing weights and activations in neural networks.
method Constraining quantization levels as sums of Powers-of-Two terms, applying reparameterization, and weight normalization.
result 4-bit quantized ResNet-50 achieves 76.6% top-1 accuracy, 22% computational cost reduction.
Generative model for HW-aware DNN quantization.
problem Efficiently tuning DNNs for various hardware platforms.
method Generative model AQGAN for generating quantization configurations based on target accuracy.
result Generative model generates quantization configurations with competitive accuracy and lower search cost.
This paper quantizes CapsNets for efficient edge deployment.
problem CapsNets require intense computations and are not suitable for resource-constrained edge devices.
method Developed a specialized quantization framework for CapsNets.
result Reduced memory footprint by 6.2x with only 0.15% accuracy loss.
Memory-augmented neural networks (MANNs) refer to a class of neural network models equipped with external memory (such as neural Turing machines and memory networks). These neural networks outperform conventional recurrent neural networks (RNNs) in terms of learning long-term dependency, allowing them to solve intrigui…
Paper shows how to integrate quantization into neural compression models.
problem Integrating quantization into neural compression models.
method Integrates uniform noise channel at test time using universal quantization.
result Eliminates mismatch between training and test phases while maintaining differentiability.
Paper proposes IIQ for compressing embedding vectors.
problem Memory issues in representing large vocabularies.
method Isotropic iterative quantization (IIQ) for binary compression.
result More than 30x compression ratio with comparable performance.
New method trains quantized neural networks to global optimality.
problem Training optimal quantized neural networks is intractable due to combinatorial non-convex optimization.
method Convex optimization strategy using hidden convexity, semidefinite lifting, and Grothendieck's identity.
result Quantized NN problems can be solved to global optimality in polynomial-time.
This paper proposes a new subspace learning method, named Quantized Fisher Discriminant Analysis (QFDA), which makes use of both machine learning and information theory. There is a lack of literature for combination of machine learning and information theory and this paper tries to tackle this gap. QFDA finds a subspac…
New method classifies manifold-valued data using Riemannian geometry.
problem Classifying data on curved Riemannian manifolds.
method Probabilistic Learning Vector Quantization on Symmetric Positive Definite Matrices.
result The method outperforms traditional Euclidean methods on manifold-valued data.
The present paper develops a novel aggregated gradient approach for distributed machine learning that adaptively compresses the gradient communication. The key idea is to first quantize the computed gradients, and then skip less informative quantized gradient communications by reusing outdated gradients. Quantizing and…
Quantizers play a critical role in digital signal processing systems. Recent works have shown that the performance of quantization systems acquiring multiple analog signals using scalar analog-to-digital converters (ADCs) can be significantly improved by properly processing the analog signals prior to quantization. How…
Discretizing multi-dimensional data distributions is a fundamental step of modern indexing methods. State-of-the-art techniques learn parameters of quantizers on training data for optimal performance, thus adapting quantizers to the data. In this work, we propose to reverse this paradigm and adapt the data to the quant…
LSQ+ improves quantization of neural nets with Swish activations, achieving state-of-the-art results.
problem Quantization of neural nets with Swish activations, especially negative activations, leads to significant performance loss.
method Introduces learnable scale and offset parameters for asymmetric quantization, and uses MSE-based initialization for quantization parameters.
result Significantly outperforms LSQ for low-bit quantization of neural nets with Swish activations, achieving up to 5.6% gain with W2A2 quantization of EfficientNet-B0.