Paper addresses quantization bias in neural networks, proposing a simple method to improve performance.
problem Quantization bias causes performance degradation in low-precision neural networks.
method Compensating for quantization-induced shift by adding a constant to the bias term.
result Simple methods for estimating correction constants can match training-based quantization performance at lower cost.
Data-free quantization method improves model performance without fine-tuning.
problem Efficient quantization of deep neural networks without performance loss.
method Equalizing weight ranges and correcting biases to achieve 8-bit quantization.
result Achieves state-of-the-art quantized model performance on various architectures.
Quantization-aware training can recover accuracy lost by post-training quantization.
problem Post-training quantization (PTQ) can fail sharply at aggressive bitwidths.
method A unified geometric framework that explains PTQ failure and QAT recovery.
result QAT has a useful bias that steers iterates back into the basin.
Unified finetuning of all quantization degrees of freedom achieves state-of-the-art 4-bit quantization.
problem Achieving high accuracy in quantized neural networks while maintaining speed and resource constraints.
method Quantization-aware finetuning (QFT) that jointly optimizes all quantization degrees of freedom.
result 4-bit weight quantization results on-par with state-of-the-art (SoTA) within PTQ constraints.
Quantized Adam reduces communication cost in deep learning training.
problem Reducing communication cost in distributed deep learning training.
method Gradient and weight quantization with error feedback in Adam.
result Proposed methods converge to first-order stationary points.
Quantized Variational Inference improves ELBO optimization with fast convergence.
problem Maximizing Evidence Lower Bound (ELBO) for variational inference.
method Optimal Voronoi Tesselation for variance-free gradients, Richardson extrapolation for asymptotic improvement.
result Quantized Variational Inference leads to fast convergence with comparable computational cost.
Recently there has been significant interest in training machine-learning models at low precision: by reducing precision, one can reduce computation and communication by one order of magnitude. We examine training at reduced precision, both from a theoretical and practical perspective, and ask: is it possible to train …
Post-training quantization saves resources for neural networks.
problem Implementing neural networks in resource-constrained hardware.
method Generalized post-training quantization method (GPFQ) with modifications for sparsity and error analysis.
result Error decays linearly with over-parametrization, showing minor loss of accuracy.
Quantizes latent space to improve disentanglement in models.
problem Learning disentangled representations from unlabeled data.
method Quantizes latent space into discrete code vectors with a learnable scalar codebook and applies high weight decay regularization.
result Quantized-latent autoencoder (QLAE) outperforms prior work in disentanglement without sacrificing data reconstruction.
QTD integrates quantization with diffusion for efficient data generation.
problem Challenges in continuous diffusion models, especially long-range transitions and biases.
method Quantized Transition Diffusion (QTD) integrates data quantization with discrete diffusion dynamics.
result QTD achieves efficient data generation with minimal score evaluations.
Just as semantic hashing can accelerate information retrieval, binary valued embeddings can significantly reduce latency in the retrieval of graphical data. We introduce a simple but effective model for learning such binary vectors for nodes in a graph. By imagining the embeddings as independent coin flips of varying b…
Survey on quantization methods on Kähler manifolds.
problem None explicitly stated; focuses on methods.
method Deformation quantization, geometric quantization, Berezin-Toeplitz quantization, BV quantization.
result New relationships among quantization methods on Kähler manifolds.
The paper classifies quantizable functions and explores symmetry in quantization methods.
problem Classifying quantizable functions and understanding symmetry in quantization methods.
method Deformation quantization and geometric quantization methods are compared and classified.
result Formal quantizable functions are of a specific form and relate to Hamiltonian Killing vector fields.
AIS corrects rollout-training mismatch in quantized RL, improving speed and stability.
problem Rollout-training mismatch in quantized RL causes bias and training collapse.
method Adaptive Importance Sampling (AIS) adjusts gradient correction per batch.
result AIS matches BF16 baseline on most tasks while improving speed.
This paper introduces a differentiable, scalable quantization method for neural networks.
problem Previous quantization methods lacked differentiability and scalability.
method The approach is differentiable and scalable, using bit-shifting and logarithmic quantization.
result The method achieves comparable accuracy to state-of-the-art approaches with less training time and lower inference cost.
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
Smart Quantization adapts binary and ternary quantization for neural networks.
problem Resource constraints in deploying neural networks on devices with limited resources.
method Adaptive combination of binary and ternary quantization with a regularization function.
result Adapts quantization depth during training to maintain high model accuracy.
This study optimizes quantized neural networks by considering model architecture and quantization types.
problem Optimizing quantized neural networks for low-power, high-throughput applications.
method Holistic approach including training methods and quantization-friendly architecture design.
result Deeper models are more sensitive to activation quantization, while wider models improve resilience to both weight and activation quantization.
RATQ is a new quantizer for optimizing noisy gradients in machine learning.
problem Optimizing noisy gradients in stochastic optimization.
method RATQ uses Hadamard transform and adaptive uniform quantization, and achieves near-optimal performance.
result RATQ nearly achieves information theoretic lower bounds for optimization accuracy.
Extends ONNX for quantized neural networks with new formats and operators.
problem Handling arbitrary-precision quantization in neural networks.
method Introduces new formats and operators in ONNX to represent quantized neural networks.
result Enabled representation of uniform quantization in neural networks.
New method for quantizing symplectic manifolds with Lagrangian bundles.
problem Quantization of symplectic manifolds with Lagrangian bundles.
method A new construction of strict deformation quantization.
result Established a correspondence between differential operators and principal symbols.
HMQ improves quantization for edge devices with mixed precision.
problem Efficient quantization for edge devices with uniform, power-of-two thresholds.
method Introduces HMQ, a mixed precision quantization block that repurposes Gumbel-Softmax for searching over quantization schemes.
result Achieves competitive and state-of-the-art results on ImageNet despite restrictions.
Introduces sheaf quantization, a topological approach to geometric quantization.
problem Topological realization of WKB-states in geometric quantization.
method Enhancement of constructible sheaves, Betti counterpart of Fukaya--Floer theory.
result Introduction to sheaf quantization as a topological realization of WKB-states.
Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network qu…
The article defines and compares two types of quantizations on compact manifolds.
problem Quantization on arbitrary compact smooth manifolds.
method Embedding into CP^n and inducing quantizations from there.
result Generalizations of earlier quantization methods.
Proposes a robust neural network quantization method.
problem Training model's dependency on specific quantization methods.
method Intrinsic robustness to various quantization processes.
result Single model capable of operating at various bit-widths and policies.
The paper studies quantization on symplectic manifolds with real polarizations, comparing different quantization methods.
problem Quantization on compact symplectic manifolds with real polarizations.
method Geometric quantization, Toeplitz operators, Fourier transforms, asymptotic expansion of traces.
result Deformation quantization is realized through asymptotic traces of Toeplitz operators.
Quantizes neural networks using frame theory for improved accuracy.
problem Improving neural network efficiency and accuracy through quantization.
method Sigma-Delta (ΣΔ) quantization with finite unit-norm tight frames. result Error bound between original and quantized neural networks derived.
Abstract proposes a new categorical approach to quantization of Poisson algebras.
problem Quantization of Poisson algebras.
method Defining quantization categories as subcategories of R-module categories with classical limits.
result Categories of strict deformation quantization, prequantization, and matrix regularization are equivalent, while Poisson enveloping algebra is not.
Geometric quantization of a Poisson manifold need not imply quantization of its symplectic leaves. We provide the leafwise geometric quantization of a Poisson manifold, seen as a foliated one, whose quantum algebra restricted to each leaf is quantized.
Meta learning optimizes neural network quantization for efficient inference.
problem Uniform bitwidth quantization is sub-optimal for neural network compression.
method Meta learning to automatically generate hybrid quantization policies.
result Meta learning outperforms uniform quantization and RL approaches.
Study quantization effects on high-dimensional linear regression learning.
problem Understanding quantization's impact on learning high-dimensional linear regression models.
method Analyzes stochastic gradient descent for high-dimensional linear regression under various quantization targets.
result Establishes precise bounds on excess risk for different quantization schemes.
We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks…
FrostNet improves INT8 quantization efficiency in mobile networks.
problem The importance of network architecture for optimal INT8 quantization.
method Quantization-aware training (QAT) with StatAssist and GradBoost, hardware-aware NAS.
result FrostNets achieve higher recognition accuracy with comparable latency when quantized.
A new method quantizes LSTM gate parameters without performance loss.
problem Quantization loss in LSTM gate parameters without performance degradation.
method Lossy quantization of gate parameters during training, weight parameters adjust to offset quantization loss.
result F1 score decreased by only 0.7% on Named Entity Recognition dataset.
Optimal gradient quantization reduces communication costs in distributed deep learning.
problem High communication costs in distributed training of deep neural networks.
method Deduced optimal gradient quantization conditions for binary and multi-level quantization, developed novel schemes for dynamic quantization levels.
result Demonstrated superior performance of proposed quantization schemes on CIFAR and ImageNet datasets.
Quantizes symplectic manifolds with bounded geometry using Berezin-Toeplitz method.
problem Quantization of symplectic manifolds with bounded geometry.
method Berezin-Toeplitz quantization theory.
result Correct semiclassical limit achieved.
WNQ uses weight normalization to reduce quantization error in deep neural networks.
problem High quantization error in deep neural networks due to long-tail distribution of weights.
method Weight normalization to avoid long-tail distribution and reduce quantization error.
result WNQ achieves state-of-the-art performance on CIFAR-100 and ImageNet.
New method quantizes neural networks for mobile devices.
problem High computational and memory costs of deep neural networks.
method Formulates quantization as a differentiable function.
result Quantization networks outperform state-of-the-art methods.
This study analyzes quantization in deep learning models using statistical physics methods.
problem The computational resource requirements for large-scale data analysis models.
method Typical case analysis from statistical physics, specifically the replica method.
result Optimal quantization width minimizes error and delays overfitting.
Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the …
Paper develops a statistical framework for quantized training of deep neural networks.
problem Lack of theoretical understanding of gradient quantization in FQT.
method Presented a statistical framework for analyzing FQT algorithms, viewing quantized gradient as a stochastic estimator of QAT gradient.
result Developed two novel gradient quantizers with smaller variance than existing per-tensor quantizer.
In [3], the authors showed the existence and the uniqueness of a sl(m+1,\R)-equivariant quantization in the non-critical situations. The curved generalization of the sl(m+1,\R)-equivariant quantization is the natural and projectively equivariant quantization. In [1] and [7], the existence of such a quantization was pro…
A unified approach to geometric, symbol and deformation quantizations on a generalized flag manifold endowed with an invariant pseudo-Kaehler structure is proposed. The Hilbert space of states is realized via the Bott-Borel-Weil theorem in the sheaf cohomology of the geometric quantization line bundle. The correspondin…
A new method for robust product Markovian quantization overcomes numerical instabilities.
problem Numerical instabilities in the PMQ algorithm limit its adoption, especially for stochastic volatility models.
method Reformulated PMQ as standard vector quantization, applying accelerated Lloyd's algorithm for robustness.
result The method overcomes numerical instabilities and extends applicability to stochastic volatility models.
Memory-augmented neural networks (MANNs) refer to a class of neural network models equipped with external memory (such as neural Turing machines and memory networks). These neural networks outperform conventional recurrent neural networks (RNNs) in terms of learning long-term dependency, allowing them to solve intrigui…
Quantizes contact structures using dynamical methods.
problem Quantizing contact structures in a flat connection.
method Constructs a dynamical quantization using a flat connection on a Hilbert tractor bundle.
result Determines a contact tractor connection whose parallel sections determine a distinguished choice of Reeb dynamics.
Paper proposes a new binary quantization method for faster DNN inference.
problem Accelerating deep neural network inference on resource-limited devices.
method Quantized Compressed Sensing (QCS) for binary quantization.
result The proposed method preserves benefits of standard methods while reducing quantization error.