Smart Quantization adapts binary and ternary quantization for neural networks.
problem Resource constraints in deploying neural networks on devices with limited resources.
method Adaptive combination of binary and ternary quantization with a regularization function.
result Adapts quantization depth during training to maintain high model accuracy.
Quantized Adam reduces communication cost in deep learning training.
problem Reducing communication cost in distributed deep learning training.
method Gradient and weight quantization with error feedback in Adam.
result Proposed methods converge to first-order stationary points.
UNIQ method quantizes neural networks adaptively, reducing BOPS.
problem Efficiently quantizing neural networks to reduce computational cost.
method Adapts to parameter distribution, using uniform noise injection.
result Shows advantages in low computational budget scenarios.
Adaptive quantization improves SGD accuracy in data-parallel settings.
problem Fixed gradient quantization schemes lead to suboptimal performance in deep learning.
method Developed adaptive quantization schemes ALQ and AMQ that update compression schemes based on gradient statistics.
result Improved validation accuracy on CIFAR-10 and ImageNet datasets by 2% and 1% respectively.
AdaQuantFL reduces communication in federated learning by adaptively quantizing model updates.
problem Efficient communication of model updates in federated learning with high-dimensional models and limited bandwidth.
method AdaQuantFL uses adaptive quantization to reduce the number of bits for model updates while maintaining low error floor.
result AdaQuantFL converges in fewer communicated bits compared to fixed quantization levels, with minimal impact on accuracy.
RATQ is a new quantizer for optimizing noisy gradients in machine learning.
problem Optimizing noisy gradients in stochastic optimization.
method RATQ uses Hadamard transform and adaptive uniform quantization, and achieves near-optimal performance.
result RATQ nearly achieves information theoretic lower bounds for optimization accuracy.
Geometric quantization adapted to polysymplectic manifolds.
problem Quantization of polysymplectic manifolds.
method Adapted geometric quantization framework to polysymplectic setting.
result Polysymplectic Guillemin-Sternberg conjecture is shown to be false with a complex polarization.
The study analyzes convergence of adaptive optimizers under low-precision training.
problem Understanding why low-precision training remains effective for large models.
method Developed a theoretical framework for analyzing convergence of adaptive optimizers under floating-point quantization.
result Adaptive optimizers retain convergence rates close to full-precision methods under logarithmic mantissa scaling.
AdaRound improves post-training quantization of neural networks.
problem Improving the accuracy of quantized weights in neural networks.
method Adaptive rounding mechanism that adapts to data and task loss.
result AdaRound outperforms rounding-to-nearest and achieves state-of-the-art performance.
This work proposes a novel approach to learn quantizers from data, improving similarity search performance.
problem Learning optimal quantizers for multi-dimensional data distributions.
method Train a neural net to form a fixed parameter-free quantizer, using uniformity in a spherical latent space as a proxy objective.
result The proposed method outperforms most learned quantization methods and is competitive with state-of-the-art approaches.
We formulate the notion of minimax estimation under storage or communication constraints, and prove an extension to Pinsker's theorem for nonparametric estimation over Sobolev ellipsoids. Placing limits on the number of bits used to encode any estimator, we give tight lower and upper bounds on the excess risk due to qu…
Quantizes neural networks to eliminate multiplications, achieving high accuracy.
problem Efficiently implementing neural networks on hardware with limited resources.
method Replace activation-weight multiplication with table-based lookup for multiply-free inference.
result Achieves high accuracy with minimal multiplications, outperforming other multiply-free networks.
Adaptive Quantization Modules enable online continual compression of non-i.i.d data streams.
problem Learning to compress and store a dataset from a non-i.i.d data stream, only observing each sample once.
method Discrete auto-encoders and Adaptive Quantization Modules (AQM) to control compression ability.
result Significant gains on continual learning benchmarks with AQM replacing episodic memory.
New protocols show 1-bit mean estimation can be order-optimal without interaction.
problem Can 1-bit mean estimation be optimal without interaction?
method Adaptive and non-adaptive threshold and interval queries, with one adaptive transition.
result Arbitrary non-adaptive quantizers can match the adaptive rate, suggesting interaction is not necessary.
The existence of a natural and projectively invariant quantization in the sense of P. Lecomte [Progr. Theoret. Phys. Suppl. (2001), no. 144, 125-132] was proved by M. Bordemann [math.DG/0208171], using the framework of Thomas-Whitehead connections. We extend the problem to the context of supermanifolds and adapt M. Bor…
A new method for robust product Markovian quantization overcomes numerical instabilities.
problem Numerical instabilities in the PMQ algorithm limit its adoption, especially for stochastic volatility models.
method Reformulated PMQ as standard vector quantization, applying accelerated Lloyd's algorithm for robustness.
result The method overcomes numerical instabilities and extends applicability to stochastic volatility models.
Optimizes deep neural network quantization for mobile devices.
problem High computational costs and memory consumption in deep neural networks.
method Proposes a measurement to estimate quantization errors and an optimization process for finding optimal bit-widths.
result Achieves 20-40% higher compression rate compared to previous methods at the same accuracy.
Proposes QEP to mitigate quantization error propagation in layer-wise post-training quantization.
problem Growth of quantization errors across layers degrades performance, especially in low-bit regimes.
method Quantization Error Propagation (QEP) framework that explicitly propagates and compensates for quantization errors.
result QEP-enhanced layer-wise PTQ achieves substantially higher accuracy, especially in low-bit regimes.
DFS dynamically decides bitwidths for layers to balance accuracy and efficiency.
problem Balancing model accuracy and inference speed for deep networks.
method Dynamic Fractional Skipping (DFS) framework that assigns bitwidths to layers for input-adaptive inference.
result DFS achieves superior tradeoff between computational cost and model accuracy.
A new method reduces communication in distributed learning by skipping less informative gradient updates.
problem Efficient communication in distributed machine learning.
method Quantizes and skips less informative gradients to reduce communication overhead.
result Proves linear convergence rate similar to gradient descent with significant communication savings.
The paper offers error bounds for quantized dynamical models.
problem Accuracy of dynamical models from dependent data sequences.
method Developed uniform error bounds for quantized models and imperfect optimization algorithms.
result Unified bounds for slow and fast rates, scaling with model encoding bits.
A quantization over a manifold can be seen as a way to construct a differential operator with prescribed principal symbol. The quantization map is moreover required to be a linear bijection. It is known that there is in general no natural quantization procedure. However, considering manifolds endowed with additional st…
New approach quantizes neural nets with guaranteed convergence to loss-optimal states.
problem Optimizing deep neural net compression by quantizing weights.
method Model compression as constrained optimization framework, alternating learning and quantization.
result Guaranteed convergence to local optimum of loss for quantized nets, achieving high compression rates.
Post-training quantization method using multiple low-precision points achieves higher precision for critical weights.
problem Discretizing pre-trained deep neural networks without re-training.
method Multipoint quantization with efficient greedy selection and adaptive point number.
result Outperforms state-of-the-art methods on ImageNet classification and PASCAL VOC object detection.
In the first part of this article we provide a geometrically oriented approach to the theory of orbispaces which originally had been introduced by Chen. We explain the notion of a vector orbibundle and characterize the good sections of a reduced vector orbibundle as the smooth stratified sections. In the second part of…
A new algorithm for compressing latent representations in deep models.
problem Compressing continuous latent representations in deep models.
method Separates model design and training from quantization; uses adaptive quantization based on posterior uncertainty.
result Image compression with the proposed algorithm outperforms JPEG over a wide range of bit rates.
Q-GADMM reduces communication in decentralized ML by quantizing model updates.
problem Reducing communication in decentralized ML while maintaining accuracy.
method Quantized group ADMM (Q-GADMM) with adaptive quantization.
result Q-GADMM achieves similar accuracy and convergence to non-quantized methods with less communication.
Distributed sensors compress and send features to a fusion center for linear regression.
problem Efficiently compress and transmit features from distributed sensors to a fusion center under varying communication constraints.
method Designs a distributed and adaptive feature compression scheme using optimal quantizers and simple adaptive strategies.
result Demonstrates improved inference performance through simulated experiments.
Paper proposes a new method for quantizing channel state information to optimize resource allocation.
problem Optimizing resource allocation for a receiver sending quantized channel state information to a transmitter.
method Introducing a task-oriented approach where the receiver sends the right amount of information to the transmitter.
result Analytical solution for optimal task-oriented CSI quantizer found for a specific energy-efficient power control problem.
The concept of conformally equivariant quantizations was introduced by Duval, Lecomte and Ovsienko in \cite{DLO} for manifolds endowed with flat conformal structures. They obtained results of existence and uniqueness (up to normalization) of such a quantization procedure. A natural generalization of this concept is to …
MQGrad uses reinforcement learning to dynamically adjust gradient quantization bits.
problem Reduction of communication overhead in large-scale machine learning model training.
method Reinforcement learning applied to gradient quantization in parameter server.
result MQGrad accelerates deep neural network learning while maintaining prediction accuracy.
We investigate the concept of equivariant quantization over the superspace R^{p+q|2r}, with respect to the orthosymplectic algebra osp(p+1,q+1|2r). Our methods and results vary upon the superdimension p+q-2r. When the superdimension is nonzero, we manage to obtain a result which is similar to the classical theorem of D…
Develops a faster soybean genome clustering method combining spectral and vector quantization.
problem Clustering soybean whole genome sequences efficiently.
method Combines Spectral Clustering and Vector Quantization for computational efficiency.
result Significantly outperforms existing methods in cluster quality and time complexity.
ReLeQ automates deep quantization of neural networks, preserving accuracy and reducing resource usage.
problem Deep Neural Networks require massive computation for inference, and quantization can reduce bitwidth but often leads to accuracy loss.
method ReLeQ uses deep reinforcement learning to automate the selection of quantization levels for each layer, balancing speed and accuracy.
result ReLeQ preserves accuracy (<= 0.3% loss) while significantly reducing computation and storage costs for various deep networks.
VQ-DRAW compresses images and generates realistic samples.
problem Learning compact discrete representations of images.
method Sequential discrete VAE with vector quantization.
result VQ-DRAW effectively compresses and generates images.
WaveQ uses sinusoidal regularization to optimize deep quantization for neural networks, improving both efficiency and accuracy.
problem Deep quantization reduces bitwidth but can lead to significant accuracy loss due to inter-layer dependencies.
method WaveQ employs sinusoidal regularization to learn multiple quantization parameters during gradient-based training, balancing compute efficiency and accuracy.
result WaveQ achieves accuracy preservation and efficiency gains across various deep networks, outperforming state-of-the-art techniques.
BCGD algorithm improves training of quantized neural networks.
problem Training quantized deep neural networks at low bit-widths.
method Introduces coarse gradient descent and blended correction for training.
result BCGD achieves high accuracy in quantized neural networks.
A new method improves quantile regression for high-dimensional data.
problem Handling heteroscedastic, multimodal, or skewed data in quantile regression.
method Dynamic prototypes-based probability density estimation with conformalized high-density quantile regression.
result Enhanced prediction regions with valid coverage guarantees and scalability to higher dimensions.
Improved vector quantization using Gaussian mixtures for better codebook utilization.
problem Training instability and information loss in discrete vector quantization.
method Generalized vector quantization with Gaussian mixture model and aggregated categorical posterior evidence lower bound.
result GM-VQ improves codebook utilization and reduces information loss without heuristics.
The paper proposes a nonparametric test for incomplete samples quantized to B bits.
problem Statistical inference with lossy or incomplete samples.
method Nonparametric testing procedure based on B-bit quantized samples.
result The proposed test achieves the classical minimax rate of testing for spline models when B exceeds a threshold.
Winograd convolutions are used to improve quantized neural networks.
problem Improving quantized neural networks using Winograd convolutions.
method Proposed a Winograd-aware formulation of convolution layers to expose numerical inaccuracies to learning.
result Up to 10% higher classification accuracy on CIFAR-10 with Winograd-aware layers.
Paper proves existence of a universal codebook for low-precision quantization.
problem Optimizing low-precision approximation of matrix products in machine learning.
method Develops a universal codebook that is near-optimal for all possible statistics of input data.
result Proves existence of a universal codebook with a 0.11 bit per dimension reduction in rate.
Estimating mean from one-bit samples of symmetric log-concave distributions.
problem Estimating the mean of a symmetric log-concave distribution with limited one-bit measurements.
method Analyzes mean squared error in three settings: centralized, adaptive, and distributed, with and without quantization.
result One round of adaptivity is sufficient to achieve optimal mean-square error in the adaptive setting.
GLAD improves latent graph generation by quantizing discrete latent space.
problem Latent space graph generative models lack performance and make unnatural assumptions.
method Adapting diffusion bridges to a discrete latent space, avoiding data space decompositions.
result GLAD achieves competitive performance on graph benchmark datasets.
Survey on quantization methods on Kähler manifolds.
problem None explicitly stated; focuses on methods.
method Deformation quantization, geometric quantization, Berezin-Toeplitz quantization, BV quantization.
result New relationships among quantization methods on Kähler manifolds.
We show that using the family of adapted Kähler polarizations of the phase space of a compact, simply connected, Riemannian symmetric space of rank-1, the obtained field Hcorr of quantum Hilbert spaces produced by geometric quantization including the half-form correction is flat if M is the 3-dimensional sphere …
AIS corrects rollout-training mismatch in quantized RL, improving speed and stability.
problem Rollout-training mismatch in quantized RL causes bias and training collapse.
method Adaptive Importance Sampling (AIS) adjusts gradient correction per batch.
result AIS matches BF16 baseline on most tasks while improving speed.
The paper classifies quantizable functions and explores symmetry in quantization methods.
problem Classifying quantizable functions and understanding symmetry in quantization methods.
method Deformation quantization and geometric quantization methods are compared and classified.
result Formal quantizable functions are of a specific form and relate to Hamiltonian Killing vector fields.