The paper proposes a nonparametric test for incomplete samples quantized to B bits.
problem Statistical inference with lossy or incomplete samples.
method Nonparametric testing procedure based on B-bit quantized samples.
result The proposed test achieves the classical minimax rate of testing for spline models when B exceeds a threshold.
In this paper, we propose to (seamlessly) integrate b-bit minwise hashing with linear SVM to substantially improve the training (and testing) efficiency using much smaller memory, with essentially no loss of accuracy. Theoretically, we prove that the resemblance matrix, the minwise hashing matrix, and the b-bit minwise…
Minwise hashing is the standard technique in the context of search and databases for efficiently estimating set (e.g., high-dimensional 0/1 vector) similarities. Recently, b-bit minwise hashing was proposed which significantly improves upon the original minwise hashing in practice by storing only the lowest b bits of e…
We generated a dataset of 200 GB with 10^9 features, to test our recent b-bit minwise hashing algorithms for training very large-scale logistic regression and SVM. The results confirm our prior work that, compared with the VW hashing algorithm (which has the same variance as random projections), b-bit minwise hashing i…
In this paper, we first demonstrate that b-bit minwise hashing, whose estimators are positive definite kernels, can be naturally integrated with learning algorithms such as SVM and logistic regression. We adopt a simple scheme to transform the nonlinear (resemblance) kernel into linear (inner product) kernel; and hence…
Paper presents a new trie for integer sketches to improve similarity searches.
problem Efficient similarity searches on integer sketches.
method Introduces a novel b-bit sketch trie that leverages succinct data structures. result Significantly improves search time and space-efficiency of similarity searches.
New method tightens federated probe-logit distillation rates under varying bandwidths.
problem Estimating conditional distributions in federated learning with heterogeneous bandwidth constraints.
method Developed a new federated probe-logit distillation (FPLD) method with optimal allocation for varying bandwidths.
result Achieved matching lower and upper bounds for the minimax rate under heterogeneous bandwidths.
Large-scale regression problems where both the number of variables, p, and the number of observations, n, may be large and in the order of millions or more, are becoming increasingly more common. Typically the data are sparse: only a fraction of a percent of the entries in the design matrix are non-zero. Neverthele…
Survey on quantization methods on Kähler manifolds.
problem None explicitly stated; focuses on methods.
method Deformation quantization, geometric quantization, Berezin-Toeplitz quantization, BV quantization.
result New relationships among quantization methods on Kähler manifolds.
The paper classifies quantizable functions and explores symmetry in quantization methods.
problem Classifying quantizable functions and understanding symmetry in quantization methods.
method Deformation quantization and geometric quantization methods are compared and classified.
result Formal quantizable functions are of a specific form and relate to Hamiltonian Killing vector fields.
This paper introduces a differentiable, scalable quantization method for neural networks.
problem Previous quantization methods lacked differentiability and scalability.
method The approach is differentiable and scalable, using bit-shifting and logarithmic quantization.
result The method achieves comparable accuracy to state-of-the-art approaches with less training time and lower inference cost.
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
Smart Quantization adapts binary and ternary quantization for neural networks.
problem Resource constraints in deploying neural networks on devices with limited resources.
method Adaptive combination of binary and ternary quantization with a regularization function.
result Adapts quantization depth during training to maintain high model accuracy.
This study optimizes quantized neural networks by considering model architecture and quantization types.
problem Optimizing quantized neural networks for low-power, high-throughput applications.
method Holistic approach including training methods and quantization-friendly architecture design.
result Deeper models are more sensitive to activation quantization, while wider models improve resilience to both weight and activation quantization.
RATQ is a new quantizer for optimizing noisy gradients in machine learning.
problem Optimizing noisy gradients in stochastic optimization.
method RATQ uses Hadamard transform and adaptive uniform quantization, and achieves near-optimal performance.
result RATQ nearly achieves information theoretic lower bounds for optimization accuracy.
Extends ONNX for quantized neural networks with new formats and operators.
problem Handling arbitrary-precision quantization in neural networks.
method Introduces new formats and operators in ONNX to represent quantized neural networks.
result Enabled representation of uniform quantization in neural networks.
New method for quantizing symplectic manifolds with Lagrangian bundles.
problem Quantization of symplectic manifolds with Lagrangian bundles.
method A new construction of strict deformation quantization.
result Established a correspondence between differential operators and principal symbols.
HMQ improves quantization for edge devices with mixed precision.
problem Efficient quantization for edge devices with uniform, power-of-two thresholds.
method Introduces HMQ, a mixed precision quantization block that repurposes Gumbel-Softmax for searching over quantization schemes.
result Achieves competitive and state-of-the-art results on ImageNet despite restrictions.
Introduces sheaf quantization, a topological approach to geometric quantization.
problem Topological realization of WKB-states in geometric quantization.
method Enhancement of constructible sheaves, Betti counterpart of Fukaya--Floer theory.
result Introduction to sheaf quantization as a topological realization of WKB-states.
Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network qu…
The article defines and compares two types of quantizations on compact manifolds.
problem Quantization on arbitrary compact smooth manifolds.
method Embedding into CP^n and inducing quantizations from there.
result Generalizations of earlier quantization methods.
Quantized Adam reduces communication cost in deep learning training.
problem Reducing communication cost in distributed deep learning training.
method Gradient and weight quantization with error feedback in Adam.
result Proposed methods converge to first-order stationary points.
Proposes a robust neural network quantization method.
problem Training model's dependency on specific quantization methods.
method Intrinsic robustness to various quantization processes.
result Single model capable of operating at various bit-widths and policies.
The paper studies quantization on symplectic manifolds with real polarizations, comparing different quantization methods.
problem Quantization on compact symplectic manifolds with real polarizations.
method Geometric quantization, Toeplitz operators, Fourier transforms, asymptotic expansion of traces.
result Deformation quantization is realized through asymptotic traces of Toeplitz operators.
Quantizes neural networks using frame theory for improved accuracy.
problem Improving neural network efficiency and accuracy through quantization.
method Sigma-Delta (ΣΔ) quantization with finite unit-norm tight frames. result Error bound between original and quantized neural networks derived.
Unified finetuning of all quantization degrees of freedom achieves state-of-the-art 4-bit quantization.
problem Achieving high accuracy in quantized neural networks while maintaining speed and resource constraints.
method Quantization-aware finetuning (QFT) that jointly optimizes all quantization degrees of freedom.
result 4-bit weight quantization results on-par with state-of-the-art (SoTA) within PTQ constraints.
Abstract proposes a new categorical approach to quantization of Poisson algebras.
problem Quantization of Poisson algebras.
method Defining quantization categories as subcategories of R-module categories with classical limits.
result Categories of strict deformation quantization, prequantization, and matrix regularization are equivalent, while Poisson enveloping algebra is not.
Geometric quantization of a Poisson manifold need not imply quantization of its symplectic leaves. We provide the leafwise geometric quantization of a Poisson manifold, seen as a foliated one, whose quantum algebra restricted to each leaf is quantized.
Meta learning optimizes neural network quantization for efficient inference.
problem Uniform bitwidth quantization is sub-optimal for neural network compression.
method Meta learning to automatically generate hybrid quantization policies.
result Meta learning outperforms uniform quantization and RL approaches.
Study quantization effects on high-dimensional linear regression learning.
problem Understanding quantization's impact on learning high-dimensional linear regression models.
method Analyzes stochastic gradient descent for high-dimensional linear regression under various quantization targets.
result Establishes precise bounds on excess risk for different quantization schemes.
We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks…
FrostNet improves INT8 quantization efficiency in mobile networks.
problem The importance of network architecture for optimal INT8 quantization.
method Quantization-aware training (QAT) with StatAssist and GradBoost, hardware-aware NAS.
result FrostNets achieve higher recognition accuracy with comparable latency when quantized.
A new method quantizes LSTM gate parameters without performance loss.
problem Quantization loss in LSTM gate parameters without performance degradation.
method Lossy quantization of gate parameters during training, weight parameters adjust to offset quantization loss.
result F1 score decreased by only 0.7% on Named Entity Recognition dataset.
Optimal gradient quantization reduces communication costs in distributed deep learning.
problem High communication costs in distributed training of deep neural networks.
method Deduced optimal gradient quantization conditions for binary and multi-level quantization, developed novel schemes for dynamic quantization levels.
result Demonstrated superior performance of proposed quantization schemes on CIFAR and ImageNet datasets.
Quantizes symplectic manifolds with bounded geometry using Berezin-Toeplitz method.
problem Quantization of symplectic manifolds with bounded geometry.
method Berezin-Toeplitz quantization theory.
result Correct semiclassical limit achieved.
WNQ uses weight normalization to reduce quantization error in deep neural networks.
problem High quantization error in deep neural networks due to long-tail distribution of weights.
method Weight normalization to avoid long-tail distribution and reduce quantization error.
result WNQ achieves state-of-the-art performance on CIFAR-100 and ImageNet.
A new method defends neural networks from adversarial attacks.
problem Adversarial attacks on quantized neural networks.
method Defensive Quantization (DQ) method to control Lipschitz constant.
result DQ method defends neural networks from adversarial attacks and improves accuracy.
New method quantizes neural networks for mobile devices.
problem High computational and memory costs of deep neural networks.
method Formulates quantization as a differentiable function.
result Quantization networks outperform state-of-the-art methods.
Quantization-aware training can recover accuracy lost by post-training quantization.
problem Post-training quantization (PTQ) can fail sharply at aggressive bitwidths.
method A unified geometric framework that explains PTQ failure and QAT recovery.
result QAT has a useful bias that steers iterates back into the basin.
This study analyzes quantization in deep learning models using statistical physics methods.
problem The computational resource requirements for large-scale data analysis models.
method Typical case analysis from statistical physics, specifically the replica method.
result Optimal quantization width minimizes error and delays overfitting.
Paper develops a statistical framework for quantized training of deep neural networks.
problem Lack of theoretical understanding of gradient quantization in FQT.
method Presented a statistical framework for analyzing FQT algorithms, viewing quantized gradient as a stochastic estimator of QAT gradient.
result Developed two novel gradient quantizers with smaller variance than existing per-tensor quantizer.
In [3], the authors showed the existence and the uniqueness of a sl(m+1,\R)-equivariant quantization in the non-critical situations. The curved generalization of the sl(m+1,\R)-equivariant quantization is the natural and projectively equivariant quantization. In [1] and [7], the existence of such a quantization was pro…
A unified approach to geometric, symbol and deformation quantizations on a generalized flag manifold endowed with an invariant pseudo-Kaehler structure is proposed. The Hilbert space of states is realized via the Bott-Borel-Weil theorem in the sheaf cohomology of the geometric quantization line bundle. The correspondin…
A new method for robust product Markovian quantization overcomes numerical instabilities.
problem Numerical instabilities in the PMQ algorithm limit its adoption, especially for stochastic volatility models.
method Reformulated PMQ as standard vector quantization, applying accelerated Lloyd's algorithm for robustness.
result The method overcomes numerical instabilities and extends applicability to stochastic volatility models.
A new method improves maximum inner product search by locally decomposing residual vectors.
problem Maximum inner product search efficiency and accuracy.
method Local Orthogonal Decomposition (LOD) combined with multiscale quantization.
result LOD consistently achieves higher recall than previous methods under the same bitrates.
Memory-augmented neural networks (MANNs) refer to a class of neural network models equipped with external memory (such as neural Turing machines and memory networks). These neural networks outperform conventional recurrent neural networks (RNNs) in terms of learning long-term dependency, allowing them to solve intrigui…
Quantizes contact structures using dynamical methods.
problem Quantizing contact structures in a flat connection.
method Constructs a dynamical quantization using a flat connection on a Hilbert tractor bundle.
result Determines a contact tractor connection whose parallel sections determine a distinguished choice of Reeb dynamics.
Paper proposes a new binary quantization method for faster DNN inference.
problem Accelerating deep neural network inference on resource-limited devices.
method Quantized Compressed Sensing (QCS) for binary quantization.
result The proposed method preserves benefits of standard methods while reducing quantization error.