A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Quantized Adam reduces communication cost in deep learning training.
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
Distributed quantization improves classification accuracy with less data.
Optimal gradient quantization reduces communication costs in distributed deep learning.
Quantized Stochastic Primal-Dual Methods for Distributed Optimization
Due to its efficiency and ease to implement, stochastic gradient descent (SGD) has been widely used in machine learning. In particular, SGD is one of the most popular optimization methods for distributed learning. Recently, quantized SGD (QSGD), which adopts quantization to reduce the communication cost in SGD-based di…
In this paper, we study the randomized distributed coordinate descent algorithm with quantized updates. In the literature, the iteration complexity of the randomized distributed coordinate descent algorithm has been characterized under the assumption that machines can exchange updates with an infinite precision. We con…
New quantization methods improve accuracy of Random Fourier Features.
We propose Additive Powers-of-Two~(APoT) quantization, an efficient non-uniform quantization scheme for the bell-shaped and long-tailed distribution of weights and activations in neural networks. By constraining all quantization levels as the sum of Powers-of-Two terms, APoT quantization enjoys high computational effic…
Global-QSGD accelerates distributed training by up to 3.51%.
With the development of deep neural networks, the size of network models becomes larger and larger. Model compression has become an urgent need for deploying these network models to mobile or embedded devices. Model quantization is a representative model compression technique. Although a lot of quantization methods hav…
The present paper develops a novel aggregated gradient approach for distributed machine learning that adaptively compresses the gradient communication. The key idea is to first quantize the computed gradients, and then skip less informative quantized gradient communications by reusing outdated gradients. Quantizing and…
Modern distributed training of machine learning models suffers from high communication overhead for synchronizing stochastic gradients and model parameters. In this paper, to reduce the communication complexity, we propose \emph{double quantization}, a general scheme for quantizing both model parameters and gradients. …
A simple and computationally efficient scheme for tree-structured vector quantization is presented. Unlike previous methods, its quantization error depends only on the intrinsic dimension of the data distribution, rather than the apparent dimension of the space in which the data happen to lie.
Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network qu…
A new gradient quantization scheme improves communication efficiency in distributed training.
Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can…
We consider the problem of decentralized consensus optimization, where the sum of smooth and strongly convex functions are minimized over distributed agents that form a connected network. In particular, we consider the case that the communicated local decision variables among nodes are quantized in order to all…
Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. …
We study the dependence of geometric quantization of the standard symplectic torus on the choice of invariant polarization. Real and mixed polarizations are interpreted as degenerate complex structures. Using a weak version of the equations of covariant constancy, and the Weil-Brezin expansion to describe distributiona…
We present Rotated Adaptive Tetra-iterated Quantizer (RATQ), a fixed-length quantizer for gradients in first order stochastic optimization. RATQ is easy to implement and involves only a Hadamard transform computation and adaptive uniform quantization with appropriately chosen dynamic ranges. For noisy gradients with al…
The paper studies quantization on symplectic manifolds with real polarizations, comparing different quantization methods.
We present a novel method for neural network quantization that emulates a non-uniform -quantile quantizer, which adapts to the distribution of the quantized parameters. Our approach provides a novel alternative to the existing uniform quantization techniques for neural networks. We suggest to compare the results as …
Paper proposes a 1-bit quantization scheme for high-dimensional statistical estimation.
vqSGD reduces communication in distributed optimization with convergence guarantees.
A new method quantizes conditional probability measures using deep learning.
Quantized-TinyLLaVA reduces communication costs in split learning for multimodal models.
Paper quantizes heavy-tailed data for near optimal estimation rates.
QDSB accelerates Schrödinger bridge learning with quantized approximations.
Paper shows how to integrate quantization into neural compression models.
Differentially quantized gradient methods improve convergence in noisy communication channels.
High-risk domains require reliable confidence estimates from predictive models. Deep latent variable models provide these, but suffer from the rigid variational distributions used for tractable inference, which err on the side of overconfidence. We propose Stochastic Quantized Activation Distributions (SQUAD), which im…
Low-precision streaming PCA estimates the leading eigenvector with limited precision.
Adaptive quantization improves SGD accuracy in data-parallel settings.
Training large machine learning models requires a distributed computing approach, with communication of the model updates being the bottleneck. For this reason, several methods based on the compression (e.g., sparsification and/or quantization) of updates were recently proposed, including QSGD (Alistarh et al., 2017), …
New algorithm improves clustering and quantization using MMD.
Bayesian Bits unifies quantization and pruning through gradient optimization.
A new deep learning framework for efficient IoT data compression and inference.
Quantization techniques have been applied in many challenging finance applications, including pricing claims with path dependence and early exercise features, stochastic optimal control, filtering problems and efficient calibration of large derivative books. Recursive Marginal Quantization of the Euler scheme has recen…
BiTAT improves neural network quantization for edge devices by focusing on weight dependencies and disentangling them.
Discretizing multi-dimensional data distributions is a fundamental step of modern indexing methods. State-of-the-art techniques learn parameters of quantizers on training data for optimal performance, thus adapting quantizers to the data. In this work, we propose to reverse this paradigm and adapt the data to the quant…
This paper studies parallelization schemes for stochastic Vector Quantization algorithms in order to obtain time speed-ups using distributed resources. We show that the most intuitive parallelization scheme does not lead to better performances than the sequential algorithm. Another distributed scheme is therefore intro…
Recent research implies that training and inference of deep neural networks (DNN) can be computed with low precision numerical representations of the training/test data, weights and gradients without a general loss in accuracy. The benefit of such compact representations is twofold: they allow a significant reduction o…
Improved vector quantization using Gaussian mixtures for better codebook utilization.
It is of fundamental importance to find algorithms obtaining optimal performance for learning of statistical models in distributed and communication limited systems. Aiming at characterizing the optimal strategies, we consider learning of Gaussian Processes (GPs) in distributed systems as a pivotal example. We first ad…
Efficient decentralized learning framework reduces communication costs.
A new framework enhances generative modeling by learning local flows over complex manifolds.
Post-training quantization saves resources for neural networks.