Low-precision training reduces computational cost and produces efficient models. Recent research in developing new low-precision training algorithms often relies on simulation to empirically evaluate the statistical effects of quantization while avoiding the substantial overhead of building specific hardware. To suppor…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Low-Precision Arithmetic”
The use of low-precision fixed-point arithmetic along with stochastic rounding has been proposed as a promising alternative to the commonly used 32-bit floating point arithmetic to enhance training neural networks training in terms of performance and energy efficiency. In the first part of this paper, the behaviour of …
Efficient Winograd convolution for INT8 networks using RNS.
problem Difficulty in applying Winograd algorithm to low-precision quantized networks.
method Extends Winograd algorithm to Residue Number System (RNS) for efficient INT8 convolution.
result Arithmetic complexity reduction up to 7.03x with performance improvement up to 2.30x-4.69x.
Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a cruc…
Deep neural networks (DNN) are powerful models for many pattern recognition tasks, yet their high computational complexity and memory requirement limit them to applications on high-performance computing platforms. In this paper, we propose a new method to evaluate DNNs trained with 32bit floating point (float32) accura…
Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towa…
Degree-Quant improves GNN efficiency by quantizing them without losing accuracy.
problem Efficiency of graph neural networks at inference time.
method Architecturally-agnostic method, Degree-Quant, for quantizing GNNs.
result Degree-Quant trained models perform as well as full-precision models and achieve up to 26% gains.
This paper explains how low-precision arithmetic causes loss spikes in deep learning models.
problem Loss spikes during long-term training of deep neural networks.
method Analyzes the impact of floating-point precision limits on gradient updates and feature means.
result Numerical Feature Inflation (NFI) explains loss spikes and rapid parameter norm growth.
Paper explores low-precision SGLD for neural networks, reducing costs without sacrificing performance.
problem Infeasibility of low-precision sampling in large-scale scenarios.
method Developed low-precision SGLD with quantization function and full-precision gradient accumulators.
result Low-precision SGLD achieves comparable performance to full-precision SGLD with only 8 bits.
This paper improves low-precision sampling using SGHMC for deep learning models.
problem Enhancing training efficiency of deep neural networks with low-precision training.
method Investigates low-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) for both log-concave and non-log-concave distributions.
result Low-precision SGHMC achieves quadratic improvement in error compared to SGLD for non-log-concave distributions.
Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precisi…
Deploying Customized Data Representation and Approximate Computing in Machine Learning Applicationscs.LG
Major advancements in building general-purpose and customized hardware have been one of the key enablers of versatility and pervasiveness of machine learning models such as deep neural networks. To sustain this ubiquitous deployment of machine learning models and cope with their computational and storage complexity, se…
Low-precision computation is often used to lower the time and energy cost of machine learning, and recently hardware accelerators have been developed to support it. Still, it has been used primarily for inference - not training. Previous low-precision training algorithms suffered from a fundamental tradeoff: as the num…
The study analyzes numerical stability in large language models using mixed-precision arithmetic.
problem Numerical stability of large language models using low-precision arithmetic.
method Developed a mixed-precision analysis of transformer inference, deriving bounds for condition numbers and forward error.
result Established that numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream.
Low-precision streaming PCA estimates the leading eigenvector with limited precision.
problem Estimating the leading eigenvector in a streaming setting with limited precision.
method Oja's algorithm with linear and nonlinear stochastic quantization.
result A batched version of the quantized variants achieves the lower bound on quantization error up to logarithmic factors.
GNMR controls runtime stability in low-precision language model training.
problem Efficient low-precision training faces numerical risks at specific operators.
method GNMR compares gradient norms to historical means, applying bounded recovery actions.
result GNMR preserves high-fidelity quality with sparse, budgeted recovery.
Log-Normal Multiplicative Dynamics improves low-precision training of neural networks.
problem Training large neural networks with low precision is unstable.
method Derive a Bayesian learning rule with log-normal posterior distributions and multiplicative updates.
result LMD achieves stable and accurate training for Vision Transformer and GPT-2.
We propose K-TanH, a novel, highly accurate, hardware efficient approximation of popular activation function TanH for Deep Learning. K-TanH consists of parameterized low-precision integer operations, such as, shift and add/subtract (no floating point operation needed) where parameters are stored in very small look-up t…
Gradient descent stagnates in low-precision, but unbiased rounding schemes improve convergence.
problem Stagnation of gradient descent in low-precision computation.
method Proposed unbiased stochastic rounding schemes that trade zero bias for larger probability of preserving small gradients.
result Unbiased rounding methods typically improve convergence rate of gradient descent for convex problems.
SINGD improves KFAC for memory-efficiency and stability in low-precision training.
problem Memory inefficiency and numerical instability of KFAC in low-precision training.
method Formulated inverse-free KFAC update and imposed structures in Kronecker factors.
result SINGD is memory-efficient and numerically robust, often outperforming AdamW in half precision.
Low precision networks in the reinforcement learning (RL) setting are relatively unexplored because of the limitations of binary activations for function approximation. Here, in the discrete action ATARI domain, we demonstrate, for the first time, that low precision policy distillation from a high precision network pro…
The study analyzes convergence of adaptive optimizers under low-precision training.
problem Understanding why low-precision training remains effective for large models.
method Developed a theoretical framework for analyzing convergence of adaptive optimizers under floating-point quantization.
result Adaptive optimizers retain convergence rates close to full-precision methods under logarithmic mantissa scaling.
State of the art deep learning models have made steady progress in the fields of computer vision and natural language processing, at the expense of growing model sizes and computational complexity. Deploying these models on low power and mobile devices poses a challenge due to their limited compute capabilities and str…
Low precision weights, activations, and gradients have been proposed as a way to improve the computational efficiency and memory footprint of deep neural networks. Recently, low precision networks have even shown to be more robust to adversarial attacks. However, typical implementations of low precision DNNs use unifor…
This work studies scaling laws for low-precision training in high-dimensional linear regression.
problem Optimizing trade-off between model quality and training costs in high-dimensional linear regression.
method Theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression framework, analyzing multiplicative and additive quantization.
result Multiplicative quantization maintains full-precision model size, while additive quantization reduces effective model size.
Recently, the posit numerical format has shown promise for DNN data representation and compute with ultra-low precision ([5..8]-bit). However, majority of studies focus only on DNN inference. In this work, we propose DNN training using posits and compare with the floating point training. We evaluate on both MNIST and F…
Neural networks with low-precision weights and activations offer compelling efficiency advantages over their full-precision equivalents. The two most frequently discussed benefits of quantization are reduced memory consumption, and a faster forward pass when implemented with efficient bitwise operations. We propose a t…
Designing a deep neural network (DNN) with good generalization capability is a complex process especially when the weights are severely quantized. Model averaging is a promising approach for achieving the good generalization capability of DNNs, especially when the loss surface for training contains many sharp minima. W…
Paper proves existence of a universal codebook for low-precision quantization.
problem Optimizing low-precision approximation of matrix products in machine learning.
method Develops a universal codebook that is near-optimal for all possible statistics of input data.
result Proves existence of a universal codebook with a 0.11 bit per dimension reduction in rate.
Low-precision DNNs have been extensively explored in order to reduce the size of DNN models for edge devices. Recently, the posit numerical format has shown promise for DNN data representation and compute with ultra-low precision in [5..8]-bits. However, previous studies were limited to studying posit for DNN inference…
Recently there has been significant interest in training machine-learning models at low precision: by reducing precision, one can reduce computation and communication by one order of magnitude. We examine training at reduced precision, both from a theoretical and practical perspective, and ask: is it possible to train …
Gradient descent with biased rounding errors converges faster under certain conditions.
problem Stagnation or negative impact of rounding errors in neural network training with low precision.
method Analysis of gradient descent with stochastic fixed-point rounding errors under the Polyak-Lojasiewicz inequality.
result Biased rounding errors can improve convergence rates, especially when the Polyak-Lojasiewicz inequality holds.
CALDERA compresses large language models by approximating weight matrices with low-rank, low-precision factors.
problem The large sizes of Large Language Models (LLMs) hinder deployment on edge devices.
method CALDERA approximates weight matrices as , where and are low-rank factors quantized to low precision.
result CALDERA achieves better zero-shot performance than existing techniques, especially with low bit precision.
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
The Straight-Through Estimator (STE) is widely used for back-propagating gradients through the quantization function, but the STE technique lacks a complete theoretical understanding. We propose an alternative methodology called alpha-blending (AB), which quantizes neural networks to low-precision using stochastic grad…
Algorithm compresses large matrices by approximating them as low rank and low precision factors.
problem Efficiently storing and processing large matrices with billions of elements.
method Randomized sketching and quantization of matrix columns to achieve low rank and low precision factorization.
result Achieves compression ratios as low as one bit per matrix coordinate while maintaining or improving performance.
A plethora of recent research has focused on improving the memory footprint and inference speed of deep networks by reducing the complexity of (i) numerical representations (for example, by deterministic or stochastic quantization) and (ii) arithmetic operations (for example, by binarization of weights). We propose a s…
Deep neural networks (DNNs) can be made hardware-efficient by reducing the numerical precision of the weights and activations of the network and by improving the network's resilience to noise. However, this gain in efficiency often comes at the cost of significantly reduced accuracy. In this paper, we present a novel a…
In this article, we investigate when the set of primitive geodesic lengths on a Riemannian manifold have arbitrarily long arithmetic progressions. We prove that in the space of negatively curved metrics, a metric having such arithmetic progressions is quite rare. We introduce almost arithmetic progressions, a coarsific…
Simplified optimization for structured matrices in deep learning.
problem Computational challenges in Riemannian submanifold optimization for structured symmetric positive-definite matrices.
method Proposed a generalized Riemannian normal coordinates that dynamically orthonormalizes the metric and converts the problem into an unconstrained Euclidean space problem.
result Simplified existing approaches for structured covariances and developed matrix-inverse-free 2nd-order optimizers for deep learning with low precision.
Develops arithmetic PDE geometry concepts like curvature and cohomology.
problem Creating a geometry framework for arithmetic PDEs.
method Introducing arithmetic analogues of Levi-Civita and Chern connections, then developing curvature and characteristic classes.
result Arithmetic analogues of curvature and characteristic classes have been developed.
Mixed-precision CA-SGD for generalized linear models on GPUs
problem SGD communication bottleneck
method Mixed-precision CA-SGD
result Matches FP32 SGD loss within 0.5% on various problems
New geometric invariant limits the number of semi-arithmetic groups.
problem Understanding the structure of semi-arithmetic Fuchsian groups.
method Introducing a new geometric invariant called stretch and using the arithmetic Margulis lemma.
result There exist only finitely many conjugacy classes of semi-arithmetic groups with bounded arithmetic dimension, stretch, and coarea.
Arithmetic groupsmath.GT
Course on arithmetic lattices at EPFL.
problem Understanding arithmetic lattices.
method Introductory course on arithmetic lattices.
result Introduction to arithmetic lattices.
We investigate how to train kernel approximation methods that generalize well under a memory budget. Building on recent theoretical work, we define a measure of kernel approximation error which we find to be more predictive of the empirical generalization performance of kernel approximation methods than conventional me…
Paper shows non-arithmetic surface with unique geometric property.
problem Non-arithmetic surfaces with unique geometric properties.
method Example of a non-arithmetic surface with marked length variety rigidity.
result Found a non-arithmetic surface with marked length variety rigidity.
New classification of hyperbolic Coxeter prisms.
problem Classifying hyperbolic Coxeter prisms.
method Determine which prisms are quasi-arithmetic or arithmetic.
result New insights into commensurability and systoles of associated orbifolds.
Arithmetic Dijkgraaf-Witten theory constructs analogues in Chern-Simons TQFT.
problem Developing arithmetic analogues in Chern-Simons TQFT.
method Constructing arithmetic analogues of Chern-Simons 1-cocycle, prequantization bundle, and Chern-Simons functional.
result Decomposition and gluing formulas for arithmetic Chern-Simons invariants and arithmetic Dijkgraaf-Witten partition functions.