Paper proposes training deep neural networks with 8-bit floating point precision.
problem Challenges in training deep neural networks at 8-bit precision due to higher precision and dynamic range requirements.
method Proposes a method to train deep neural networks using 8-bit floating point for weights, activations, errors, and gradients. Introduces an enhanced loss scaling method and stochastic rounding technique.
result Demonstrates state-of-the-art accuracy across multiple datasets and workloads compared to full precision baseline.
Cheetah framework optimizes DNNs for edge devices using low-precision formats.
problem Reducing DNN model size for edge devices while maintaining accuracy.
method Mixed low-precision hardware and software co-design framework using posit and other formats.
result 16-bit posits outperform 16-bit floating point in training, and [5..8]-bit posits improve inference performance.
The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, un…
AdaptivFloat improves deep learning inference accuracy at low precision.
problem Low precision quantization issues in deep learning inference.
method Dynamic floating-point representation with adaptive clipping.
result Consistently higher inference accuracy at low precision compared to other methods.
We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks…
Posits improve DNN training efficiency on edge devices.
problem Training deep neural networks with low-precision formats.
method Used posits (5-8 bit) for DNN training compared to floating point.
result 16-bit posits outperform 16-bit floating point for end-to-end training.
Gradient descent stagnates in low-precision, but unbiased rounding schemes improve convergence.
problem Stagnation of gradient descent in low-precision computation.
method Proposed unbiased stochastic rounding schemes that trade zero bias for larger probability of preserving small gradients.
result Unbiased rounding methods typically improve convergence rate of gradient descent for convex problems.
This work proposes a complete 8-bit quantization framework for large-scale deep neural networks.
problem Training large-scale deep neural networks with high performance and low memory footprint.
method WAGEUBN framework that quantizes all data paths including weights, activations, gradients, errors, updates, and batch normalization.
result Achieves competitive accuracy on the ImageNet dataset using only 8-bit integers.
Memory-augmented neural networks (MANNs) refer to a class of neural network models equipped with external memory (such as neural Turing machines and memory networks). These neural networks outperform conventional recurrent neural networks (RNNs) in terms of learning long-term dependency, allowing them to solve intrigui…
This paper evaluates quantization techniques for deep learning inference.
problem Reducing the size and improving inference of deep neural networks.
method Review and empirical evaluation of quantization parameters for various neural network models.
result 8-bit quantization maintains accuracy within 1% of floating-point models.
Efficient Winograd convolution for INT8 networks using RNS.
problem Difficulty in applying Winograd algorithm to low-precision quantized networks.
method Extends Winograd algorithm to Residue Number System (RNS) for efficient INT8 convolution.
result Arithmetic complexity reduction up to 7.03x with performance improvement up to 2.30x-4.69x.
Improved machine translation with INT8 hardware using a novel training method.
problem Training accurate machine translation models with limited hardware precision.
method Convert all Transformer matrix multiplications to 8-bit integer (INT8) without sacrificing accuracy.
result INT8 Transformer models achieve BLEU scores 99.3% to 100% relative to FP32 models.
The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek t…
Researchers find floating point errors can mislead neural network verifiers.
problem Floating point arithmetic inaccuracies mislead neural network verifiers.
method Efficiently searches inputs and constructs neural network architectures to exploit verification errors.
result Floating point errors can systematically mislead neural network verifiers.
FullyQT quantizes Transformer models to 8-bit precision without sacrificing translation quality.
problem Reducing computational costs of neural machine translation methods without performance loss.
method Proposes FullyQT, an all-inclusive quantization strategy for the Transformer.
result Achieves state-of-the-art quantization results, maintaining or improving translation quality.
For successful deployment of deep neural networks on highly--resource-constrained devices (hearing aids, earbuds, wearables), we must simplify the types of operations and the memory/power resources used during inference. Completely avoiding inference-time floating-point operations is one of the simplest ways to design …
The use of low-precision fixed-point arithmetic along with stochastic rounding has been proposed as a promising alternative to the commonly used 32-bit floating point arithmetic to enhance training neural networks training in terms of performance and energy efficiency. In the first part of this paper, the behaviour of …
Low-bit training framework reduces energy consumption in CNNs.
problem Reducing energy consumption in convolutional neural networks.
method Low-bit training framework using MLS tensor format with dynamic quantization.
result Achieves superior trade-off between accuracy and bit-width.
Data-free quantization method improves model performance without fine-tuning.
problem Efficient quantization of deep neural networks without performance loss.
method Equalizing weight ranges and correcting biases to achieve 8-bit quantization.
result Achieves state-of-the-art quantized model performance on various architectures.
Constructs minimal immersions with singularities.
problem Minimal immersions with singularities in metric spaces.
method Constructs minimal immersions with catenoidal necks or floating disks converging to a singular point.
result Constructs minimal immersions with singularities.
Quantized Neural Networks (QNNs) are often used to improve network efficiency during the inference phase, i.e. after the network has been trained. Extensive research in the field suggests many different quantization schemes. Still, the number of bits required, as well as the best quantization scheme, are yet unknown. O…
Convolutional neural networks have recently achieved significant breakthroughs in various image classification tasks. However, they are computationally expensive,which can make their feasible mplementation on embedded and low-power devices difficult. In this paper convolutional neural network binarization is implemente…
Deep learning as a means to inferencing has proliferated thanks to its versatility and ability to approach or exceed human-level accuracy. These computational models have seemingly insatiable appetites for computational resources not only while training, but also when deployed at scales ranging from data centers all th…
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neura…
For a convex body on the Euclidean unit sphere the spherical convex floating body is introduced. The asymptotic behavior of the volume difference of a spherical convex body and its spherical floating body is investigated. This gives rise to a new spherical area measure, the floating area. Remarkably, this floating area…
Asymptotic results for weighted floating bodies are established and used to obtain new proofs for the existence of floating areas on the sphere and in hyperbolic space and to establish the existence of floating areas in Hilbert geometries. Results on weighted best and random approximation and the new approach to floati…
IntSGD compresses SGD gradients without floats, converging as SGD.
problem Efficiently compressing stochastic gradients in distributed SGD.
method Adaptive integer compression of gradients, estimating scaling adaptively.
result IntSGD matches SGD's iteration complexity for convex and non-convex functions.
G-Net constructs binary neural networks with high accuracy using randomized binary embeddings.
problem Creating high-accuracy binary neural networks with theoretical guarantees.
method Proposes a novel floating-point G-Net family with randomized binary embeddings and theoretical accuracy guarantees.
result Empirically, G-Net achieves almost 30% higher accuracy on CIFAR-10 compared to prior HDC models.
We carry out a systematic investigation on floating bodies in real space forms. A new unifying approach not only allows us to treat the important classical case of Euclidean space as well as the recent extension to the Euclidean unit sphere, but also the new extension of floating bodies to hyperbolic space. Our main re…
BFLOAT16 achieves SOTA results in deep learning training without hyper-parameter tuning.
problem Ensuring deep learning training achieves SOTA results without hyper-parameter tuning.
method Implemented a method to emulate BFLOAT16 operations in various frameworks.
result BFLOAT16 achieves SOTA results in deep learning training without hyper-parameter tuning.
Hamiltonian method applied to floating barrier options pricing.
problem Pricing of floating barrier options.
method Hamiltonian approach in quantum mechanics applied to barrier options.
result Analytical expressions for pricing kernel and option price derived.
The study analyzes convergence of adaptive optimizers under low-precision training.
problem Understanding why low-precision training remains effective for large models.
method Developed a theoretical framework for analyzing convergence of adaptive optimizers under floating-point quantization.
result Adaptive optimizers retain convergence rates close to full-precision methods under logarithmic mantissa scaling.
Floating Gossip improves continuous machine learning in a decentralized manner.
problem Continuous learning in infrastructure-less environments.
method Mean field approach to analyze the impact of communication and computing in Floating Gossip.
result Floating Gossip can effectively incorporate large amounts of data into machine learning models.
Deep neural networks (DNN) are powerful models for many pattern recognition tasks, yet their high computational complexity and memory requirement limit them to applications on high-performance computing platforms. In this paper, we propose a new method to evaluate DNNs trained with 32bit floating point (float32) accura…
With ever-increasing computational demand for deep learning, it is critical to investigate the implications of the numeric representation and precision of DNN model weights and activations on computational efficiency. In this work, we explore unconventional narrow-precision floating-point representations as it relates …
A novel method quantizes Batch Normalization for QNNs, maintaining accuracy and efficiency.
problem Quantization challenges in Batch Normalization for QNNs.
method Converts BN to fixed-point operation with shared scale, suitable for hardware.
result Maintains same outputs through rigorous analysis and experiments.
Introduces new weighted floating functions and affine surface areas.
problem Developing new mathematical concepts for convex bodies.
method Introducing weighted floating functions and weighted functional affine surface areas.
result New relations to traditional and classical affine surface areas.
We establish a connection between capillary floating in neutral equilibrium and the billiard ball problem. This allows us to reduce the question of floating in neutral equilibrium at any orientation with a prescribed contact angle for infinite homogeneous cylinders to a question about billiard caustics for their orthog…
Paper examines floating exercise boundaries for American options in time-inhomogeneous models.
problem Floating exercise boundaries in time-inhomogeneous models with negative interest rates or yields.
method Semi-analytical approach for pricing American options.
result Specialized pricing methodologies are required for models with floating exercise boundaries.
NeuZip compresses neural network weights to save memory during training and inference.
problem Memory constraints in neural network training and inference.
method Entropy-based dynamic weight compression.
result Significant reduction in memory usage without performance loss.
Deep neural networks struggle with numerical instability during training.
problem Numerical instability in gradient descent training of deep neural networks.
method Analysis of floating-point arithmetic and gradient descent in ReLU neural networks.
result It is highly unlikely for ReLU networks to maintain a superlinear number of affine pieces during training.
The floating body approach to affine surface area is adapted to a holomorphic context providing an alternate approach to Fefferman's invariant hypersurface measure.
Winograd convolution improves DNN accuracy in fp16 and bf16 formats.
problem Improving DNN accuracy in floating point formats.
method Investigated a wider range of Winograd algorithms for DNNs.
result Significant improvement in FP accuracy in fp16 and bf16 formats.
Study spherical convex bodies using Lp-floating areas and curvature entropy.
problem Analogous isoperimetric inequalities for spherical convex bodies.
method Introduced Lp-floating areas and curvature entropy for spherical convex bodies. result Established isoperimetric inequalities and dual isoperimetric inequalities.
K-TanH is a fast, efficient deep learning activation function.
problem Improving the speed and efficiency of deep learning models.
method K-TanH is a novel, highly accurate, low-precision integer approximation of TanH.
result K-TanH achieves up to 5x speedup over Intel SVML and is more efficient.
We investigate weighted floating bodies of polytopes. We show that the weighted volume depends on the complete flags of the polytope. This connection is obtained by introducing flag simplices, which translate between the metric and combinatorial structure. Our results are applied in spherical and hyperbolic space. This…
WrapNet optimizes inference for low-resolution neural networks by using 8-bit additions.
problem Reducing multiplication complexity in low-resolution neural networks.
method Adapting neural networks to use low-resolution (8-bit) additions in accumulators, with a cyclic activation layer and overflow penalty regularizer.
result Achieves comparable classification accuracy to 32-bit counterparts using low-resolution additions.