The use of low-precision fixed-point arithmetic along with stochastic rounding has been proposed as a promising alternative to the commonly used 32-bit floating point arithmetic to enhance training neural networks training in terms of performance and energy efficiency. In the first part of this paper, the behaviour of …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,042 papers · 148 categories
Trend · papers per month
48 results for “floating point arithmetic”
The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek t…
Deep neural networks (DNN) are powerful models for many pattern recognition tasks, yet their high computational complexity and memory requirement limit them to applications on high-performance computing platforms. In this paper, we propose a new method to evaluate DNNs trained with 32bit floating point (float32) accura…
Researchers find floating point errors can mislead neural network verifiers.
problem Floating point arithmetic inaccuracies mislead neural network verifiers.
method Efficiently searches inputs and constructs neural network architectures to exploit verification errors.
result Floating point errors can systematically mislead neural network verifiers.
Deep learning as a means to inferencing has proliferated thanks to its versatility and ability to approach or exceed human-level accuracy. These computational models have seemingly insatiable appetites for computational resources not only while training, but also when deployed at scales ranging from data centers all th…
Deep neural networks struggle with numerical instability during training.
problem Numerical instability in gradient descent training of deep neural networks.
method Analysis of floating-point arithmetic and gradient descent in ReLU neural networks.
result It is highly unlikely for ReLU networks to maintain a superlinear number of affine pieces during training.
K-TanH is a fast, efficient deep learning activation function.
problem Improving the speed and efficiency of deep learning models.
method K-TanH is a novel, highly accurate, low-precision integer approximation of TanH.
result K-TanH achieves up to 5x speedup over Intel SVML and is more efficient.
SIMD operations boost Bayesian computations up to 6x faster.
problem Expensive Bayesian computations are computationally intensive and parallelizable.
method Demonstrated the utility of SIMD operations for Bayesian applications using standard libraries.
result Up to 6x improvement in floating point arithmetic performance.
IntSGD compresses SGD gradients without floats, converging as SGD.
problem Efficiently compressing stochastic gradients in distributed SGD.
method Adaptive integer compression of gradients, estimating scaling adaptively.
result IntSGD matches SGD's iteration complexity for convex and non-convex functions.
The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, un…
The rising popularity of intelligent mobile devices and the daunting computational cost of deep learning-based models call for efficient and accurate on-device inference schemes. We propose a quantization scheme that allows inference to be carried out using integer-only arithmetic, which can be implemented more efficie…
Efficient Winograd convolution for INT8 networks using RNS.
problem Difficulty in applying Winograd algorithm to low-precision quantized networks.
method Extends Winograd algorithm to Residue Number System (RNS) for efficient INT8 convolution.
result Arithmetic complexity reduction up to 7.03x with performance improvement up to 2.30x-4.69x.
We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks…
Researchers show NN-based communication algorithms can be implemented on hardware without significant performance loss.
problem Reducing complexity and improving performance of NN-based communication algorithms for practical hardware implementation.
method Implementation of NN-based algorithms in fixed-point arithmetic with quantized weights on specialized hardware (FPGAs, ASICs).
result It is possible to implement NN-based algorithms in fixed-point arithmetic with quantized weights on hardware without significant performance loss.
Deploying Customized Data Representation and Approximate Computing in Machine Learning Applicationscs.LG
Major advancements in building general-purpose and customized hardware have been one of the key enablers of versatility and pervasiveness of machine learning models such as deep neural networks. To sustain this ubiquitous deployment of machine learning models and cope with their computational and storage complexity, se…
For successful deployment of deep neural networks on highly--resource-constrained devices (hearing aids, earbuds, wearables), we must simplify the types of operations and the memory/power resources used during inference. Completely avoiding inference-time floating-point operations is one of the simplest ways to design …
Asian option as a fixed-pointq-fin.PR
We characterize the price of an Asian option, a financial contract, as a fixed-point of a non-linear operator. In recent years, there has been interest in incorporating changes of regime into the parameters describing the evolution of the underlying asset price, namely the interest rate and the volatility, to model sud…
AdaptivFloat improves deep learning inference accuracy at low precision.
problem Low precision quantization issues in deep learning inference.
method Dynamic floating-point representation with adaptive clipping.
result Consistently higher inference accuracy at low precision compared to other methods.
Paper proposes training deep neural networks with 8-bit floating point precision.
problem Challenges in training deep neural networks at 8-bit precision due to higher precision and dynamic range requirements.
method Proposes a method to train deep neural networks using 8-bit floating point for weights, activations, errors, and gradients. Introduces an enhanced loss scaling method and stochastic rounding technique.
result Demonstrates state-of-the-art accuracy across multiple datasets and workloads compared to full precision baseline.
Constructs minimal immersions with singularities.
problem Minimal immersions with singularities in metric spaces.
method Constructs minimal immersions with catenoidal necks or floating disks converging to a singular point.
result Constructs minimal immersions with singularities.
Popular deep neural networks (DNNs) spend the majority of their execution time computing convolutions. The Winograd family of algorithms can greatly reduce the number of arithmetic operations required and is present in many DNN software frameworks. However, the performance gain is at the expense of a reduction in float…
Kernel methods are widespread in machine learning; however, they are limited by the quadratic complexity of the construction, application, and storage of kernel matrices. Low-rank matrix approximation algorithms are widely used to address this problem and reduce the arithmetic and storage cost. However, we observed tha…
Cheetah framework optimizes DNNs for edge devices using low-precision formats.
problem Reducing DNN model size for edge devices while maintaining accuracy.
method Mixed low-precision hardware and software co-design framework using posit and other formats.
result 16-bit posits outperform 16-bit floating point in training, and [5..8]-bit posits improve inference performance.
We consider the problem of estimating the arithmetic average of a finite collection of real vectors stored in a distributed fashion across several compute nodes subject to a communication budget constraint. Our analysis does not rely on any statistical assumptions about the source of the vectors. This problem arises as…
Posits improve DNN training efficiency on edge devices.
problem Training deep neural networks with low-precision formats.
method Used posits (5-8 bit) for DNN training compared to floating point.
result 16-bit posits outperform 16-bit floating point for end-to-end training.
Convolutional neural networks have recently achieved significant breakthroughs in various image classification tasks. However, they are computationally expensive,which can make their feasible mplementation on embedded and low-power devices difficult. In this paper convolutional neural network binarization is implemente…
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
Low-bit training framework reduces energy consumption in CNNs.
problem Reducing energy consumption in convolutional neural networks.
method Low-bit training framework using MLS tensor format with dynamic quantization.
result Achieves superior trade-off between accuracy and bit-width.
New method for pricing discrete Asian and Lookback options under Heston model.
problem Efficient pricing of discrete Asian and Lookback options under Heston model.
method Data-driven approach using artificial neural networks and stochastic collocation points.
result High accuracy and significant computational time reduction compared to classical methods.
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neura…
Study shows spectral gaps limit points on surfaces.
problem Understanding spectral gaps on arithmetic hyperbolic surfaces.
method Analyzes closed arithmetic hyperbolic surfaces to find limit points of spectral gaps.
result Limit points of spectral gaps are on the interval [0, 1/4].
Study finds limit points of bass notes on hyperbolic surfaces.
problem Understanding the limit points of bass notes on hyperbolic surfaces.
method Proved using mathematical analysis of arithmetic hyperbolic surfaces.
result Limit points are found to be within the interval [0, 1/4].
For a convex body on the Euclidean unit sphere the spherical convex floating body is introduced. The asymptotic behavior of the volume difference of a spherical convex body and its spherical floating body is investigated. This gives rise to a new spherical area measure, the floating area. Remarkably, this floating area…
Asymptotic results for weighted floating bodies are established and used to obtain new proofs for the existence of floating areas on the sphere and in hyperbolic space and to establish the existence of floating areas in Hilbert geometries. Results on weighted best and random approximation and the new approach to floati…
G-Net constructs binary neural networks with high accuracy using randomized binary embeddings.
problem Creating high-accuracy binary neural networks with theoretical guarantees.
method Proposes a novel floating-point G-Net family with randomized binary embeddings and theoretical accuracy guarantees.
result Empirically, G-Net achieves almost 30% higher accuracy on CIFAR-10 compared to prior HDC models.
We carry out a systematic investigation on floating bodies in real space forms. A new unifying approach not only allows us to treat the important classical case of Euclidean space as well as the recent extension to the Euclidean unit sphere, but also the new extension of floating bodies to hyperbolic space. Our main re…
BFLOAT16 achieves SOTA results in deep learning training without hyper-parameter tuning.
problem Ensuring deep learning training achieves SOTA results without hyper-parameter tuning.
method Implemented a method to emulate BFLOAT16 operations in various frameworks.
result BFLOAT16 achieves SOTA results in deep learning training without hyper-parameter tuning.
Study transcendence of abelian differential periods from bi-algebraic perspective.
problem Arithmetic and functional transcendence of periods of abelian differentials.
method Bi-algebraic structure on strata of abelian differentials.
result Characterization of arithmetic points and proof of linear bi-algebraic curves.
Hamiltonian method applied to floating barrier options pricing.
problem Pricing of floating barrier options.
method Hamiltonian approach in quantum mechanics applied to barrier options.
result Analytical expressions for pricing kernel and option price derived.
The study analyzes convergence of adaptive optimizers under low-precision training.
problem Understanding why low-precision training remains effective for large models.
method Developed a theoretical framework for analyzing convergence of adaptive optimizers under floating-point quantization.
result Adaptive optimizers retain convergence rates close to full-precision methods under logarithmic mantissa scaling.
This paper explains how low-precision arithmetic causes loss spikes in deep learning models.
problem Loss spikes during long-term training of deep neural networks.
method Analyzes the impact of floating-point precision limits on gradient updates and feature means.
result Numerical Feature Inflation (NFI) explains loss spikes and rapid parameter norm growth.
Floating Gossip improves continuous machine learning in a decentralized manner.
problem Continuous learning in infrastructure-less environments.
method Mean field approach to analyze the impact of communication and computing in Floating Gossip.
result Floating Gossip can effectively incorporate large amounts of data into machine learning models.
With ever-increasing computational demand for deep learning, it is critical to investigate the implications of the numeric representation and precision of DNN model weights and activations on computational efficiency. In this work, we explore unconventional narrow-precision floating-point representations as it relates …
The essay discusses Margulis' theorems and their implications.
problem Understanding the implications of Margulis' theorems.
method Historical review and discussion of open questions.
result Discussion of open questions stemming from Margulis' work.
A novel method quantizes Batch Normalization for QNNs, maintaining accuracy and efficiency.
problem Quantization challenges in Batch Normalization for QNNs.
method Converts BN to fixed-point operation with shared scale, suitable for hardware.
result Maintains same outputs through rigorous analysis and experiments.
Study counts and equidistributes rational points in quaternionic Heisenberg groups.
problem Counting and equidistribution of rational points in quaternionic Heisenberg groups.
method Arithmetic group actions on quaternionic hyperbolic spaces, Mertens counting formula, Neville equidistribution theorem.
result Proved Mertens counting formula and Neville equidistribution theorem for rational points over definite quaternion algebras.
Introduces new weighted floating functions and affine surface areas.
problem Developing new mathematical concepts for convex bodies.
method Introducing weighted floating functions and weighted functional affine surface areas.
result New relations to traditional and classical affine surface areas.
We establish a connection between capillary floating in neutral equilibrium and the billiard ball problem. This allows us to reduce the question of floating in neutral equilibrium at any orientation with a prescribed contact angle for infinite homogeneous cylinders to a question about billiard caustics for their orthog…