A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Researchers find floating point errors can mislead neural network verifiers.
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
Conventional hardware-friendly quantization methods, such as fixed-point or integer, tend to perform poorly at very low word sizes as their shrinking dynamic ranges cannot adequately capture the wide data distributions commonly seen in sequence transduction models. We present AdaptivFloat, a floating-point inspired num…
The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek t…
Reduced precision computation for deep neural networks is one of the key areas addressing the widening compute gap driven by an exponential growth in model size. In recent years, deep learning training has largely migrated to 16-bit precision, with significant gains in performance and energy efficiency. However, attemp…
The study analyzes convergence of adaptive optimizers under low-precision training.
For successful deployment of deep neural networks on highly--resource-constrained devices (hearing aids, earbuds, wearables), we must simplify the types of operations and the memory/power resources used during inference. Completely avoiding inference-time floating-point operations is one of the simplest ways to design …
The use of low-precision fixed-point arithmetic along with stochastic rounding has been proposed as a promising alternative to the commonly used 32-bit floating point arithmetic to enhance training neural networks training in terms of performance and energy efficiency. In the first part of this paper, the behaviour of …
Recently, the posit numerical format has shown promise for DNN data representation and compute with ultra-low precision ([5..8]-bit). However, majority of studies focus only on DNN inference. In this work, we propose DNN training using posits and compare with the floating point training. We evaluate on both MNIST and F…
Constructs minimal immersions with singularities.
Popular deep neural networks (DNNs) spend the majority of their execution time computing convolutions. The Winograd family of algorithms can greatly reduce the number of arithmetic operations required and is present in many DNN software frameworks. However, the performance gain is at the expense of a reduction in float…
Low-precision DNNs have been extensively explored in order to reduce the size of DNN models for edge devices. Recently, the posit numerical format has shown promise for DNN data representation and compute with ultra-low precision in [5..8]-bits. However, previous studies were limited to studying posit for DNN inference…
Gradient descent stagnates in low-precision, but unbiased rounding schemes improve convergence.
Convolutional neural networks have recently achieved significant breakthroughs in various image classification tasks. However, they are computationally expensive,which can make their feasible mplementation on embedded and low-power devices difficult. In this paper convolutional neural network binarization is implemente…
Deep learning as a means to inferencing has proliferated thanks to its versatility and ability to approach or exceed human-level accuracy. These computational models have seemingly insatiable appetites for computational resources not only while training, but also when deployed at scales ranging from data centers all th…
Energy-efficient sampling for machine learning using magnetic tunnel junctions.
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neura…
For a convex body on the Euclidean unit sphere the spherical convex floating body is introduced. The asymptotic behavior of the volume difference of a spherical convex body and its spherical floating body is investigated. This gives rise to a new spherical area measure, the floating area. Remarkably, this floating area…
The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, un…
This paper presents the first comprehensive empirical study demonstrating the efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep Learning training across image classification, speech recognition, language modeling, generative networks and industrial recommendation systems. BFLOAT16 is attrac…
Binary neural networks (BNN) have been studied extensively since they run dramatically faster at lower memory and power consumption than floating-point networks, thanks to the efficiency of bit operations. However, contemporary BNNs whose weights and activations are both single bits suffer from severe accuracy degradat…
Asymptotic results for weighted floating bodies are established and used to obtain new proofs for the existence of floating areas on the sphere and in hyperbolic space and to establish the existence of floating areas in Hilbert geometries. Results on weighted best and random approximation and the new approach to floati…
We consider the problem of estimating the arithmetic average of a finite collection of real vectors stored in a distributed fashion across several compute nodes subject to a communication budget constraint. Our analysis does not rely on any statistical assumptions about the source of the vectors. This problem arises as…
IntSGD compresses SGD gradients without floats, converging as SGD.
We carry out a systematic investigation on floating bodies in real space forms. A new unifying approach not only allows us to treat the important classical case of Euclidean space as well as the recent extension to the Euclidean unit sphere, but also the new extension of floating bodies to hyperbolic space. Our main re…
G-Net constructs binary neural networks with high accuracy using randomized binary embeddings.
Reduced numerical precision is a common technique to reduce computational cost in many Deep Neural Networks (DNNs). While it has been observed that DNNs are resilient to small errors and noise, no general result exists that is capable of predicting a given DNN system architecture's sensitivity to reduced precision. In …
Hamiltonian method applied to floating barrier options pricing.
Floating Gossip improves continuous machine learning in a decentralized manner.
Deep neural networks (DNN) are powerful models for many pattern recognition tasks, yet their high computational complexity and memory requirement limit them to applications on high-performance computing platforms. In this paper, we propose a new method to evaluate DNNs trained with 32bit floating point (float32) accura…
With ever-increasing computational demand for deep learning, it is critical to investigate the implications of the numeric representation and precision of DNN model weights and activations on computational efficiency. In this work, we explore unconventional narrow-precision floating-point representations as it relates …
We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks…
A novel method quantizes Batch Normalization for QNNs, maintaining accuracy and efficiency.
Introduces new weighted floating functions and affine surface areas.
We establish a connection between capillary floating in neutral equilibrium and the billiard ball problem. This allows us to reduce the question of floating in neutral equilibrium at any orientation with a prescribed contact angle for infinite homogeneous cylinders to a question about billiard caustics for their orthog…
Paper examines floating exercise boundaries for American options in time-inhomogeneous models.
We propose K-TanH, a novel, highly accurate, hardware efficient approximation of popular activation function TanH for Deep Learning. K-TanH consists of parameterized low-precision integer operations, such as, shift and add/subtract (no floating point operation needed) where parameters are stored in very small look-up t…
NeuZip compresses neural network weights to save memory during training and inference.
In this article we study the convergence of a European lookback option with floating strike evaluated with the binomial model of Cox-Ross-Rubinstein to its evaluation with the Black-Scholes model. We do the same for its delta. We confirm that these convergences are of order 1/Sqrt(n). For this, we use the binomial mode…
The floating body approach to affine surface area is adapted to a holomorphic context providing an alternate approach to Fefferman's invariant hypersurface measure.
Deep neural networks struggle with numerical instability during training.
Study spherical convex bodies using -floating areas and curvature entropy.
We investigate weighted floating bodies of polytopes. We show that the weighted volume depends on the complete flags of the polytope. This connection is obtained by introducing flag simplices, which translate between the metric and combinatorial structure. Our results are applied in spherical and hyperbolic space. This…
Floating geodesic planes in Hitchin manifolds have fractal closures with non-integer dimensions.
Unique floating and buoyancy surfaces identify convex polytopes.
Study on thermodynamic costs of simple linear regression.
AI progress measured by reduced compute needed to reach past performance.
We compare the results of our earlier paper on the floating in neutral equilibrium at arbitrary orientation in the sense of Finn-Young with the literature on its counterpart in the sense of Archimedes. We add a few remarks of personal and social-historical character.