The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek t…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,982 papers · 148 categories
Trend · papers per month
48 results for “Floating point”
Researchers find floating point errors can mislead neural network verifiers.
problem Floating point arithmetic inaccuracies mislead neural network verifiers.
method Efficiently searches inputs and constructs neural network architectures to exploit verification errors.
result Floating point errors can systematically mislead neural network verifiers.
For successful deployment of deep neural networks on highly--resource-constrained devices (hearing aids, earbuds, wearables), we must simplify the types of operations and the memory/power resources used during inference. Completely avoiding inference-time floating-point operations is one of the simplest ways to design …
AdaptivFloat improves deep learning inference accuracy at low precision.
problem Low precision quantization issues in deep learning inference.
method Dynamic floating-point representation with adaptive clipping.
result Consistently higher inference accuracy at low precision compared to other methods.
Paper proposes training deep neural networks with 8-bit floating point precision.
problem Challenges in training deep neural networks at 8-bit precision due to higher precision and dynamic range requirements.
method Proposes a method to train deep neural networks using 8-bit floating point for weights, activations, errors, and gradients. Introduces an enhanced loss scaling method and stochastic rounding technique.
result Demonstrates state-of-the-art accuracy across multiple datasets and workloads compared to full precision baseline.
The use of low-precision fixed-point arithmetic along with stochastic rounding has been proposed as a promising alternative to the commonly used 32-bit floating point arithmetic to enhance training neural networks training in terms of performance and energy efficiency. In the first part of this paper, the behaviour of …
Constructs minimal immersions with singularities.
problem Minimal immersions with singularities in metric spaces.
method Constructs minimal immersions with catenoidal necks or floating disks converging to a singular point.
result Constructs minimal immersions with singularities.
Cheetah framework optimizes DNNs for edge devices using low-precision formats.
problem Reducing DNN model size for edge devices while maintaining accuracy.
method Mixed low-precision hardware and software co-design framework using posit and other formats.
result 16-bit posits outperform 16-bit floating point in training, and [5..8]-bit posits improve inference performance.
Posits improve DNN training efficiency on edge devices.
problem Training deep neural networks with low-precision formats.
method Used posits (5-8 bit) for DNN training compared to floating point.
result 16-bit posits outperform 16-bit floating point for end-to-end training.
Convolutional neural networks have recently achieved significant breakthroughs in various image classification tasks. However, they are computationally expensive,which can make their feasible mplementation on embedded and low-power devices difficult. In this paper convolutional neural network binarization is implemente…
Deep learning as a means to inferencing has proliferated thanks to its versatility and ability to approach or exceed human-level accuracy. These computational models have seemingly insatiable appetites for computational resources not only while training, but also when deployed at scales ranging from data centers all th…
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neura…
For a convex body on the Euclidean unit sphere the spherical convex floating body is introduced. The asymptotic behavior of the volume difference of a spherical convex body and its spherical floating body is investigated. This gives rise to a new spherical area measure, the floating area. Remarkably, this floating area…
The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, un…
Asymptotic results for weighted floating bodies are established and used to obtain new proofs for the existence of floating areas on the sphere and in hyperbolic space and to establish the existence of floating areas in Hilbert geometries. Results on weighted best and random approximation and the new approach to floati…
IntSGD compresses SGD gradients without floats, converging as SGD.
problem Efficiently compressing stochastic gradients in distributed SGD.
method Adaptive integer compression of gradients, estimating scaling adaptively.
result IntSGD matches SGD's iteration complexity for convex and non-convex functions.
G-Net constructs binary neural networks with high accuracy using randomized binary embeddings.
problem Creating high-accuracy binary neural networks with theoretical guarantees.
method Proposes a novel floating-point G-Net family with randomized binary embeddings and theoretical accuracy guarantees.
result Empirically, G-Net achieves almost 30% higher accuracy on CIFAR-10 compared to prior HDC models.
We carry out a systematic investigation on floating bodies in real space forms. A new unifying approach not only allows us to treat the important classical case of Euclidean space as well as the recent extension to the Euclidean unit sphere, but also the new extension of floating bodies to hyperbolic space. Our main re…
BFLOAT16 achieves SOTA results in deep learning training without hyper-parameter tuning.
problem Ensuring deep learning training achieves SOTA results without hyper-parameter tuning.
method Implemented a method to emulate BFLOAT16 operations in various frameworks.
result BFLOAT16 achieves SOTA results in deep learning training without hyper-parameter tuning.
Hamiltonian method applied to floating barrier options pricing.
problem Pricing of floating barrier options.
method Hamiltonian approach in quantum mechanics applied to barrier options.
result Analytical expressions for pricing kernel and option price derived.
The study analyzes convergence of adaptive optimizers under low-precision training.
problem Understanding why low-precision training remains effective for large models.
method Developed a theoretical framework for analyzing convergence of adaptive optimizers under floating-point quantization.
result Adaptive optimizers retain convergence rates close to full-precision methods under logarithmic mantissa scaling.
Floating Gossip improves continuous machine learning in a decentralized manner.
problem Continuous learning in infrastructure-less environments.
method Mean field approach to analyze the impact of communication and computing in Floating Gossip.
result Floating Gossip can effectively incorporate large amounts of data into machine learning models.
Deep neural networks (DNN) are powerful models for many pattern recognition tasks, yet their high computational complexity and memory requirement limit them to applications on high-performance computing platforms. In this paper, we propose a new method to evaluate DNNs trained with 32bit floating point (float32) accura…
With ever-increasing computational demand for deep learning, it is critical to investigate the implications of the numeric representation and precision of DNN model weights and activations on computational efficiency. In this work, we explore unconventional narrow-precision floating-point representations as it relates …
We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks…
A novel method quantizes Batch Normalization for QNNs, maintaining accuracy and efficiency.
problem Quantization challenges in Batch Normalization for QNNs.
method Converts BN to fixed-point operation with shared scale, suitable for hardware.
result Maintains same outputs through rigorous analysis and experiments.
Introduces new weighted floating functions and affine surface areas.
problem Developing new mathematical concepts for convex bodies.
method Introducing weighted floating functions and weighted functional affine surface areas.
result New relations to traditional and classical affine surface areas.
We establish a connection between capillary floating in neutral equilibrium and the billiard ball problem. This allows us to reduce the question of floating in neutral equilibrium at any orientation with a prescribed contact angle for infinite homogeneous cylinders to a question about billiard caustics for their orthog…
Paper examines floating exercise boundaries for American options in time-inhomogeneous models.
problem Floating exercise boundaries in time-inhomogeneous models with negative interest rates or yields.
method Semi-analytical approach for pricing American options.
result Specialized pricing methodologies are required for models with floating exercise boundaries.
NeuZip compresses neural network weights to save memory during training and inference.
problem Memory constraints in neural network training and inference.
method Entropy-based dynamic weight compression.
result Significant reduction in memory usage without performance loss.
Deep neural networks struggle with numerical instability during training.
problem Numerical instability in gradient descent training of deep neural networks.
method Analysis of floating-point arithmetic and gradient descent in ReLU neural networks.
result It is highly unlikely for ReLU networks to maintain a superlinear number of affine pieces during training.
The floating body approach to affine surface area is adapted to a holomorphic context providing an alternate approach to Fefferman's invariant hypersurface measure.
Winograd convolution improves DNN accuracy in fp16 and bf16 formats.
problem Improving DNN accuracy in floating point formats.
method Investigated a wider range of Winograd algorithms for DNNs.
result Significant improvement in FP accuracy in fp16 and bf16 formats.
Study spherical convex bodies using -floating areas and curvature entropy.
problem Analogous isoperimetric inequalities for spherical convex bodies.
method Introduced -floating areas and curvature entropy for spherical convex bodies.
result Established isoperimetric inequalities and dual isoperimetric inequalities.
K-TanH is a fast, efficient deep learning activation function.
problem Improving the speed and efficiency of deep learning models.
method K-TanH is a novel, highly accurate, low-precision integer approximation of TanH.
result K-TanH achieves up to 5x speedup over Intel SVML and is more efficient.
We investigate weighted floating bodies of polytopes. We show that the weighted volume depends on the complete flags of the polytope. This connection is obtained by introducing flag simplices, which translate between the metric and combinatorial structure. Our results are applied in spherical and hyperbolic space. This…
Floating geodesic planes in Hitchin manifolds have fractal closures with non-integer dimensions.
problem Rigidity of geodesic planes in Hitchin manifolds.
method Constructing a specific surface group and analyzing its action on the Hitchin manifold.
result Existence of floating geodesic planes in Hitchin manifolds with fractal closures.
Unique floating and buoyancy surfaces identify convex polytopes.
problem Identifying convex polytopes from their flotation and buoyancy surfaces.
method Proving uniqueness of surfaces for polytopes with uniform or prescribed density.
result Floating and buoyancy surfaces uniquely determine convex polytopes.
AI progress measured by reduced compute needed to reach past performance.
problem Quantifying algorithmic progress in AI.
method Analysis of floating-point operations required to train neural networks.
result Algorithmic efficiency doubled every 16 months over 7 years.
We compare the results of our earlier paper on the floating in neutral equilibrium at arbitrary orientation in the sense of Finn-Young with the literature on its counterpart in the sense of Archimedes. We add a few remarks of personal and social-historical character.
Tensor factorization has become an increasingly popular approach to knowledge graph completion(KGC), which is the task of automatically predicting missing facts in a knowledge graph. However, even with a simple model like CANDECOMP/PARAFAC(CP) tensor decomposition, KGC on existing knowledge graphs is impractical in res…
Gradient descent stagnates in low-precision, but unbiased rounding schemes improve convergence.
problem Stagnation of gradient descent in low-precision computation.
method Proposed unbiased stochastic rounding schemes that trade zero bias for larger probability of preserving small gradients.
result Unbiased rounding methods typically improve convergence rate of gradient descent for convex problems.
Energy-efficient sampling for machine learning using magnetic tunnel junctions.
problem Costly and inefficient random sampling in machine learning.
method Energy-efficient algorithm using stochastic magnetic tunnel junctions for uniform Float16 sampling.
result Higher energy efficiency than state-of-the-art algorithms, with a minimum factor of 9721.
SWAT reduces training efficiency and speed for CNNs.
problem Efficient training of neural networks on sparse hardware.
method SWAT modifies backpropagation to eliminate small weights and activations.
result SWAT reduces FLOPS by 80% on ResNet-50 for ImageNet, improving training speed by 3.3x.
Method trains sparse neural networks without sacrificing accuracy.
problem Training sparse neural networks limits model size.
method Updates sparse network topology during training.
result Requires fewer FLOPs to achieve accuracy.
This work proposes a method to learn sparse representations that are more efficient for large-scale data retrieval.
problem Efficient retrieval of high-dimensional representations from large databases is computationally challenging.
method The approach minimizes the number of floating-point operations (FLOPs) by learning sparse embeddings with uniform non-zero entries.
result The proposed method achieves a similar or better speed-vs-accuracy tradeoff compared to existing baselines.
Hamel's Equations and Geometric Mechanics of Constrained and Floating Multibody and Space Systemsmath.DS
Geometric mechanics approach to constrained and floating multibody systems using Hamel's equations.
problem Analytical mechanics of constrained and floating multibody systems.
method Geometric approach using bundle structures and connections, with Hamel's equations as a universal non-holonomic formulation.
result Achieved intrinsic splitting and inertial decoupling of reduced Euler-Lagrange equations.