Binarized CNNs improve GPU inference efficiency on resource-constrained devices.
problem Efficient inference on resource-constrained devices for image classification.
method Binarization of weights and computations in CNNs, implemented on GPUs.
result 7.4X speedup with 4.4% accuracy loss on embedded GPU platforms.
Hybrid BFP-FP improves DNN training accuracy with 8.5x higher throughput.
problem Limited dynamic range of fixed-point arithmetic for DNN training convergence.
method Introducing HBFP, a hybrid BFP-FP approach.
result HBFP matches floating point's accuracy while delivering up to 8.5x higher throughput.
No floating point, no multiplications, no problem! Training efficient networks for resource-constrained devices.
problem Designing efficient neural networks for resource-constrained devices without floating-point operations.
method Discretizing both in-network non-linearities and network weights to avoid floating-point and multiplication operations.
result Training networks without floating-point operations can achieve comparable performance to those using floating-point operations, with less memory usage.
Researchers find floating point errors can mislead neural network verifiers.
problem Floating point arithmetic inaccuracies mislead neural network verifiers.
method Efficiently searches inputs and constructs neural network architectures to exploit verification errors.
result Floating point errors can systematically mislead neural network verifiers.
Paper analyzes Winograd convolution errors and proposes methods to reduce them.
problem Reduction of floating point error in Winograd convolution for deep neural networks.
method Analysis of worst case FP error, estimation of norm and conditioning, proposed evaluation orderings, sampling points selection, mixed-precision convolution, pairwise summation.
result Proposed methods significantly reduce FP error for a given block size, allowing larger block sizes and reduced computation.
Channel gating reduces CNN computation cost by skipping ineffective feature regions.
problem Reducing computation cost in CNNs while maintaining accuracy.
method Dynamic, fine-grained pruning scheme that identifies and skips computation on ineffective feature regions.
result 2.7-8.0x reduction in FLOPs and 2.0-4.4x reduction in memory accesses with minimal accuracy loss.
Paper explores low-precision arithmetic for neural networks, improving efficiency.
problem Training neural networks with high precision and energy efficiency.
method 12-bit fixed-point, 12-bit floating-point, local scaling, Power-of-Two arithmetic.
result 7-bit Power-of-Two arithmetic achieves minimal loss in accuracy with reduced computation.
AdaptivFloat improves deep learning inference accuracy at low precision.
problem Low precision quantization issues in deep learning inference.
method Dynamic floating-point representation with adaptive clipping.
result Consistently higher inference accuracy at low precision compared to other methods.
AI progress measured by reduced compute needed to reach past performance.
problem Quantifying algorithmic progress in AI.
method Analysis of floating-point operations required to train neural networks.
result Algorithmic efficiency doubled every 16 months over 7 years.
Training deep neural networks with 8-bit floating point numbers is now possible and more efficient.
problem Challenges in training DNNs with reduced precision, especially for gradient computations.
method Introduction of chunk-based accumulation and floating point stochastic rounding to reduce arithmetic precision to 16 bits.
result Successful training of DNNs using 8-bit floating point numbers, maintaining accuracy on various models and datasets.
Paper proposes training deep neural networks with 8-bit floating point precision.
problem Challenges in training deep neural networks at 8-bit precision due to higher precision and dynamic range requirements.
method Proposes a method to train deep neural networks using 8-bit floating point for weights, activations, errors, and gradients. Introduces an enhanced loss scaling method and stochastic rounding technique.
result Demonstrates state-of-the-art accuracy across multiple datasets and workloads compared to full precision baseline.
Speed up neural networks by 2x with 5% mAP loss.
problem High computational cost of neural network forward passes.
method Deep Learning Approximation: lossless and lossy optimizations.
result 2x speedup in network forward pass with 5% mAP drop.
SWAT reduces training efficiency and speed for CNNs.
problem Efficient training of neural networks on sparse hardware.
method SWAT modifies backpropagation to eliminate small weights and activations.
result SWAT reduces FLOPS by 80% on ResNet-50 for ImageNet, improving training speed by 3.3x.
Constructs minimal immersions with singularities.
problem Minimal immersions with singularities in metric spaces.
method Constructs minimal immersions with catenoidal necks or floating disks converging to a singular point.
result Constructs minimal immersions with singularities.
Structural-Jump-LSTM speeds up reading by skipping and jumping text.
problem Sequential inference in RNNs makes reading time linearly dependent on input length.
method Introduces a novel LSTM model with agents for skipping and jumping text.
result Structural-Jump-LSTM achieves best FLOP reduction and maintains or improves accuracy.
A method to reduce knowledge graph embedding models by binarizing parameters.
problem Large memory requirements for tensor factorization models in knowledge graph completion.
method Introducing a quantization function to binarize parameters of CP tensor decomposition.
result Successfully reduced model size by more than an order of magnitude while maintaining task performance.
Review and compare model order reduction methods for process engineering.
problem Creating computationally efficient yet accurate models for real-time applications.
method Nonlinear model order reduction methods, including general-purpose and tailored approaches for chemical processes.
result Comparison of eight model order reduction methods applied to an air separation process model.
We present a theoretical analysis and empirical evaluations of a novel set of techniques for computational cost reduction of classifiers that are based on learned transform and soft-threshold. By modifying optimization procedures for dictionary and classifier training, as well as the resulting dictionary entries, our t…
Cheetah framework optimizes DNNs for edge devices using low-precision formats.
problem Reducing DNN model size for edge devices while maintaining accuracy.
method Mixed low-precision hardware and software co-design framework using posit and other formats.
result 16-bit posits outperform 16-bit floating point in training, and [5..8]-bit posits improve inference performance.
Custom narrow-precision representations boost DNN inference speed by 7.6x with minimal accuracy loss.
problem Improving computational efficiency of deep neural networks.
method Exploring and utilizing unconventional narrow-precision floating-point representations for DNN weights and activations.
result Average speedup of 7.6x with less than 1% accuracy loss.
Posits improve DNN training efficiency on edge devices.
problem Training deep neural networks with low-precision formats.
method Used posits (5-8 bit) for DNN training compared to floating point.
result 16-bit posits outperform 16-bit floating point for end-to-end training.
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
Quantization scheme improves inference efficiency for deep learning models.
problem Limited computational resources and energy constraints in deploying deep learning models.
method Proposes a quantization scheme to reduce precision and improve efficiency without sacrificing accuracy.
result End-to-end post-quantization accuracies comparable to reference model achieved using a single inference batch calibration.
For a convex body on the Euclidean unit sphere the spherical convex floating body is introduced. The asymptotic behavior of the volume difference of a spherical convex body and its spherical floating body is investigated. This gives rise to a new spherical area measure, the floating area. Remarkably, this floating area…
Flexpoint improves deep learning training efficiency by using adaptive 16-bit format.
problem Training deep neural networks in low bit-width formats is challenging.
method Flexpoint uses a shared exponent dynamically adjusted to minimize overflows and maximize dynamic range.
result 16-bit Flexpoint tensors closely match 32-bit floating point in training deep networks without tuning.
Asymptotic results for weighted floating bodies are established and used to obtain new proofs for the existence of floating areas on the sphere and in hyperbolic space and to establish the existence of floating areas in Hilbert geometries. Results on weighted best and random approximation and the new approach to floati…
IntSGD compresses SGD gradients without floats, converging as SGD.
problem Efficiently compressing stochastic gradients in distributed SGD.
method Adaptive integer compression of gradients, estimating scaling adaptively.
result IntSGD matches SGD's iteration complexity for convex and non-convex functions.
G-Net constructs binary neural networks with high accuracy using randomized binary embeddings.
problem Creating high-accuracy binary neural networks with theoretical guarantees.
method Proposes a novel floating-point G-Net family with randomized binary embeddings and theoretical accuracy guarantees.
result Empirically, G-Net achieves almost 30% higher accuracy on CIFAR-10 compared to prior HDC models.
We carry out a systematic investigation on floating bodies in real space forms. A new unifying approach not only allows us to treat the important classical case of Euclidean space as well as the recent extension to the Euclidean unit sphere, but also the new extension of floating bodies to hyperbolic space. Our main re…
BFLOAT16 achieves SOTA results in deep learning training without hyper-parameter tuning.
problem Ensuring deep learning training achieves SOTA results without hyper-parameter tuning.
method Implemented a method to emulate BFLOAT16 operations in various frameworks.
result BFLOAT16 achieves SOTA results in deep learning training without hyper-parameter tuning.
Researchers show NN-based communication algorithms can be implemented on hardware without significant performance loss.
problem Reducing complexity and improving performance of NN-based communication algorithms for practical hardware implementation.
method Implementation of NN-based algorithms in fixed-point arithmetic with quantized weights on specialized hardware (FPGAs, ASICs).
result It is possible to implement NN-based algorithms in fixed-point arithmetic with quantized weights on hardware without significant performance loss.
The study analyzes convergence of adaptive optimizers under low-precision training.
problem Understanding why low-precision training remains effective for large models.
method Developed a theoretical framework for analyzing convergence of adaptive optimizers under floating-point quantization.
result Adaptive optimizers retain convergence rates close to full-precision methods under logarithmic mantissa scaling.
Hamiltonian method applied to floating barrier options pricing.
problem Pricing of floating barrier options.
method Hamiltonian approach in quantum mechanics applied to barrier options.
result Analytical expressions for pricing kernel and option price derived.
Study floating bodies of polytopes, linking volume to flags.
problem Understanding floating bodies of polytopes in various spaces.
method Introducing flag simplices to connect metric and combinatorial structures.
result Weighted volume depends on complete flags of polytopes.
Floating Gossip improves continuous machine learning in a decentralized manner.
problem Continuous learning in infrastructure-less environments.
method Mean field approach to analyze the impact of communication and computing in Floating Gossip.
result Floating Gossip can effectively incorporate large amounts of data into machine learning models.
A novel method quantizes Batch Normalization for QNNs, maintaining accuracy and efficiency.
problem Quantization challenges in Batch Normalization for QNNs.
method Converts BN to fixed-point operation with shared scale, suitable for hardware.
result Maintains same outputs through rigorous analysis and experiments.
Introduces new weighted floating functions and affine surface areas.
problem Developing new mathematical concepts for convex bodies.
method Introducing weighted floating functions and weighted functional affine surface areas.
result New relations to traditional and classical affine surface areas.
We establish a connection between capillary floating in neutral equilibrium and the billiard ball problem. This allows us to reduce the question of floating in neutral equilibrium at any orientation with a prescribed contact angle for infinite homogeneous cylinders to a question about billiard caustics for their orthog…
Quantizes CNNs for efficient inference with 8-bit weights and activations.
problem Reducing model size and inference time for deep learning models.
method Quantization of integer weights and activations, post-training and quantization-aware training.
result Classification accuracy within 2% of floating point networks, model size reduction by 4x.
Paper examines floating exercise boundaries for American options in time-inhomogeneous models.
problem Floating exercise boundaries in time-inhomogeneous models with negative interest rates or yields.
method Semi-analytical approach for pricing American options.
result Specialized pricing methodologies are required for models with floating exercise boundaries.
Efficient Winograd convolution for INT8 networks using RNS.
problem Difficulty in applying Winograd algorithm to low-precision quantized networks.
method Extends Winograd algorithm to Residue Number System (RNS) for efficient INT8 convolution.
result Arithmetic complexity reduction up to 7.03x with performance improvement up to 2.30x-4.69x.
NeuZip compresses neural network weights to save memory during training and inference.
problem Memory constraints in neural network training and inference.
method Entropy-based dynamic weight compression.
result Significant reduction in memory usage without performance loss.
Paper proposes efficient DNN inference using low precision arithmetic.
problem High computational complexity and memory requirement of DNNs.
method Combining 2bit or 3bit integer arithmetics with binary shift and clipping for float32 accuracy.
result Efficient DNN inference on low precision hardware with minimal performance loss.
Deep neural networks struggle with numerical instability during training.
problem Numerical instability in gradient descent training of deep neural networks.
method Analysis of floating-point arithmetic and gradient descent in ReLU neural networks.
result It is highly unlikely for ReLU networks to maintain a superlinear number of affine pieces during training.
Winograd convolution improves DNN accuracy in fp16 and bf16 formats.
problem Improving DNN accuracy in floating point formats.
method Investigated a wider range of Winograd algorithms for DNNs.
result Significant improvement in FP accuracy in fp16 and bf16 formats.
The floating body approach to affine surface area is adapted to a holomorphic context providing an alternate approach to Fefferman's invariant hypersurface measure.
K-TanH is a fast, efficient deep learning activation function.
problem Improving the speed and efficiency of deep learning models.
method K-TanH is a novel, highly accurate, low-precision integer approximation of TanH.
result K-TanH achieves up to 5x speedup over Intel SVML and is more efficient.
This work adds FLOPs to neural network training objectives.
problem Training resource-efficient neural networks with varying FLOPs requirements.
method Extends a state-of-the-art technique to incorporate FLOPs as an optimization objective.
result Different neural networks can be successfully trained for image classification given a desired FLOPs requirement.