arXiv research

Low precision weights, activations, and gradients have been proposed as a way to improve the computational efficiency and memory footprint of deep neural networks. Recently, low precision networks have even shown to be more robust to adversarial attacks. However, typical implementations of low precision DNNs use unifor…

Mixed-precision CA-SGD for generalized linear models on GPUs

Paper explores low-precision SGLD for neural networks, reducing costs without sacrificing performance.

Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precisi…

This paper improves low-precision sampling using SGHMC for deep learning models.

Low-precision computation is often used to lower the time and energy cost of machine learning, and recently hardware accelerators have been developed to support it. Still, it has been used primarily for inference - not training. Previous low-precision training algorithms suffered from a fundamental tradeoff: as the num…

Efficient Bitwidth Search optimizes neural network quantization for better performance.

Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using hal…

The study examines how class imbalance affects precision-recall curves.

It is well-known that the precision of data, hyperparameters, and internal representations employed in learning systems directly impacts its energy, throughput, and latency. The precision requirements for the training algorithm are also important for systems that learn on-the-fly. Prior work has shown that the data and…

This paper introduces a new method to train normalizing flows using precision-recall divergences.

Efforts to reduce the numerical precision of computations in deep learning training have yielded systems that aggressively quantize weights and activations, yet employ wide high-precision accumulators for partial sums in inner-product operations to preserve the quality of convergence. The absence of any framework to an…

Paper explores reducing precision in SVM for faster text classification.

We consider the post-training quantization problem, which discretizes the weights of pre-trained deep neural networks without re-training the model. We propose multipoint quantization, a quantization method that approximates a full-precision weight vector using a linear combination of multiple vectors of low-bit number…

Low-precision streaming PCA estimates the leading eigenvector with limited precision.

Low-precision training reduces computational cost and produces efficient models. Recent research in developing new low-precision training algorithms often relies on simulation to empirically evaluate the statistical effects of quantization while avoiding the substantial overhead of building specific hardware. To suppor…

EMPIR combines low and full precision DNNs to enhance robustness against adversarial attacks.

Lower bounds set for infinite-precision transformers.

LLMs struggle with arithmetic tasks unless they use high numerical precision.

Precision farming uses data analysis to optimize crop management.

The paper improves Bayesian precision matrix estimation for high-dimensional sparse data.

We propose a penalized likelihood framework for estimating multiple precision matrices from different classes. Most existing methods either incorporate no information on relationships between the precision matrices, or require this information be known a priori. The framework proposed in this article allows for simulta…

New proposed models are often compared to state-of-the-art using statistical significance testing. Literature is scarce for classifier comparison using metrics other than accuracy. We present a survey of statistical methods that can be used for classifier comparison using precision, accounting for inter-precision corre…

Reduced precision computation for deep neural networks is one of the key areas addressing the widening compute gap driven by an exponential growth in model size. In recent years, deep learning training has largely migrated to 16-bit precision, with significant gains in performance and energy efficiency. However, attemp…

SINGD improves KFAC for memory-efficiency and stability in low-precision training.

Gaussian graphical models (GGMs) are probabilistic tools of choice for analyzing conditional dependencies between variables in complex systems. Finding changepoints in the structural evolution of a GGM is therefore essential to detecting anomalies in the underlying system modeled by the GGM. In order to detect structur…

A new R package for high-dimensional regression and precision matrix estimation.

In this paper, we study the precise asymptotics of noncompact Type-IIb solutions to the mean curvature flow. Precisely, for each real number $γ>0$, we construct mean curvature flow solutions, in the rotationally symmetric class, with the following precise asymptotics as $t\nearrow\infty$: (1) The highest curvature conc…

Correct method found for drawing precise envelope of straight lines.

Finite-precision learning of $anh$ networks is limited by the Monte Carlo rate.

Maximizing the speed and precision of communication while minimizing power dissipation is a fundamental engineering design goal. Also, biological systems achieve remarkable speed, precision and power efficiency using poorly understood physical design principles. Powerful theories like information theory and thermodynam…

HMQ improves quantization for edge devices with mixed precision.

Diagonal transformations preserve independence structures in non-Gaussian distributions.

This work studies scaling laws for low-precision training in high-dimensional linear regression.

Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves…

Precise computations of Dehn functions for subgroups of free group products.

With ever-increasing computational demand for deep learning, it is critical to investigate the implications of the numeric representation and precision of DNN model weights and activations on computational efficiency. In this work, we explore unconventional narrow-precision floating-point representations as it relates …

The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained computing systems. Many network complexity reduction techniques have been proposed including fixed-point implementation. However, a systematic approach for designin…

Proposes PQ, a more precise Bayesian quantifier for prevalence estimation.

Optimizes CM for stochastic convex optimization with progressive precision.

The inverse covariance matrix provides considerable insight for understanding statistical models in the multivariate setting. In particular, when the distribution over variables is assumed to be multivariate normal, the sparsity pattern in the inverse covariance matrix, commonly referred to as the precision matrix, cor…

CARE method estimates precision matrix for compositional data, achieving optimality in high dimensions.

Low precision networks in the reinforcement learning (RL) setting are relatively unexplored because of the limitations of binary activations for function approximation. Here, in the discrete action ATARI domain, we demonstrate, for the first time, that low precision policy distillation from a high precision network pro…

Knowledge bases (KBs) are the backbone of many ubiquitous applications and are thus required to exhibit high precision. However, for KBs that store subjective attributes of entities, e.g., whether a movie is "kid friendly", simply estimating precision is complicated by the inherent ambiguity in measuring subjective phe…

The study formalizes temporal precision and recall for anomaly detection in sequences.

Recently there has been significant interest in training machine-learning models at low precision: by reducing precision, one can reduce computation and communication by one order of magnitude. We examine training at reduced precision, both from a theoretical and practical perspective, and ask: is it possible to train …

Deep neural networks (DNNs) can be made hardware-efficient by reducing the numerical precision of the weights and activations of the network and by improving the network's resilience to noise. However, this gain in efficiency often comes at the cost of significantly reduced accuracy. In this paper, we present a novel a…

Active learning selects both observations and annotation precision for Gaussian Processes.