EMPIR combines low and full precision DNNs to enhance robustness against adversarial attacks.
problem Vulnerability of DNNs to adversarial attacks that misclassify inputs with small perturbations.
method Ensemble of quantized DNN models with different numerical precisions.
result EMPIR ensembles increase adversarial robustness by 42.6% on average across different tasks.
Efficient hybrid networks improve AI performance at the edge.
problem Achieving AI performance at the edge with minimal energy and memory usage.
method Proposed hybrid networks combining binary and full-precision layers.
result Hybrid networks achieve close to full-precision performance with up to 21.8x memory compression.
A new method quantizes neural networks to low-precision without STE, improving accuracy.
problem Quantization of neural networks to low-precision without a complete theoretical understanding.
method Alpha-blending (AB) using stochastic gradient descent (SGD) to quantize weights and gradually increase the coefficient α. result Improves top-1 accuracy by 0.9% on 1-bit BinaryNet, 0.82% on 8-bit MobileNet v1, and 2.93% on 4-bit ResNet_50 v1/2 compared to STE.
Paper explores low-precision SGLD for neural networks, reducing costs without sacrificing performance.
problem Infeasibility of low-precision sampling in large-scale scenarios.
method Developed low-precision SGLD with quantization function and full-precision gradient accumulators.
result Low-precision SGLD achieves comparable performance to full-precision SGLD with only 8 bits.
Low-precision computation is often used to lower the time and energy cost of machine learning, and recently hardware accelerators have been developed to support it. Still, it has been used primarily for inference - not training. Previous low-precision training algorithms suffered from a fundamental tradeoff: as the num…
BCGD algorithm improves training of quantized neural networks.
problem Training quantized deep neural networks at low bit-widths.
method Introduces coarse gradient descent and blended correction for training.
result BCGD achieves high accuracy in quantized neural networks.
Neural networks with low-precision weights and activations offer compelling efficiency advantages over their full-precision equivalents. The two most frequently discussed benefits of quantization are reduced memory consumption, and a faster forward pass when implemented with efficient bitwise operations. We propose a t…
We simplify neural network training by assigning minimal precision to all parameters, speeding up convergence.
problem Slow and resource-intensive neural network training on constrained systems.
method Analytical precision assignment for activations, weights, gradients, and accumulators.
result Empirically validated near-optimal precision assignment for four datasets, reducing complexity.
SWALP averages SGD iterates for low-precision training, improving scalability and performance.
problem Improving scalability and performance in low-precision training.
method Averages low-precision SGD iterates with a modified learning rate schedule.
result SWALP matches full-precision SGD performance with 8-bit quantization and converges to optimal solutions.
BatchNorm helps train quantized networks by avoiding gradient explosion.
problem Training quantized neural networks is difficult due to gradient issues.
method Investigated the impact of BatchNorm on both full-precision and quantized networks.
result BatchNorm avoids gradient explosion in quantized networks, contrary to expectations.
The precise modeling of subatomic particle interactions and propagation through matter is paramount for the advancement of nuclear and particle physics searches and precision measurements. The most computationally expensive step in the simulation pipeline of a typical experiment at the Large Hadron Collider (LHC) is th…
High-dimensional inference for sparse spectral precision matrices
problem Inference on the spectral precision matrix at a fixed frequency
method Full likelihood-based inference using neighboring discrete Fourier transforms
result Simultaneous control of regularization, finite-sample truncation, and smoothing biases
MCQ uses Monte Carlo methods to efficiently quantize neural networks without re-training.
problem Efficiently quantizing neural networks for lower power consumption.
method Importance sampling for quantization of weights and activations.
result Minimal accuracy loss with quantized networks compared to full-precision networks.
We define generalized currents associated with immersions of abstract oriented solenoids with a transversal measure. We realize geometrically the full real homology of a compact manifold with these generalized currents, and more precisely with immersions of minimal uniquely ergodic solenoids. This makes precise and geo…
Ternary MobileNets improve efficiency and accuracy on constrained devices.
problem Efficiently compressing MobileNets for real-time applications on constrained devices.
method Per-layer hybrid filter banks for ternary quantization of MobileNets.
result 27.98% energy savings and 51.07% reduction in model size with comparable accuracy.
New model uses Half-Full/Half-Empty approach for better portfolio selection.
problem Improving portfolio selection through behavioral finance.
method Generalized Half-Full/Half-Empty approach to positive/negative lotteries, developing nonconvex optimization and mixed-integer linear programming models.
result The Half-Full/Half-Empty model outperforms other methods in risk and profitability.
We consider complex projective structures on Riemann surfaces and their groups of projective automorphisms. We show that the structures achieving the maximal possible number of projective automorphisms allowed by their genus are precisely the Fuchsian uniformizations of Hurwitz surfaces by hyperbolic metrics. More gene…
New satellite knots found that can't be represented by positive braids with full twists.
problem Satellite knots that cannot be represented by positive braids with full twists.
method Analyzing positive minimal braids and their closures to find satellite knots.
result Infinitely many satellite knots with Lorenz patterns and companions are not Lorenz knots.
Paper tackles score following in full-page sheet music images.
problem Score following in unprocessed sheet music images.
method Directly predicts score positions from audio and image input.
result Outperforms state-of-the-art methods in alignment precision.
Post-training quantization method using multiple low-precision points achieves higher precision for critical weights.
problem Discretizing pre-trained deep neural networks without re-training.
method Multipoint quantization with efficient greedy selection and adaptive point number.
result Outperforms state-of-the-art methods on ImageNet classification and PASCAL VOC object detection.
Paper generates full events from partons using machine learning.
problem Challenges of multiplicity variations between parton and reconstructed object spaces.
method Employing transformers, score-based models, and normalizing flows.
result Achieves remarkably accurate results in generating full events.
A method improves deep network accuracy with low precision quantization.
problem Maintaining high accuracy in low precision deep networks.
method Learned Step Size Quantization, improving quantizer configuration and gradient estimation.
result Achieves highest accuracy on ImageNet with 2-4 bit precision models.
For fast and energy-efficient deployment of trained deep neural networks on resource-constrained embedded hardware, each learned weight parameter should ideally be represented and stored using a single bit. Error-rates usually increase when this requirement is imposed. Here, we report large improvements in error rates …
We introduce a novel scheme to train binary convolutional neural networks (CNNs) -- CNNs with weights and activations constrained to {-1,+1} at run-time. It has been known that using binary weights and activations drastically reduce memory size and accesses, and can replace arithmetic operations with more efficient bit…
DCTR uses neural networks to improve particle physics simulations and parameter tuning.
problem High computational cost limits precise scientific analysis in particle physics.
method Deep neural networks for reweighting and parameter tuning of simulations.
result DCTR enables precise simulations and parameter tuning, improving model accuracy.
CodeX improves DNN acceleration on FPGAs by encoding and customizing bitwidth.
problem Efficiently accelerating deep neural networks on FPGAs with limited memory.
method Nonlinear encoding, automated bitwidth customization, FPGA streaming buffers.
result Average 4.65x throughput improvement on MNIST, SVHN, CIFAR-10.
This paper improves low-precision sampling using SGHMC for deep learning models.
problem Enhancing training efficiency of deep neural networks with low-precision training.
method Investigates low-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) for both log-concave and non-log-concave distributions.
result Low-precision SGHMC achieves quadratic improvement in error compared to SGLD for non-log-concave distributions.
Quantization-aware training can recover accuracy lost by post-training quantization.
problem Post-training quantization (PTQ) can fail sharply at aggressive bitwidths.
method A unified geometric framework that explains PTQ failure and QAT recovery.
result QAT has a useful bias that steers iterates back into the basin.
Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towa…
Paper proposes training deep neural networks with 8-bit floating point precision.
problem Challenges in training deep neural networks at 8-bit precision due to higher precision and dynamic range requirements.
method Proposes a method to train deep neural networks using 8-bit floating point for weights, activations, errors, and gradients. Introduces an enhanced loss scaling method and stochastic rounding technique.
result Demonstrates state-of-the-art accuracy across multiple datasets and workloads compared to full precision baseline.
This work studies scaling laws for low-precision training in high-dimensional linear regression.
problem Optimizing trade-off between model quality and training costs in high-dimensional linear regression.
method Theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression framework, analyzing multiplicative and additive quantization.
result Multiplicative quantization maintains full-precision model size, while additive quantization reduces effective model size.
We study convergence properties of the full truncation Euler scheme for the Cox-Ingersoll-Ross process in the regime where the boundary point zero is inaccessible. Under some conditions on the model parameters (precisely, when the Feller ratio is greater than three), we establish the strong order 1/2 convergence in $L^…
The paper tackles sparse graph learning under Laplacian-related constraints, improving upon existing methods.
problem Learning a sparse undirected graph from multivariate data under Laplacian-related constraints.
method Modifications to penalized log-likelihood approaches to enforce total positivity and lasso/adaptive lasso penalties using ADMM.
result The proposed constrained adaptive lasso approach significantly outperforms existing Laplacian-based approaches.
Low-precision quantization improves kernel approximation under memory constraints.
problem Training kernel approximation methods efficiently with limited memory.
method Low-precision quantization of random Fourier features (LP-RFFs).
result LP-RFFs can match the performance of full-precision RFFs and Nyström method with significantly less memory.
FQ-Conv quantizes CNNs for efficient inference with low-precision weights and activations.
problem Reducing precision in DNNs leads to reduced accuracy.
method Fully quantized convolutional neural networks (FQ-Conv) using novel quantization and training techniques.
result Ternary-weight CNNs perform nearly as well as full-precision networks.
Paper proposes a new confidence dimension to measure DNN generalization.
problem Measuring the generalization ability of deep neural networks is challenging.
method Introduces confidence dimension (CD) based on Hoeffding's inequality and VC-dimension.
result CD provides a feasible framework to calculate the upper bound of generalization.
New techniques improve distributed training with compressed gradients.
problem Gradient mismatch problem in local error feedback.
method Step-ahead error feedback and error averaging techniques.
result Our methods handle gradient mismatch and train faster than full-precision training.
The study analyzes convergence of adaptive optimizers under low-precision training.
problem Understanding why low-precision training remains effective for large models.
method Developed a theoretical framework for analyzing convergence of adaptive optimizers under floating-point quantization.
result Adaptive optimizers retain convergence rates close to full-precision methods under logarithmic mantissa scaling.
Study shows deformations of quaternionic Kähler manifolds are locally inhomogeneous.
problem Understanding deformations of quaternionic Kähler manifolds.
method Proved one-loop deformation of quaternionic Kähler manifolds are locally inhomogeneous.
result Full isometry group of one-loop deformations has cohomogeneity one.
We establish a full h−principle (C0−close, relative, parametric) for the simplification of singularities of Lagrangian and Legendrian fronts. More precisely, we prove that if there is no homotopy theoretic obstruction to simplifying the singularities of tangency of a Lagrangian or Legendrian submanifold with respe…
This research bridges binary and spiking neural networks for efficient on-chip AI.
problem Reducing compute requirements in machine learning frameworks.
method Training Spiking Neural Networks in extreme quantization regime and utilizing standard training techniques for conversion.
result Training Spiking Neural Networks in extreme quantization regime achieves near full precision accuracies.
Many modern statistical applications ask for the estimation of a covariance (or precision) matrix in settings where the number of variables is larger than the number of observations. There exists a broad class of ridge-type estimators that employs regularization to cope with the subsequent singularity of the sample cov…
We define generalized currents associated with immersions of abstract solenoids with a transversal measure. We realize geometrically the full real homology of a compact manifold with these generalized currents, and more precisely with immersions of minimal uniquely ergodic solenoids. This makes precise and geometric De…
We study a regular closure operator in the category of quandles. We show that the regular closure operator and the pullback closure operator corresponding to the reflector from the category of quandles to its full subcategory of trivial quandles coincide, we give a simple description of this closure operator, and analy…
FullyQT quantizes Transformer models to 8-bit precision without sacrificing translation quality.
problem Reducing computational costs of neural machine translation methods without performance loss.
method Proposes FullyQT, an all-inclusive quantization strategy for the Transformer.
result Achieves state-of-the-art quantization results, maintaining or improving translation quality.
Framework generates precise synthetic populations for scalable modeling.
problem Generating accurate synthetic populations without personal data.
method Constraint-programming framework encoding aggregated statistics and structural relations.
result Exact control of demographic profiles without requiring microdata.
We algorithmically construct multi-output Gaussian process priors which satisfy linear differential equations. Our approach attempts to parametrize all solutions of the equations using Gröbner bases. If successful, a push forward Gaussian process along the paramerization is the desired prior. We consider several exampl…
Recently there has been significant interest in training machine-learning models at low precision: by reducing precision, one can reduce computation and communication by one order of magnitude. We examine training at reduced precision, both from a theoretical and practical perspective, and ask: is it possible to train …