MCQ uses Monte Carlo methods to efficiently quantize neural networks without re-training.
problem Efficiently quantizing neural networks for lower power consumption.
method Importance sampling for quantization of weights and activations.
result Minimal accuracy loss with quantized networks compared to full-precision networks.
Bayesian Bits unifies quantization and pruning through gradient optimization.
problem Joint mixed precision quantization and pruning for efficient neural networks.
method Gradient-based optimization with a novel bit width decomposition and learnable stochastic gates.
result Bayesian Bits achieves better accuracy vs. efficiency trade-off compared to static bit width networks.
Low-bit training framework reduces energy consumption in CNNs.
problem Reducing energy consumption in convolutional neural networks.
method Low-bit training framework using MLS tensor format with dynamic quantization.
result Achieves superior trade-off between accuracy and bit-width.
Similar to convolution neural networks, recurrent neural networks (RNNs) typically suffer from over-parameterization. Quantizing bit-widths of weights and activations results in runtime efficiency on hardware, yet it often comes at the cost of reduced accuracy. This paper proposes a quantization approach that increases…
BCGD algorithm improves training of quantized neural networks.
problem Training quantized deep neural networks at low bit-widths.
method Introduces coarse gradient descent and blended correction for training.
result BCGD achieves high accuracy in quantized neural networks.
AdaptivFloat improves deep learning inference accuracy at low precision.
problem Low precision quantization issues in deep learning inference.
method Dynamic floating-point representation with adaptive clipping.
result Consistently higher inference accuracy at low precision compared to other methods.
Improves matrix multiplication throughput for asymmetric bit-width operands.
problem Matrix multiplications between asymmetric bit-width operands, especially 8- and 4-bit, are not efficiently handled by existing SIMD instructions.
method Proposes a new SIMD matrix multiplication instruction that uses mixed precision on inputs (8- and 4-bit) and accumulates into 16-bit output, improving throughput.
result Offers 2x improvement in throughput compared to existing symmetric-operand-size instructions, with negligible overflow.
Proposes a robust neural network quantization method.
problem Training model's dependency on specific quantization methods.
method Intrinsic robustness to various quantization processes.
result Single model capable of operating at various bit-widths and policies.
We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-deman…
Study finds optimal learning rate schedules for sub-100M quantization-aware training across bit-widths.
problem Optimal learning rate schedules for quantization-aware training depend on bit-width.
method Factorial grid testing over bit-width, warmdown fraction, LR magnitude, model size, and seed.
result INT6 QAT requires a different schedule than higher-precision training, falsifying the primary hypothesis.
Paper optimizes KWS models using NAS and quantization for limited resources.
problem Developing efficient keyword spotting models in resource-constrained environments.
method Neural Architecture Search (NAS) for model structure optimization and quantization of weights and activations.
result Achieved high accuracy (95.55%) with minimal parameters and operations using NAS and quantization.
New framework reduces cost of financial option pricing simulations on FPGAs.
problem Efficiently simulate financial option pricing with reduced computational cost.
method Nested MLMC framework with low precision calculations on FPGAs.
result Higher computational savings compared to existing mixed-precision MLMC frameworks.
BIL allows binary input data in CNNs, improving performance on multimodal datasets.
problem Efficient execution of CNNs on edge devices with reduced bit width.
method BIL concept that learns bit-specific binary weights for binary input data.
result BIL outperforms full precision weights by 1.92% on multimodal datasets.
In recent years Deep Neural Networks (DNNs) have been rapidly developed in various applications, together with increasingly complex architectures. The performance gain of these DNNs generally comes with high computational costs and large memory consumption, which may not be affordable for mobile platforms. Deep model q…
Paper explores reducing precision in SVM for faster text classification.
problem Efficiency in multi-class text classification training.
method Comparison of SVM trained with reduced precision (16-bit, half) vs original.
result Reduced precision training maintains text classification accuracy.
Probabilistic BLRNet uses binary weights and activations for efficient neural networks.
problem Efficiently training and deploying deep neural networks with limited memory and compute.
method Probabilistic training method for binary weights and activations, introducing stochastic operations.
result BLRNet achieves performance comparable to full-precision networks while using fewer bits.
Meta learning optimizes neural network quantization for efficient inference.
problem Uniform bitwidth quantization is sub-optimal for neural network compression.
method Meta learning to automatically generate hybrid quantization policies.
result Meta learning outperforms uniform quantization and RL approaches.
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neura…
LSQ+ improves quantization of neural nets with Swish activations, achieving state-of-the-art results.
problem Quantization of neural nets with Swish activations, especially negative activations, leads to significant performance loss.
method Introduces learnable scale and offset parameters for asymmetric quantization, and uses MSE-based initialization for quantization parameters.
result Significantly outperforms LSQ for low-bit quantization of neural nets with Swish activations, achieving up to 5.6% gain with W2A2 quantization of EfficientNet-B0.
A novel method quantizes Batch Normalization for QNNs, maintaining accuracy and efficiency.
problem Quantization challenges in Batch Normalization for QNNs.
method Converts BN to fixed-point operation with shared scale, suitable for hardware.
result Maintains same outputs through rigorous analysis and experiments.
This work optimizes neural network bit-width and layer-width for efficiency.
problem Efficient optimization of deep neural networks for reduced size and computational demands.
method Cluster-based tree-structured Parzen estimator for surrogate modeling, Hessian-based pruning for parameter reduction.
result 20% decrease in model size with 12x reduction in search time compared to existing methods.
Improved neural quantization reduces accuracy loss to less than 1% with 4-bit weights.
problem Reducing accuracy loss in neural quantization below 8-bits.
method Layer-wise calibration and integer programming to optimize bit-width allocation.
result Less than 1% accuracy degradation with 4-bit weights and activations.
Flexible deep learning models for dynamic accuracy and speed trade-offs.
problem Dynamic accuracy and speed trade-offs in real-world applications.
method Training deep neural networks with a new method allowing flexible numerical precision during inference.
result Achieved comparable accuracy to dedicated models trained at the same precision with dynamic precision settings.
CoDeQ simplifies joint model compression by integrating pruning and quantization.
problem Joint pruning and quantization methods are complex and require additional procedures.
method CoDeQ uses a dead-zone quantizer to directly induce sparsity and learn quantization parameters.
result CoDeQ achieves high sparsity and low-precision accuracy with minimal bit operations.
New method analyzes accumulation precision in deep learning networks.
problem Lack of precision analysis for accumulation in deep learning training.
method Statistical approach to analyze partial sum accumulations and derive equations for minimum required bits.
result Reduced accumulation precision can lead to loss of information and degraded network quality.
Reduced numerical precision is a common technique to reduce computational cost in many Deep Neural Networks (DNNs). While it has been observed that DNNs are resilient to small errors and noise, no general result exists that is capable of predicting a given DNN system architecture's sensitivity to reduced precision. In …
Unified framework LPCD optimizes quantization of complex submodules.
problem Quantization of complex submodules in neural networks.
method Layer-Projected Coordinate Descent (LPCD) for quantizing arbitrary submodules.
result LPCD enhances both layer-wise PTQ methods and existing submodule approaches.
QABBA improves time series storage efficiency while preserving shape information.
problem Efficient storage and shape preservation of time series data.
method Quantized symbolic time series approximation (QABBA) using ABBA technique.
result QABBA achieves a new state-of-the-art on Monash regression dataset.
This work proposes a complete 8-bit quantization framework for large-scale deep neural networks.
problem Training large-scale deep neural networks with high performance and low memory footprint.
method WAGEUBN framework that quantizes all data paths including weights, activations, gradients, errors, updates, and batch normalization.
result Achieves competitive accuracy on the ImageNet dataset using only 8-bit integers.
New insights into quantized neural networks reveal learning dynamics and generalization errors.
problem Understanding the impact of quantization hyperparameters on learning dynamics in high-dimensional models.
method Theoretical analysis and fixed-point analysis of STE dynamics in quantized models.
result STE training in quantized models converges to a plateau followed by a sharp drop in generalization error, influenced by quantization range.
HMQ improves quantization for edge devices with mixed precision.
problem Efficient quantization for edge devices with uniform, power-of-two thresholds.
method Introduces HMQ, a mixed precision quantization block that repurposes Gumbel-Softmax for searching over quantization schemes.
result Achieves competitive and state-of-the-art results on ImageNet despite restrictions.
Role-wise data augmentation improves knowledge distillation effectiveness.
problem Existing knowledge distillation methods fail to utilize the full potential of teacher-student data interaction.
method Design and implement data augmentation agents with distinct roles for teacher and student.
result Specially tailored data points enhance the demonstration of teacher's knowledge to the student.
Quantization can be used to form new vectors/matrices with shared values close to the original. In recent years, the popularity of scalar quantization for value-sharing applications has been soaring as it has been found huge utilities in reducing the complexity of neural networks. Existing clustering-based quantization…
Paper proposes a method for estimating complex low-rank matrices from phase-only measurements.
problem Estimating complex low-rank matrices from magnitude-only measurements.
method A hierarchical prior model with a Gaussian-Wishart distribution is used to promote low-rankness. A variational EM algorithm is developed to solve the problem.
result The proposed method is less sensitive to initialization and performs well with random initialization.
New spectral methods improve matrix estimation in RL with low-rank structure.
problem Estimating matrices with low-rank structure in reinforcement learning.
method Spectral-based matrix estimation approaches.
result Spectral methods efficiently recover singular subspaces and minimize entry-wise error.
SWALP averages SGD iterates for low-precision training, improving scalability and performance.
problem Improving scalability and performance in low-precision training.
method Averages low-precision SGD iterates with a modified learning rate schedule.
result SWALP matches full-precision SGD performance with 8-bit quantization and converges to optimal solutions.
This paper improves low-precision sampling using SGHMC for deep learning models.
problem Enhancing training efficiency of deep neural networks with low-precision training.
method Investigates low-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) for both log-concave and non-log-concave distributions.
result Low-precision SGHMC achieves quadratic improvement in error compared to SGLD for non-log-concave distributions.
Algorithm recovers multiple low-rank matrices from unlabeled data.
problem Learning mixtures of low-rank models from unlabelled data.
method Three-stage meta-algorithm that copes with non-convexity and noise.
result Near-optimal sample and computational complexities under Gaussian designs.
Low-rank modeling generally refers to a class of methods that solve problems by representing variables of interest as low-rank matrices. It has achieved great success in various fields including computer vision, data mining, signal processing and bioinformatics. Recently, much progress has been made in theories, algori…
Low precision RL distillation enables real-time game play on neuromorphic hardware.
problem Limited use of low precision networks in reinforcement learning.
method Policy distillation from high precision to low precision networks.
result Low precision distillation enables real-time game play on low-power hardware.
Low-cost sensors improve air quality prediction accuracy significantly.
problem Improving air quality monitoring networks with affordable sensors.
method Developed a high-resolution air quality prediction engine using low-cost sensors and official data.
result The use of low-cost sensors improves prediction accuracy by 25% and 15% for PM2.5 and PM10 respectively in densely monitored areas.
Paper explores low-precision SGLD for neural networks, reducing costs without sacrificing performance.
problem Infeasibility of low-precision sampling in large-scale scenarios.
method Developed low-precision SGLD with quantization function and full-precision gradient accumulators.
result Low-precision SGLD achieves comparable performance to full-precision SGLD with only 8 bits.
Paper proposes a new technique to compress CNNs while maintaining accuracy.
problem CNNs struggle with traditional low-rank approximation methods, leading to degraded accuracy.
method Introduces a training technique that finds a flat minimum in low-rank approximation without a decomposed structure.
result CNN models can be compressed with higher accuracy and lower computation than conventional methods.
SGD with mini-batches can solve convex low-rank matrix problems efficiently.
problem Solving large-scale convex low-rank matrix problems efficiently.
method Stochastic Gradient Descent with mini-batches and low-rank projections.
result SGD with mini-batches produces low-rank iterates with high probability.
Novel method for efficient low-rank matrix estimation and bandit algorithms.
problem Low-rank matrix estimation and bandit problems.
method LowPopArt method for low-rank matrix estimation and novel experimental design criterion.
result Improved recovery guarantees and regret bounds for low-rank bandit algorithms.
New models improve morpheme segmentation in low-resource languages.
problem Improving morpheme segmentation in low-resource languages.
method Two new models: LSTM pointer-generator and sequence-to-sequence with hard monotonic attention.
result Novel models outperform existing ones by up to 11.4% accuracy in low-resource settings.
Paper improves MVSC using tensor low-rank modeling.
problem Improving multi-view spectral clustering.
method Structured tensor low-rank norm for MVSC optimization.
result Proposed method outperforms state-of-the-art methods.
Study subjective perception of low light restored images and develop an unsupervised QA model.
problem Lack of subjective QA for low light restored images and challenges in collecting human opinion scores.
method Create a dataset, conduct subjective QA study, develop self-supervised contrastive learning technique to extract features.
result Unsupervised NR QA model achieves state-of-the-art performance for low light restored images.