Low-bit training framework reduces energy consumption in CNNs.
problem Reducing energy consumption in convolutional neural networks.
method Low-bit training framework using MLS tensor format with dynamic quantization.
result Achieves superior trade-off between accuracy and bit-width.
SMGD trains low-bit neural networks with memory constraints.
problem Training large neural networks with limited memory.
method Stochastic Markov Gradient Descent (SMGD).
result Encouraging numerical results and theoretical guarantees.
Low bit-width integer weights and activations are very important for efficient inference, especially with respect to lower power consumption. We propose Monte Carlo methods to quantize the weights and activations of pre-trained neural networks without any re-training. By performing importance sampling we obtain quantiz…
Kernel Quantization improves CNN compression without sacrificing performance.
problem Efficiently compressing CNN models without significant performance loss.
method Quantizes convolution kernels as the unit, learning a codebook for low-bit indexes.
result Significant compression ratio achieved with minimal accuracy loss.
LSQ+ improves quantization of neural nets with Swish activations, achieving state-of-the-art results.
problem Quantization of neural nets with Swish activations, especially negative activations, leads to significant performance loss.
method Introduces learnable scale and offset parameters for asymmetric quantization, and uses MSE-based initialization for quantization parameters.
result Significantly outperforms LSQ for low-bit quantization of neural nets with Swish activations, achieving up to 5.6% gain with W2A2 quantization of EfficientNet-B0.
In order to efficiently transmit and store speech signals, speech codecs create a minimally redundant representation of the input signal which is then decoded at the receiver with the best possible perceptual quality. In this work we demonstrate that a neural network architecture based on VQ-VAE with a WaveNet decoder …
Although deep neural networks are highly effective, their high computational and memory costs severely challenge their applications on portable devices. As a consequence, low-bit quantization, which converts a full-precision neural network into a low-bitwidth integer version, has been an active and promising research t…
Proposes QEP to mitigate quantization error propagation in layer-wise post-training quantization.
problem Growth of quantization errors across layers degrades performance, especially in low-bit regimes.
method Quantization Error Propagation (QEP) framework that explicitly propagates and compensates for quantization errors.
result QEP-enhanced layer-wise PTQ achieves substantially higher accuracy, especially in low-bit regimes.
Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network qu…
Diffusion models improve image compression at low bit-rates.
problem Efficiently compressing images at very low bit-rates.
method Encoding into an embedding, using diffusion models to refine the embedding iteratively.
result Realistic reconstructions can be generated at extremely low bit-rates.
Meta learning optimizes neural network quantization for efficient inference.
problem Uniform bitwidth quantization is sub-optimal for neural network compression.
method Meta learning to automatically generate hybrid quantization policies.
result Meta learning outperforms uniform quantization and RL approaches.
Quantized deep neural networks (QDNNs) are attractive due to their much lower memory storage and faster inference speed than their regular full precision counterparts. To maintain the same performance level especially at low bit-widths, QDNNs must be retrained. Their training involves piecewise constant activation func…
Recent machine learning methods use increasingly large deep neural networks to achieve state of the art results in various tasks. The gains in performance come at the cost of a substantial increase in computation and storage requirements. This makes real-time implementations on limited resources hardware a challenging …
Low bit-width weights and activations are an effective way of combating the increasing need for both memory and compute power of Deep Neural Networks. In this work, we present a probabilistic training method for Neural Network with both binary weights and activations, called BLRNet. By embracing stochasticity during tr…
New framework reduces LLM complexity by directly finetuning in Boolean domain.
problem Reducing the complexity of large language models (LLMs) while maintaining performance.
method Proposes a novel framework using multi-kernel Boolean parameters for direct finetuning in the Boolean domain.
result Significantly reduces complexity during both finetuning and inference, outperforming recent techniques.
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neura…
A new quantization strategy reduces Transformer model size and inference time.
problem Heavy computation load and memory overhead in Transformer models for mobile devices.
method Mixed precision quantization with varying bits per word in embedding blocks.
result 11.8x smaller model size and 3.5x speed up for on-device NMT.
Neural network models are resource hungry. It is difficult to deploy such deep networks on devices with limited resources, like smart wearables, cellphones, drones, and autonomous vehicles. Low bit quantization such as binary and ternary quantization is a common approach to alleviate this resource requirements. Ternary…
Post-training quantization method using multiple low-precision points achieves higher precision for critical weights.
problem Discretizing pre-trained deep neural networks without re-training.
method Multipoint quantization with efficient greedy selection and adaptive point number.
result Outperforms state-of-the-art methods on ImageNet classification and PASCAL VOC object detection.
We present any-precision deep neural networks (DNNs), which are trained with a new method that allows the learned DNNs to be flexible in numerical precision during inference. The same model in runtime can be flexibly and directly set to different bit-widths, by truncating the least significant bits, to support dynamic …
Moniqua improves SGD convergence with quantized communication.
problem Efficiently communicating in decentralized SGD with limited bandwidth.
method Modulo quantized communication in decentralized SGD.
result Moniqua converges at the same rate as full-precision communication with less bits.
Recent research implies that training and inference of deep neural networks (DNN) can be computed with low precision numerical representations of the training/test data, weights and gradients without a general loss in accuracy. The benefit of such compact representations is twofold: they allow a significant reduction o…
Implementing large-scale deep neural networks with high computational complexity on low-cost IoT devices may inevitably be constrained by limited computation resource, making the devices hard to respond in real-time. This disjunction makes the state-of-art deep learning algorithms, i.e. CNN (Convolutional Neural Networ…
New method compresses facial videos using GANs and latent space optimization.
problem Efficiently compressing facial videos at low bit rates.
method Leverages StyleGAN for latent space representation and compression, learns optimal compression through entropy model and perceptual loss.
result Significantly reduces perceptual distortion at low bit rates compared to state-of-the-art codecs.
Quantization-aware training can recover accuracy lost by post-training quantization.
problem Post-training quantization (PTQ) can fail sharply at aggressive bitwidths.
method A unified geometric framework that explains PTQ failure and QAT recovery.
result QAT has a useful bias that steers iterates back into the basin.
Deep neural network (DNN) quantization converting floating-point (FP) data in the network to integers (INT) is an effective way to shrink the model size for memory saving and simplify the operations for compute acceleration. Recently, researches on DNN quantization develop from inference to training, laying a foundatio…
MSD removes dequantization bottleneck in LLM inference by approximating high-precision activations.
problem Dequantization bottleneck in LLM inference on modern AI accelerators.
method MSD decomposes high-precision activations into multiple low-precision components for direct multiplication with quantized weights.
result MSD avoids INT8-to-BF16 weight conversion, reducing dequantization cycles and HBM traffic.
Constrained sequence codes have been widely used in modern communication and data storage systems. Sequences encoded with constrained sequence codes satisfy constraints imposed by the physical channel, hence enabling efficient and reliable transmission of coded symbols. Traditional encoding and decoding of constrained …
Study quantization effects on high-dimensional linear regression learning.
problem Understanding quantization's impact on learning high-dimensional linear regression models.
method Analyzes stochastic gradient descent for high-dimensional linear regression under various quantization targets.
result Establishes precise bounds on excess risk for different quantization schemes.
Paper introduces adversarial lossy compression for video artifacts reduction.
problem Unpleasant reconstruction artifacts in standard video coding schemes at low bit-rates.
method Adversarial lossy video compression model minimizing an adversarial distortion objective.
result Reduction of perceptual artifacts and detail reconstruction under extreme compression.
Weight quantization is one of the most important techniques of Deep Neural Networks (DNNs) model compression method. A recent work using systematic framework of DNN weight quantization with the advanced optimization algorithm ADMM (Alternating Direction Methods of Multipliers) achieves one of state-of-art results in we…
New quantization methods improve accuracy of Random Fourier Features.
problem Improving accuracy of Random Fourier Features for machine learning.
method Sigma-Delta and distributed noise-shaping quantization methods for 1-bit and low bit-depth quantization.
result Quantized RFFs allow high accuracy approximation of underlying kernels with polynomial error decay.
Enhances LLM quantization with MDBF, improving perplexity and accuracy.
problem Limited performance of Double Binary Factorization in extreme quantization.
method Introduces Multi-envelope DBF, retaining sign matrices and replacing single envelope with rank-l envelope. result Improves perplexity and zero-shot accuracy over previous binary formats.
A novel method quantizes Batch Normalization for QNNs, maintaining accuracy and efficiency.
problem Quantization challenges in Batch Normalization for QNNs.
method Converts BN to fixed-point operation with shared scale, suitable for hardware.
result Maintains same outputs through rigorous analysis and experiments.
QABBA improves time series storage efficiency while preserving shape information.
problem Efficient storage and shape preservation of time series data.
method Quantized symbolic time series approximation (QABBA) using ABBA technique.
result QABBA achieves a new state-of-the-art on Monash regression dataset.
This paper proposes a new method for efficient data compression using Bayesian neural networks.
problem Efficient compression of data represented as functions mapping coordinates to signal values.
method Overfitting variational Bayesian neural networks to the data and compressing an approximate posterior weight sample using relative entropy coding.
result Our method achieves strong performance on image and audio compression while retaining simplicity.
Bayesian Bits unifies quantization and pruning through gradient optimization.
problem Joint mixed precision quantization and pruning for efficient neural networks.
method Gradient-based optimization with a novel bit width decomposition and learnable stochastic gates.
result Bayesian Bits achieves better accuracy vs. efficiency trade-off compared to static bit width networks.
Taylorized training improves neural network training at finite width.
problem Understanding and improving neural network training at finite width.
method Training the k-th order Taylor expansion of the neural network at initialization.
result Taylorized training agrees with full neural network training better as k increases and can significantly close the performance gap.
Self-training outperforms pre-training on COCO object detection and segmentation datasets.
problem The effectiveness of pre-training in improving object detection and segmentation models is limited.
method Investigated self-training as an alternative method to utilize additional data.
result Self-training consistently improves model performance across various dataset sizes and data augmentation levels.
Un-trained neural networks outperform trained methods in MRI reconstruction.
problem Accelerated MRI reconstruction with minimal training data.
method Variation of Deep Decoder without training data.
result Un-trained approach significantly outperforms other methods in reconstruction accuracy.
Meta-learning performance depends on train-validation split type.
problem Understanding the importance of train-validation split in meta-learning.
method Theoretical and experimental study comparing train-val and train-train methods.
result Train-train method can achieve strictly better excess loss in realizable cases.
New method reveals how training data influence diffusion model outputs.
problem Difficulty in assessing training data impact on diffusion model outputs.
method Use of ensembles trained on carefully engineered splits of training data to identify influential training examples.
result Demonstrated the viability of ensembles as generative models and validity of assessing influence.
Paper shows adversarial training can be fooled by new type of noise.
problem Adversarial training can be fooled by new types of noise.
method Designing ADVIN, a new type of inducing noise.
result ADVIN can degrade adversarial training robustness by 99.9%.
New MIP methods improve training of integer-valued neural networks.
problem Training integer-valued neural networks with limited data and resources.
method Formulated new MIP models to optimize training efficiency and handle more data.
result Significantly outperforms previous state-of-the-art methods in accuracy, training time, and data usage.
Free adversarial training reduces the generalization gap compared to vanilla method.
problem Improving generalization in adversarial training.
method Analysis of algorithmic stability in free adversarial training.
result Free adversarial training shows a lower generalization gap.
Making neural networks robust against adversarial inputs has resulted in an arms race between new defenses and attacks. The most promising defenses, adversarially robust training and verifiably robust training, have limitations that restrict their practical applications. The adversarially robust training only makes the…
learn2mix trains neural nets faster by adjusting class proportions dynamically.
problem Training neural nets efficiently with limited resources and imbalanced classes.
method Adaptive class proportion adjustment during training.
result Neural nets trained with learn2mix converge faster than static methods.
Loss-guided training accelerates node embedding methods on graphs.
problem Training efficiency in graph learning methods with implicit positive examples.
method Dynamic adjustment of training distribution based on loss values.
result Significant acceleration in training and computation over static methods.