Paper introduces a technique to simplify RNN policies for better understanding and analysis.
problem Difficulty in explaining and analyzing RNN policies due to continuous-valued memory vectors and observation features.
method Quantized Bottleneck Insertion technique to learn finite representations of RNN vectors and features.
result Finite representations of RNN policies can be as small as 3 discrete memory states and 10 observations, improving interpretability.
The paper interprets VQ-VAE loss as a form of information bottleneck.
problem Understanding the VQ-VAE loss function.
method Interpreted VQ-VAE loss as variational deterministic information bottleneck (VDIB) and variational information bottleneck (VIB).
result VQ-VAE loss can be derived from VDIB and approximated by VIB.
AgrLearn improves deep learning by optimizing multi-sample quantization.
problem Improving deep learning performance with fewer training samples.
method Develops AgrLearn, a vector IB quantization-based framework.
result Significant improvements in image recognition and text classification.
FrostNet improves INT8 quantization efficiency in mobile networks.
problem The importance of network architecture for optimal INT8 quantization.
method Quantization-aware training (QAT) with StatAssist and GradBoost, hardware-aware NAS.
result FrostNets achieve higher recognition accuracy with comparable latency when quantized.
This paper introduces a quantization-based regularizer for autoencoders to improve latent representations.
problem Autoencoders can overfit and collapse, leading to poor latent representations.
method The authors combine VQ-VAE and denoising methods to introduce a bottleneck Bayesian estimator that soft quantizes latent codes.
result The method results in better latent representations for supervised and clustering tasks.
FleXOR trains fractional quantization for neural networks, improving accuracy and size.
problem Quantization limits to integer bits restricts compression and accuracy.
method Encryption algorithm with XOR gates for fractional bits during inference.
result FleXOR achieves high accuracy with fractional sub-1-bit weights.
MSD removes dequantization bottleneck in LLM inference by approximating high-precision activations.
problem Dequantization bottleneck in LLM inference on modern AI accelerators.
method MSD decomposes high-precision activations into multiple low-precision components for direct multiplication with quantized weights.
result MSD avoids INT8-to-BF16 weight conversion, reducing dequantization cycles and HBM traffic.
This work reduces model size by 86.11% for recommender systems using 4-bit quantization.
problem Large memory consumption in embedding vectors for recommender systems.
method Post-training 4-bit quantization on embedding tables, including row-wise uniform quantization and codebook-based quantization.
result Consistently reduces accuracy degradation while significantly reducing model size.
Qsparse-local-SGD reduces communication in large-scale learning models.
problem Communication bottleneck in distributed optimization of large-scale models.
method Combines sparsification, quantization, and local computation with error compensation.
result Converges at the same rate as vanilla distributed SGD for many sparsifiers and quantizers.
EQ-Net combines LLR estimation and quantization using deep learning.
problem Unified solution for LLR estimation and quantization.
method Two-stage algorithm using LLR compression as a pretext task.
result Achieves state-of-the-art results with gains in efficiency and latency.
PCBM improves neural network generalization by partially observing concepts.
problem Decreased generalization performance due to observing all concepts in CBM.
method Developed a theoretical analysis of PCBM's Bayesian generalization error.
result PCBM's generalization error is lower than CBM's due to partial concept observation.
AdaQuantFL reduces communication in federated learning by adaptively quantizing model updates.
problem Efficient communication of model updates in federated learning with high-dimensional models and limited bandwidth.
method AdaQuantFL uses adaptive quantization to reduce the number of bits for model updates while maintaining low error floor.
result AdaQuantFL converges in fewer communicated bits compared to fixed quantization levels, with minimal impact on accuracy.
RFX accelerates and compresses Random Forests for large datasets.
problem Memory bottleneck in proximity matrices limits Random Forest analysis.
method QLORA compression, CPU TriBlock storage, GPU batch sizing, 3D MDS visualization.
result Proximity-based Random Forest analysis on larger datasets is feasible.
We consider the problem of decentralized consensus optimization, where the sum of n smooth and strongly convex functions are minimized over n distributed agents that form a connected network. In particular, we consider the case that the communicated local decision variables among nodes are quantized in order to all…
WrapNet optimizes inference for low-resolution neural networks by using 8-bit additions.
problem Reducing multiplication complexity in low-resolution neural networks.
method Adapting neural networks to use low-resolution (8-bit) additions in accumulators, with a cyclic activation layer and overflow penalty regularizer.
result Achieves comparable classification accuracy to 32-bit counterparts using low-resolution additions.
DJPQ optimizes neural network pruning and quantization for hardware efficiency.
problem Efficiently compress neural networks for hardware inference.
method Joint gradient-based optimization of pruning and quantization into a differentiable loss function.
result Significant reduction in Bit-Operations (BOPs) with minimal accuracy loss.
Transformer improves sequence generation with insertion and deletion phases.
problem Sequence generation challenges in machine translation.
method Insertion-Deletion Transformer with iterative insertion and deletion phases.
result Significant BLEU score improvement over insertion-only models.
A new framework for neural network classification using vector quantization.
problem Learning a neural network classifier under the IB principle.
method Aggregated Learning framework, combining vector quantization and variational techniques.
result The effectiveness of Aggregated Learning verified through experiments.
Global-QSGD accelerates distributed training by up to 3.51%.
problem High communication overhead in distributed deep learning.
method Allreduce-compatible gradient quantization with theoretical guarantees.
result Global-QSGD accelerates distributed training by up to 3.51%.
Novel algorithm reduces delays and communication in decentralized learning.
problem Decentralized learning with straggling nodes and high communication costs.
method QuanTimed-DSGD: deadline-imposed gradient computation and quantized model exchange.
result Converges to global optimal for convex functions, finds first-order stationary points for non-convex.
This paper enables deep network inference on microcontrollers with improved accuracy and reduced memory usage.
problem Deploying deep networks on resource-constrained edge-devices with low memory and computational constraints.
method Mixed low-bitwidth compression, rule-based iterative procedure for bit precision determination, quantization-aware retraining, and integer-only model conversion.
result Improved Top1 accuracy of 68% on a 2MB FLASH memory STM32H7 microcontroller, 8% higher than 8-bit implementations.
Recent research implies that training and inference of deep neural networks (DNN) can be computed with low precision numerical representations of the training/test data, weights and gradients without a general loss in accuracy. The benefit of such compact representations is twofold: they allow a significant reduction o…
Insertion Transformer generates sequences with flexible insertion orders.
problem Generating sequences with arbitrary orderings.
method Iterative, partially autoregressive model with insertion operations.
result Outperforms non-autoregressive approaches in translation tasks.
PRISM-VQ combines financial priors with vector quantization for better stock prediction.
problem Predicting cross-sectional stock returns is hard due to low signal-to-noise ratios and changing market conditions.
method Integrates expert priors, vector-quantized latent factors, and dynamic factor loadings.
result Consistent improvements in cross-sectional return prediction and portfolio performance.
New methods improve distributed optimization on non-iid data.
problem Communication bottleneck in distributed machine learning models.
method Two types of distributed gradient compression methods (D-QSGD and D-EF-SGD) analyzed for non-iid data.
result D-EF-SGD performs better than D-QSGD on non-iid data but can still slow down with high data skewness.
Study clarifies Bayesian generalization error in CBM for 3-layered linear neural networks.
problem Understanding the generalization error in concept bottleneck models.
method Mathematical analysis of Bayesian generalization error and free energy in CBM for 3-layered linear neural networks.
result CBM significantly alters the parameter region and Bayesian generalization error compared to standard models.
Improved training for VQ-VAE models with robust codebook learning.
problem Challenges in training discrete latent variable models, especially VQ-VAEs.
method Increased learning rate and periodic re-initialization of codebook for robust training.
result More robust training and increased usage of latent codewords, even for large codebooks.
Comparing with traditional learning criteria, such as mean square error (MSE), the minimum error entropy (MEE) criterion is superior in nonlinear and non-Gaussian signal processing and machine learning. The argument of the logarithm in Renyis entropy estimator, called information potential (IP), is a popular MEE cost i…
Improved vector quantization using Gaussian mixtures for better codebook utilization.
problem Training instability and information loss in discrete vector quantization.
method Generalized vector quantization with Gaussian mixture model and aggregated categorical posterior evidence lower bound.
result GM-VQ improves codebook utilization and reduces information loss without heuristics.
One of the most significant bottleneck in training large scale machine learning models on parameter server (PS) is the communication overhead, because it needs to frequently exchange the model gradients between the workers and servers during the training iterations. Gradient quantization has been proposed as an effecti…
DIANA compresses gradient differences for distributed learning.
problem Learning gradients for convergence in distributed learning.
method Compression of gradient differences in a distributed setting.
result DIANA achieves superior convergence rates compared to existing methods.
FedPAQ improves federated learning efficiency by averaging and quantizing updates.
problem Communication bottlenecks and scalability issues in federated learning.
method Periodic averaging, partial device participation, and quantized message-passing.
result FedPAQ achieves near-optimal theoretical guarantees and demonstrates communication-computation tradeoffs.
ID-ExpO fine-tunes neural networks for more faithful explanations.
problem Improving the faithfulness of explanations for complex machine learning models.
method Differentiable insertion/deletion metric-aware regularizers for optimization.
result Fine-tuned predictors produce more faithful explanations.
XL-Editor improves sentence post-editing using XLNet's variable-length insertion probability.
problem Post-editing sentences to refine generated text.
method XL-Editor trains XLNet to estimate variable-length insertion probabilities and apply post-editing operations.
result XL-Editor outperforms XLNet on text insertion and deletion tasks, and achieves significant style transfer improvements.
In this work we explore the use of metric index structures, which accelerate nearest neighbor queries, in the scenario where we need to interleave insertions and queries during deployment. This use-case is inspired by a real-life need in malware analysis triage, and is surprisingly understudied. Existing literature ten…
New framework for consistent submodular maximization with insertions and deletions.
problem Maintaining near-optimal solutions in a dynamic setting with insertions and deletions.
method Developed a general framework for fully dynamic submodular maximization, instantiated for cardinality and rank-k matroid constraints.
result First constant-factor approximations with sublinear consistency for both cardinality and rank-k matroid constraints.
A new approach for efficient data compression in split DNN computing.
problem Optimizing data compression for DNN models split between mobile devices and edge servers.
method Systematic design and training of bottleneck units that can be inserted at the split point.
result Achieves excellent rate-distortion performance with minimal compute and storage overhead.
HSQ reduces communication costs in federated learning.
problem High cost of communicating gradients in federated learning.
method Hyper-sphere quantization (HSQ) framework for efficient gradient compression.
result HSQ achieves O(logd) per-iteration communication cost, significantly reducing costs without compromising accuracy. New neural network model measures systematic risk without feature engineering.
problem Measuring systematic risk with limited prior knowledge.
method Neural network factor model that automatically selects suitable factors.
result Achieves performance comparable to existing models without feature engineering.
New attacks improve privacy audits by analyzing model updates.
problem Improving privacy audits of AI models through sequence analysis.
method Developed SeMI attacks to identify target insertions in model sequences.
result SeMI attacks achieve higher power and tighter privacy audits.
Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning abstractions that are more useful to new tasks. There has been a surge in interest in discrete latent variable models, however, despite several recent improvements, the training of discrete latent variable m…
Foundation models fail to preserve continuous geometry, identified as the Geometric Alignment Tax.
problem Continuous geometry is lost in foundation models due to discrete categorical bottlenecks.
method Controlled ablations on synthetic systems and evaluation of 14 biological models using rate-distortion theory and MINE.
result Replacing cross-entropy with a continuous head reduces geometric distortion by up to 8.5x.
Models learn to transform musical scores into expressive performances.
problem Creating large volumes of paired data for sequence transformations.
method Adapted GAN techniques to sequences, creating paired data through simple transformations.
result Models can plausibly invert transformations like quantization and removing voices.
Improved SGD reduces communication overhead by sparsifying gradients.
problem Communication overhead in distributed optimization.
method Top-k sparsification and error compensation.
result Communication can be reduced by a factor of the problem dimension.
The paper compares inserting and stretching points for grid refinement near critical points.
problem Decreased accuracy of finite difference methods near discontinuities.
method Comparison of inserting and stretching points for grid refinement near critical points.
result Proposes a new fast and simple stretching function.
CMT efficiently manages memory by inserting and querying memories in logarithmic time.
problem Managing large memory stores efficiently for quick access and updates.
method Designing a Contextual Memory Tree (CMT) that inserts and retrieves memories in logarithmic time.
result CMT improves classification algorithms and image-captioning tasks, demonstrating better computational efficiency.
BEER accelerates decentralized nonconvex optimization to O(1/T) rate.
problem Communication bottleneck in decentralized machine learning.
method Communication-compressed algorithm with gradient tracking.
result Converges at O(1/T) rate, matching uncompressed performance. Shared Keyboard design improves phase I clinical trials by borrowing information across doses.
problem Interim decisions based on current dose data may overlook signals from neighboring doses.
method Bayesian model-assisted design using Beta kernel process with kernel-weighted pseudo-counts.
result Significant improvements in identifying maximum tolerated dose and safety.