Study on distributed coordinate descent with quantized updates for finite precision communication.
problem Finite precision communication limits the accuracy of updates in distributed coordinate descent.
method Introduced a randomized distributed coordinate descent algorithm with quantized updates, derived convergence conditions, and validated with experiments.
result Algorithm with quantized updates converges under certain conditions on the quantization error.
AdaQuantFL reduces communication in federated learning by adaptively quantizing model updates.
problem Efficient communication of model updates in federated learning with high-dimensional models and limited bandwidth.
method AdaQuantFL uses adaptive quantization to reduce the number of bits for model updates while maintaining low error floor.
result AdaQuantFL converges in fewer communicated bits compared to fixed quantization levels, with minimal impact on accuracy.
Algorithm optimizes quantized isotonic regression with log-linear time updates.
problem Optimizing quantized isotonic regression estimations.
method Modified PAVA algorithm for sequential optimization.
result Log-linear time updates for optimal quantized mapping.
Paper compresses neural network weight-updates for image artifacts removal.
problem Efficiently compressing neural network weight-updates for image artifacts removal.
method Fine-tuning a pre-trained artifact removal network on target data with a compression objective that encourages sparse and quantized weight-updates.
result Achieves reconstruction quality comparable to traditional codecs at comparable bitrates.
FedPAQ improves federated learning efficiency by averaging and quantizing updates.
problem Communication bottlenecks and scalability issues in federated learning.
method Periodic averaging, partial device participation, and quantized message-passing.
result FedPAQ achieves near-optimal theoretical guarantees and demonstrates communication-computation tradeoffs.
Q-GADMM reduces communication in decentralized ML by quantizing model updates.
problem Reducing communication in decentralized ML while maintaining accuracy.
method Quantized group ADMM (Q-GADMM) with adaptive quantization.
result Q-GADMM achieves similar accuracy and convergence to non-quantized methods with less communication.
DIANA compresses gradient differences for distributed learning.
problem Learning gradients for convergence in distributed learning.
method Compression of gradient differences in a distributed setting.
result DIANA achieves superior convergence rates compared to existing methods.
SQuARM-SGD improves decentralized SGD efficiency with momentum.
problem Efficient decentralized training of large-scale models over networks.
method Fixed local SGD steps with Nesterov's momentum, sparsified and quantized updates, locally computed triggering criterion.
result Convergence rate matches vanilla SGD, momentum improves test performance.
A new method for robust product Markovian quantization overcomes numerical instabilities.
problem Numerical instabilities in the PMQ algorithm limit its adoption, especially for stochastic volatility models.
method Reformulated PMQ as standard vector quantization, applying accelerated Lloyd's algorithm for robustness.
result The method overcomes numerical instabilities and extends applicability to stochastic volatility models.
Adaptive quantization improves SGD accuracy in data-parallel settings.
problem Fixed gradient quantization schemes lead to suboptimal performance in deep learning.
method Developed adaptive quantization schemes ALQ and AMQ that update compression schemes based on gradient statistics.
result Improved validation accuracy on CIFAR-10 and ImageNet datasets by 2% and 1% respectively.
Low-precision streaming PCA estimates the leading eigenvector with limited precision.
problem Estimating the leading eigenvector in a streaming setting with limited precision.
method Oja's algorithm with linear and nonlinear stochastic quantization.
result A batched version of the quantized variants achieves the lower bound on quantization error up to logarithmic factors.
Quantized Decentralized Gradient Descent (QDGD) solves distributed optimization with quantized communications.
problem Minimizing the sum of smooth and strongly convex functions over a network of distributed agents with quantized communications.
method Proposes QDGD algorithm combining quantized and local information for decentralized gradient descent.
result Achieves vanishing mean solution error under strong convexity and smoothness assumptions.
A new method reduces communication in distributed learning by skipping less informative gradient updates.
problem Efficient communication in distributed machine learning.
method Quantizes and skips less informative gradients to reduce communication overhead.
result Proves linear convergence rate similar to gradient descent with significant communication savings.
Asynchronous decentralized SGD with quantized and local updates converges in gossip model.
problem Scalable distributed machine learning challenges in decentralized optimization.
method Asynchronous decentralized SGD with quantization and local steps in gossip model.
result Decentralized optimization with quantization and local steps converges in asynchronous gossip model.
Quantization-aware training can recover accuracy lost by post-training quantization.
problem Post-training quantization (PTQ) can fail sharply at aggressive bitwidths.
method A unified geometric framework that explains PTQ failure and QAT recovery.
result QAT has a useful bias that steers iterates back into the basin.
Mirror Descent framework for efficient neural network quantization.
problem Efficiently quantize large neural networks for resource-limited devices.
method Introduce Mirror Descent framework for constrained optimization of quantized neural networks.
result Mirror Descent framework achieves state-of-the-art performance in quantized neural networks.
A method for reducing neural network size using look-up tables.
problem Reducing memory and computational footprint of deep neural networks.
method Iteratively learns value dictionaries and assignment matrices for network weights.
result General framework for network reduction that can handle various reduction problems.
Improved nearest neighbor classification for time series data.
problem High storage and computation requirements for large training sets in DTW space.
method Extends LVQ to DTW spaces using asymmetric weighted averaging.
result Asymmetric GLVQ outperforms other methods in nearest neighbor classification.
A new method quantizes neural networks to low-precision without STE, improving accuracy.
problem Quantization of neural networks to low-precision without a complete theoretical understanding.
method Alpha-blending (AB) using stochastic gradient descent (SGD) to quantize weights and gradually increase the coefficient α. result Improves top-1 accuracy by 0.9% on 1-bit BinaryNet, 0.82% on 8-bit MobileNet v1, and 2.93% on 4-bit ResNet_50 v1/2 compared to STE.
This work proposes a complete 8-bit quantization framework for large-scale deep neural networks.
problem Training large-scale deep neural networks with high performance and low memory footprint.
method WAGEUBN framework that quantizes all data paths including weights, activations, gradients, errors, updates, and batch normalization.
result Achieves competitive accuracy on the ImageNet dataset using only 8-bit integers.
QuantEase optimizes LLMs with CD-based quantization, achieving state-of-the-art performance.
problem Efficiently quantize large language models for deployment.
method Layer-wise quantization using CD-based algorithms with matrix and vector operations.
result State-of-the-art performance in perplexity and zero-shot accuracy.
SPARQ-SGD optimizes communication in decentralized SGD with event-triggered and compressed updates.
problem Efficient communication in decentralized stochastic optimization for large-scale models.
method Event-triggered and compressed algorithm with quantized and sparsified model parameters.
result SPARQ-SGD converges with efficiency comparable to uncompressed training, demonstrating significant communication savings.
MQGrad uses reinforcement learning to dynamically adjust gradient quantization bits.
problem Reduction of communication overhead in large-scale machine learning model training.
method Reinforcement learning applied to gradient quantization in parameter server.
result MQGrad accelerates deep neural network learning while maintaining prediction accuracy.
Efficient decentralized learning framework reduces communication costs.
problem Efficiently solve optimization problems in distributed learning networks.
method Censored and Quantized Generalized GADMM (CQ-GGADMM) framework.
result Achieves linear convergence rate under strong convexity assumptions.
BCGD algorithm improves training of quantized neural networks.
problem Training quantized deep neural networks at low bit-widths.
method Introduces coarse gradient descent and blended correction for training.
result BCGD achieves high accuracy in quantized neural networks.
UVeQFed tackles FL model compression over limited channels.
problem Efficiently transmitting trained models over rate-constrained channels.
method Universal vector quantization for FL (UVeQFed).
result UVeQFed minimizes distortion and converges to optimal model.
VQ-GNN scales GNNs to large graphs using vector quantization.
problem Scaling GNNs to large graphs with stable performance and speed.
method VQ-GNN uses vector quantization to preserve all messages passed to a mini-batch of nodes, avoiding the 'neighbor explosion' problem.
result VQ-GNN achieves competitive performance on large-graph node classification and link prediction benchmarks.
Kolmogorov-Arnold Networks enable ultrafast online learning with fixed-point quantization.
problem Efficient online learning for high-frequency systems with strict memory constraints.
method Fixed-point online training on FPGAs exploiting B-spline locality in KANs.
result Kolmogorov-Arnold Networks are more efficient and expressive than MLPs for low-latency tasks.
DFM binarizes feature embeddings for fast, accurate recommendation.
problem Expensive storage and computational cost due to large feature dimensions.
method DFM binarizes real-valued model parameters into binary codes for efficient storage and computation.
result DFM outperforms state-of-the-art binarized recommendation models and shows competitive performance compared to its real-valued version.
FedNew improves federated learning efficiency and privacy.
problem Low communication efficiency and privacy issues in Newton-type methods for federated learning.
method Introduces a two-level framework using ADMM for inverse Hessian-gradient approximation and Newton's method for global model updates, reducing communication overhead.
result FedNew achieves superior communication efficiency and privacy compared to existing methods.
This paper improves federated learning efficiency by auto-tuning secure aggregation parameters.
problem Efficient communication in federated learning with quantized updates.
method Auto-tuning secure aggregation parameters based on random rotation properties.
result Improved communication efficiency in federated learning with secure aggregation.
Efficiently processes dynamic inputs in AI writing assistants with incremental computation.
problem Efficiently updating AI models in real-time with dynamic inputs.
method Incremental computing using vector quantization to filter and reuse intermediate values in neural networks.
result Comparable accuracy with 12.1X fewer operations for processing dynamic inputs.
SOM-VQ tokenizes discrete models with semantic structure and navigable topology.
problem Lack of semantic structure in vector quantized representations limits interpretable human control.
method Combines vector quantization with Self-Organizing Maps to learn discrete codebooks with explicit topology.
result SOM-VQ produces more learnable token sequences and provides an explicit navigable geometry in code space.
Optimal client sampling reduces communication in federated learning.
problem Efficiently aggregate model updates from distributed clients in federated learning.
method Model weights as an Ornstein-Uhlenbeck process to estimate uncommunicated updates; optimal client sampling strategy.
result Significant reduction in communication with competitive or superior performance.
Survey on quantization methods on Kähler manifolds.
problem None explicitly stated; focuses on methods.
method Deformation quantization, geometric quantization, Berezin-Toeplitz quantization, BV quantization.
result New relationships among quantization methods on Kähler manifolds.
A framework for prototype-based classifiers in changing data environments.
problem Learning in non-stationary environments with concept drift.
method Analytical methods from statistical physics applied to LVQ systems.
result Basic LVQ algorithms are suitable for non-stationary environments, but weight decay does not improve performance.
New algorithm reduces communication in distributed learning by sharing compressed beliefs.
problem Efficiently learning from private data in a distributed setting with large hypothesis sets.
method Proposes a belief update rule for distributed cooperative learning with compressed (sparse or quantized) beliefs.
result Beliefs converge almost surely to optimal hypotheses with a linear concentration rate.
Paper explores low-precision SGLD for neural networks, reducing costs without sacrificing performance.
problem Infeasibility of low-precision sampling in large-scale scenarios.
method Developed low-precision SGLD with quantization function and full-precision gradient accumulators.
result Low-precision SGLD achieves comparable performance to full-precision SGLD with only 8 bits.
The paper classifies quantizable functions and explores symmetry in quantization methods.
problem Classifying quantizable functions and understanding symmetry in quantization methods.
method Deformation quantization and geometric quantization methods are compared and classified.
result Formal quantizable functions are of a specific form and relate to Hamiltonian Killing vector fields.
DFedAvgM is a decentralized FedAvg with momentum for privacy and communication efficiency.
problem Efficiently train models with privacy and communication efficiency in federated learning.
method Decentralized Federated Averaging with Momentum (DFedAvgM) on clients connected by an undirected graph, using stochastic gradient descent with momentum and quantization.
result DFedAvgM converges under trivial assumptions and can be improved with the PŁ property, numerically verified.
This paper introduces a differentiable, scalable quantization method for neural networks.
problem Previous quantization methods lacked differentiability and scalability.
method The approach is differentiable and scalable, using bit-shifting and logarithmic quantization.
result The method achieves comparable accuracy to state-of-the-art approaches with less training time and lower inference cost.
This paper uses deep reinforcement learning to compress CNN models, reducing size and maintaining accuracy.
problem Reducing model size for efficient deployment on limited hardware resources.
method Two-stage compression pipeline: pruning and quantization using deep reinforcement learning.
result Significant reduction in model size with minimal loss in accuracy.
StatQAT optimizes quantization for deep networks, reducing computational cost and memory usage.
problem Optimal quantization parameters selection for deep neural networks with diverse data distributions.
method Statistical error analysis framework for uniform and floating-point quantization, iterative and analytic quantizers designed for arbitrary and Gaussian-like distributions.
result Improved accuracy and stability in training low-precision neural networks.
Smart Quantization adapts binary and ternary quantization for neural networks.
problem Resource constraints in deploying neural networks on devices with limited resources.
method Adaptive combination of binary and ternary quantization with a regularization function.
result Adapts quantization depth during training to maintain high model accuracy.
This study optimizes quantized neural networks by considering model architecture and quantization types.
problem Optimizing quantized neural networks for low-power, high-throughput applications.
method Holistic approach including training methods and quantization-friendly architecture design.
result Deeper models are more sensitive to activation quantization, while wider models improve resilience to both weight and activation quantization.
Two novel network quantization approaches improve deep neural network compression.
problem Efficiently compress deep neural networks for practical usage.
method Single-level network quantization (SLQ) and Multi-level network quantization (MLQ).
result Both SLQ and MLQ achieve impressive results in compressing state-of-the-art neural networks.
RATQ is a new quantizer for optimizing noisy gradients in machine learning.
problem Optimizing noisy gradients in stochastic optimization.
method RATQ uses Hadamard transform and adaptive uniform quantization, and achieves near-optimal performance.
result RATQ nearly achieves information theoretic lower bounds for optimization accuracy.
Extends ONNX for quantized neural networks with new formats and operators.
problem Handling arbitrary-precision quantization in neural networks.
method Introduces new formats and operators in ONNX to represent quantized neural networks.
result Enabled representation of uniform quantization in neural networks.