This paper introduces a quantization-based regularizer for autoencoders to improve latent representations.
problem Autoencoders can overfit and collapse, leading to poor latent representations.
method The authors combine VQ-VAE and denoising methods to introduce a bottleneck Bayesian estimator that soft quantizes latent codes.
result The method results in better latent representations for supervised and clustering tasks.
The Simanca metric on a blown-up plane has regular quantization properties.
problem Characterizing the quantization properties of the Simanca metric.
method Using the blow-up structure and Tian-Yau-Zelditch expansion, proving regular quantization and vanishing coefficients.
result All coefficients in the Tian-Yau-Zelditch expansion for the Simanca metric vanish, and a dense subset admits Berezin quantization.
Abstract proposes a new categorical approach to quantization of Poisson algebras.
problem Quantization of Poisson algebras.
method Defining quantization categories as subcategories of R-module categories with classical limits.
result Categories of strict deformation quantization, prequantization, and matrix regularization are equivalent, while Poisson enveloping algebra is not.
We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-deman…
PAR provides a flexible framework for quantization in optimization problems.
problem Challenges in optimization problems over discrete or quantized variables.
method Piecewise-affine regularization (PAR) for modeling and computational optimization.
result PAR-regularized loss functions exhibit high quantization at critical points in the overparameterized regime.
A novel approach to quantizing neural networks using periodic functions as regularizers.
problem Quantization of neural network parameters to reduce memory usage and computational cost.
method Using periodic functions (sine, cosine, hat) as regularizers during training to push weights into discrete points.
result Quantized models achieve the same accuracy as original models on CIFAR-10 and ImageNet datasets.
Study on convergence rates for optimal transport with regularization.
problem Convergence analysis of divergence-regularized optimal transport.
method Novel methodology using quantization and martingale couplings.
result Sharp rates for various divergences and transport costs.
For a possibly singular subset of a regular Poisson manifold we construct a deformation quantization of its algebra of Whitney functions. We then extend the construction of a deformation quantization to the case where the underlying set is a subset of a not necessarily regular Poisson manifold which can be written as t…
SinReQ adds sinusoidal regularization to improve quantized neural networks.
problem Accuracy loss in quantized deep neural networks.
method SinReQ adds a periodic term to the objective function of quantized training algorithms.
result SinReQ closes the accuracy gap by 32.4% and 27.5% compared to DoReFa and WRPN respectively.
Foothill regularizer improves DNNs for edge computing by reducing accuracy gap.
problem Improving generalization error of DNNs for edge devices.
method Introducing foothill function as a quasiconvex regularizer.
result Foothill reduces accuracy gap between BNNs and full-precision DNNs.
WaveQ uses sinusoidal regularization to optimize deep quantization for neural networks, improving both efficiency and accuracy.
problem Deep quantization reduces bitwidth but can lead to significant accuracy loss due to inter-layer dependencies.
method WaveQ employs sinusoidal regularization to learn multiple quantization parameters during gradient-based training, balancing compute efficiency and accuracy.
result WaveQ achieves accuracy preservation and efficiency gains across various deep networks, outperforming state-of-the-art techniques.
A practical algorithm improves approximate OT distances using quantization.
problem Substantial computational burden in computing OT distances for large samples.
method Introduces a quantization step to estimate OT distances between measures.
result The quantization step improves the performance of approximate solvers for entropy-regularized transport.
Quantizes latent space to improve disentanglement in models.
problem Learning disentangled representations from unlabeled data.
method Quantizes latent space into discrete code vectors with a learnable scalar codebook and applies high weight decay regularization.
result Quantized-latent autoencoder (QLAE) outperforms prior work in disentanglement without sacrificing data reconstruction.
The paper studies convergence rates of Tsallis entropic regularization in optimal transport.
problem Optimal transport with regularization.
method Γ-convergence and quantization/shadow arguments.
result Derives convergence rate of Tsallis entropic regularization.
Smart Quantization adapts binary and ternary quantization for neural networks.
problem Resource constraints in deploying neural networks on devices with limited resources.
method Adaptive combination of binary and ternary quantization with a regularization function.
result Adapts quantization depth during training to maintain high model accuracy.
Study proves convergence of quantized geodesics to Mabuchi geodesics.
problem Convergence of quantized geodesics to Mabuchi geodesics in short time.
method Real-analytic initial data and convergence proof.
result Proves convergence of quantized Bergman geodesics to Mabuchi geodesics.
Simplifies neural network compression with Gaussian priors and L1 regularization.
problem Neural network overfitting and scalability issues.
method Adds Gaussian priors and L1 regularization to the optimization problem for quantization and pruning.
result Achieves results competitive with state-of-the-art methods using simple modifications.
In this article we develop tools to compute the Geometric Quantization of a symplectic manifold with respect to a regular Lagrangian foliation via sheaf cohomology and obtain important new applications in the case of real polarizations. The starting point is the definition of representation spaces due to Kostant. Besid…
We propose in this contribution a method for l one regularization in prototype based relevance learning vector quantization (LVQ) for sparse relevance profiles. Sparse relevance profiles in hyperspectral data analysis fade down those spectral bands which are not necessary for classification. In particular, we consider …
ProxQuant improves quantized neural networks using proximal operators.
problem Making neural networks work on devices with limited resources.
method Formulates quantized network training as a regularized learning problem and optimizes it via the prox-gradient method.
result ProxQuant outperforms state-of-the-art results on binary quantization and is on par with state-of-the-art on multi-bit quantization.
One-bit quantization and sparsification improve multiclass classification with strong regularization.
problem Overfitting mislabeled data in multiclass classification.
method Linear regression with regularization and one-bit quantization/sparsification.
result Sparse and one-bit solutions perform almost as well as the optimal solution with f(⋅)=∥⋅∥22. This work proposes a novel approach to learn quantizers from data, improving similarity search performance.
problem Learning optimal quantizers for multi-dimensional data distributions.
method Train a neural net to form a fixed parameter-free quantizer, using uniformity in a spherical latent space as a proxy objective.
result The proposed method outperforms most learned quantization methods and is competitive with state-of-the-art approaches.
This work improves DNN weight quantization with ADMM, achieving lossless binarization and reduced search space.
problem Improving DNN model compression and accuracy with low bit quantization.
method Extending ADMM framework for DNN weight quantization with progressive multi-step approach.
result Achieved lossless and fully binarized DNNs with reduced accuracy loss.
This study improves Ernie's accuracy for INT8 inference by modifying its training process.
problem Improving the accuracy of pre-trained models like Ernie for low precision inference.
method Integrates a regularizer into the training process to make it more robust to quantization.
result Increased INT8 accuracy for Ernie models.
New algorithm improves quantized neural networks for image classification.
problem Improving approximation capabilities of quantized neural networks.
method Proposed a novel gradient-based training algorithm for quantized neural networks.
result State-of-the-art performance on image classification benchmarks.
Paper studies quantized LRMR with random dithering for correlated tasks.
problem Estimating coefficient matrix in quantized multivariate regression.
method Uniform quantization with random dithering, constrained and regularized Lasso estimators.
result Achieves minimax optimal rate with dithering, slightly worsens quantization effect.
CoDeQ simplifies joint model compression by integrating pruning and quantization.
problem Joint pruning and quantization methods are complex and require additional procedures.
method CoDeQ uses a dead-zone quantizer to directly induce sparsity and learn quantization parameters.
result CoDeQ achieves high sparsity and low-precision accuracy with minimal bit operations.
We develop a new approach, based on quantization methods, to study higher symmetries of invariant differential operators. We focus here on conformally invariant powers of the Laplacian over a conformally flat manifold and recover results of Eastwood, Leistner, Gover and Šilhan. In particular, conformally equivariant qu…
This paper develops efficient bounds on the Wasserstein metric for discrete measures.
problem Computing the exact Wasserstein metric is computationally expensive.
method Formulates and solves a Kantorovich problem on a coarse grid using quantized measures and cost matrices, followed by upscaling and correction.
result Achieves a 10x-100x speedup while maintaining low approximation error.
The paper shows how to reduce quantization errors in neural networks.
problem Reducing degradation caused by quantization in neural networks.
method Factorizing network weights to inversely scale output channels without changing function.
result Proper factorizations significantly decrease quantization errors.
Paper proposes method to recover quantized data with missing info.
problem Recovering quantized data with missing information.
method Regularized convex cost function with Bi-factorization and Augmented Lagrangian Method.
result The method finds global minimizer of the cost function.
S2D selectively decays large singular values to improve quantization of neural activations.
problem Large activation outliers in transformer models cause accuracy drops during quantization.
method Selective Spectral Decay (S2D) that surgically regularizes only the largest singular values. result Significantly reduces activation outliers and produces well-conditioned representations.
Many mathematical models of physical phenomena that have been proposed in recent years require more general spaces than manifolds. When taking into account the symmetry group of the model, we get a reduced model on the (singular) orbit space of the symmetry group action. We investigate quantization of singular spaces o…
BitPruning learns optimal bitlengths for neural networks to balance accuracy and efficiency.
problem Finding the minimum bitlength for neural network accuracy.
method A training method that penalizes large bitlengths and minimizes other quantifiable criteria.
result The method learns efficient representations while maintaining accuracy, reducing bitlengths by 3.76 bits on average per layer.
A new multi-scale vector quantization method for unsupervised data.
problem Efficiently reconstructing unsupervised data with minimal distortion.
method Reconstruction trees, inspired by decision trees, explore data in a multi-scale fashion.
result Analysis of expected distortion under fixed unknown distribution, with asymptotic and finite sample results.
New algorithms improve scalar quantization by optimizing sparse least squares.
problem Improving efficiency and accuracy of scalar quantization for neural networks.
method Sparse least square optimization, iterative and clustering-based methods.
result Proposed algorithms outperform existing methods, especially in bit-width reduction scenarios.
A symplectic integration of a Poisson manifold (M,Λ) is a symplectic groupoid (Γ,η) which realizes the given Poisson manifold, i.e. such that the space of units Γ0 with the induced Poisson structure Λ0 is isomorphic to (M,Λ). This notion was introduced by A. Weinstein in order to quantize Poisson manifolds …
New method for identifying systems with quantized data using Gaussian process and stable spline kernel.
problem Identifying linear systems with quantized output data.
method Bayesian framework with Markov Chain Monte Carlo and Gibbs sampler.
result Effectiveness of the proposed scheme compared to state-of-the-art methods.
A new method compresses deep neural networks by predicting and quantizing weights between layers.
problem Resource constraints in deep neural networks.
method Inter-Layer Weight Prediction (ILWP) and quantization based on Smoothly Varying Weight Hypothesis (SVWH).
result The method achieves higher weight compression rates at the same accuracy level.
A new method for quantized matrix completion using Huber loss.
problem Quantized Matrix Completion with robustness to quantization errors.
method Rank minimization with Huber loss regularization, Smooth Rank Approximation.
result Our method achieves better accuracy and efficiency than state-of-the-art methods.
BCGD algorithm improves training of quantized neural networks.
problem Training quantized deep neural networks at low bit-widths.
method Introduces coarse gradient descent and blended correction for training.
result BCGD achieves high accuracy in quantized neural networks.
In this paper we introduce a novel method for linear system identification with quantized output data. We model the impulse response as a zero-mean Gaussian process whose covariance (kernel) is given by the recently proposed stable spline kernel, which encodes information on regularity and exponential stability. This s…
QDSB accelerates Schrödinger bridge learning with quantized approximations.
problem Learning generative models from unpaired samples.
method Quantized diffusion Schrödinger bridges (QDSB) using anchor-quantized distributions and cell-wise sampling.
result QDSB achieves sample quality similar to existing methods but with significantly less computational time.
The paper analyzes reducing model complexity for better generalization.
problem Improving model generalization with reduced complexity networks.
method Upper bound on Vapnik-Chervonenkis dimension, pruning, quantization, and a novel loss function.
result Quantization and the proposed loss function lead to sparser models with comparable accuracy.
DIANA compresses gradient differences for distributed learning.
problem Learning gradients for convergence in distributed learning.
method Compression of gradient differences in a distributed setting.
result DIANA achieves superior convergence rates compared to existing methods.
This paper develops a geometric framework for Wilson surfaces in higher gauge theory.
problem Quantum field theory of Wilson surfaces in higher gauge theory.
method Higher coadjoint orbit theory and derived geometric framework.
result Identification of derived coadjoint orbits and their quantization.
The paper quantizes vortex moduli spaces on compact Kahler surfaces using determinant bundles.
problem Quantizing vortex moduli spaces on compact Kahler surfaces.
method Developed holomorphic determinant bundles and geometric quantization for vortex moduli spaces.
result Quantized vortex moduli spaces on compact Kahler surfaces using determinant bundles.
This paper proposes a new weight representation scheme for efficient model compression and performance enhancement.
problem Challenges in achieving performance enhancement on devices due to irregular sparse matrix representations.
method Fine-grained and unstructured pruning method combined with structured weight encryption.
result Achieved high compression ratios and performance on various deep learning models.