To make deep neural networks feasible in resource-constrained environments (such as mobile devices), it is beneficial to quantize models by using low-precision weights. One common technique for quantizing neural networks is the straight-through gradient method, which enables back-propagation through the quantization ma…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
37 results for “straight-through”
Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networksstat.ML
Reintroduces straight-through estimators for binary neural networks.
problem Training neural networks with binary weights and activations is challenging due to gradient issues and discrete weight optimization.
method Derives ST methods as estimators in the SBN model, analyzes properties and estimation accuracy, explains latent weights and mirror descent method.
result Reintroduces ST methods as sound approximations and provides clearer application and improvements.
Paper improves Gumbel-Softmax estimator variance reduction.
problem Challenges in gradient estimation for models with discrete latent variables.
method Rao-Blackwellization applied to straight-through Gumbel-Softmax estimator.
result Reduces mean squared error and variance of Gumbel-Softmax estimator.
New insights into quantized neural networks reveal learning dynamics and generalization errors.
problem Understanding the impact of quantization hyperparameters on learning dynamics in high-dimensional models.
method Theoretical analysis and fixed-point analysis of STE dynamics in quantized models.
result STE training in quantized models converges to a plateau followed by a sharp drop in generalization error, influenced by quantization range.
The Straight-Through (ST) estimator is a widely used technique for back-propagating gradients through discrete random variables. However, this effective method lacks theoretical justification. In this paper, we show that ST can be interpreted as the simulation of the projected Wasserstein gradient flow (pWGF). Based on…
The Straight-Through Estimator (STE) is widely used for back-propagating gradients through the quantization function, but the STE technique lacks a complete theoretical understanding. We propose an alternative methodology called alpha-blending (AB), which quantizes neural networks to low-precision using stochastic grad…
We extend quantization-aware training to extreme model compression.
problem Maximizing model accuracy with minimal model size.
method Quantize a random subset of weights during training, allowing unbiased gradients through other weights.
result Established new state-of-the-art compromises between accuracy and model size.
Training activation quantized neural networks involves minimizing a piecewise constant function whose gradient vanishes almost everywhere, which is undesirable for the standard back-propagation or chain rule. An empirical way around this issue is to use a straight-through estimator (STE) (Bengio et al., 2013) in the ba…
These notes aim to shed light on the recently proposed structured projected intermediate gradient optimization technique (SPIGOT, Peng et al., 2018). SPIGOT is a variant of the straight-through estimator (Bengio et al., 2013) which bypasses gradients of the argmax function by back-propagating a surrogate "gradient." We…
A quadruple crossing is a crossing in a projection of a knot or link that has four strands of the knot passing straight through it. A quadruple crossing projection is a projection such that all of the crossings are quadruple crossings. In a previous paper, it was proved that every knot and link has a quadruple crossing…
Paper proposes DAG-DB for learning discrete DAGs via backpropagation.
problem Learning Directed Acyclic Graphs (DAGs) from data.
method DAG-DB uses Discrete Backpropagation with I-MLE and Straight-Through Estimation.
result DAG-DB learns DAGs effectively using probabilistic sampling and backpropagation.
We propose the BinaryGAN, a novel generative adversarial network (GAN) that uses binary neurons at the output layer of the generator. We employ the sigmoid-adjusted straight-through estimators to estimate the gradients for the binary neurons and train the whole network by end-to-end backpropogation. The proposed model …
A new method for optimizing models with categorical variables using diffusion.
problem Optimizing models with categorical variables, especially in discrete distributions.
method Introducing ReDGE, a diffusion-based soft reparameterization method for categorical distributions.
result ReDGE consistently matches or outperforms existing gradient-based methods in experiments.
A triple crossing is a crossing in a projection of a knot or link that has three strands of the knot passing straight through it. A triple crossing projection is a projection such that all of the crossings are triple crossings. We prove that every knot and link has a triple crossing projection and then investigate c_3(…
We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-deman…
New estimators reduce variance in training variational autoencoders with discrete latent variables.
problem Training variational autoencoders with discrete latent variables requires efficient gradient estimation.
method Introduce ReinMax-Rao and ReinMax-CV estimators using Rao-Blackwellisation and control variates.
result Demonstrate superior performance on training variational autoencoders with discrete latent spaces.
Sorting input objects is an important step in many machine learning pipelines. However, the sorting operator is non-differentiable with respect to its inputs, which prohibits end-to-end gradient-based optimization. In this work, we propose NeuralSort, a general-purpose continuous relaxation of the output of the sorting…
Efficient deep neural network (DNN) inference on mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precision networks achieve better performance than networks with homogeneous bitwidth for the same size constraint. Since choosing the optimal bitwi…
Bayesian learning rule trains binary neural networks effectively.
problem Training binary neural networks is challenging due to discrete optimization.
method Proposes the Bayesian learning rule to estimate Bernoulli weights.
result Obtains state-of-the-art performance and enables uncertainty estimation.
Two methods reduce BN and DNN complexity, balancing size and accuracy.
problem Balancing model size and prediction accuracy in Bayesian networks and deep neural networks.
method Quantization-aware training and tree-augmented naive Bayes structure learning extension.
result Pareto optimal models found for small-scale scenarios.
New method for estimating gradients in stochastic binary networks.
problem Challenges in training neural networks with binary activations and weights.
method Combines sampling and analytic approximation steps to estimate gradients accurately.
result Significantly reduced variance at the cost of small bias, leading to practical tradeoffs.
AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networksstat.ML
AskewSGD optimizes quantized neural networks with interval-constrained optimization.
problem Training deep neural networks with quantized weights.
method Formulates QNN training as smoothed interval-constrained optimization, proposes AskewSGD for solving each subproblem.
result AskewSGD avoids projections and allows infeasible iterates, performs better than state-of-the-art methods.
A new method for discrete data normalizing flows using latent transformations.
problem Challenges in parameterizing bijective transformations for discrete data.
method Predict a distribution over latent transformations to make the marginal likelihood differentiable.
result Discrete-data normalizing flows can be trained using gradient-based learning with unbiased score function estimation.
We present a theoretical and experimental investigation of the quantization problem for artificial neural networks. We provide a mathematical definition of quantized neural networks and analyze their approximation capabilities, showing in particular that any Lipschitz-continuous map defined on a hypercube can be unifor…
Differentiable model compression adds noise to parameters during training.
problem Model compression for efficient deployment.
method Adding pseudo quantization noise to model parameters during training.
result Compresses model by more than 8x on ImageNet with 0.3% accuracy loss.
Introduced recently, an n-crossing is a singular point in a projection of a link at which n strands cross such that each strand travels straight through the crossing. We introduce the notion of an übercrossing projection, a knot projection with a single n-crossing. Such a projection is necessarily composed of a collect…
Quantizing large Neural Networks (NN) while maintaining the performance is highly desirable for resource-limited devices due to reduced memory and time complexity. It is usually formulated as a constrained optimization problem and optimized via a modified version of gradient descent. In this work, by interpreting the c…
In this paper we study image captioning as a conditional GAN training, proposing both a context-aware LSTM captioner and co-attentive discriminator, which enforces semantic alignment between images and captions. We empirically focus on the viability of two training methods: Self-critical Sequence Training (SCST) and Gu…
A faster method for optimizing DNA and protein sequences using machine learning.
problem Designing DNA and protein sequences with improved function.
method Activation maximization with a straight-through approximation and adaptive entropy variable.
result Fast SeqProp achieves up to 100-fold faster convergence and improved fitness optima.
Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow simple non-differentiable functions at intermediary layers of deep neural networks. We do so by training with a differentiable…
NM-PPG optimizes adaptive feature acquisition in POMDPs for better predictions.
problem Optimizing adaptive feature acquisition in prediction problems with costly features.
method Non-myopic pathwise policy gradients (NM-PPG) with continuous relaxation and straight-through rollout.
result NM-PPG outperforms state-of-the-art AFA methods on synthetic and real-world datasets.
IDF++ improves integer discrete flows for lossless compression.
problem Theoretical limitations of integer discrete flows for lossless compression.
method Investigated and improved integer discrete flows, addressing gradient bias and architecture modifications.
result Different architecture modifications improve integer discrete flows for lossless compression.
Polyhedra can be Rupert if they can be slightly rotated to fit inside each other.
problem Proving that every convex polyhedron can be Rupert.
method Developed two sufficient conditions for a polyhedron to be Rupert, involving small rotations.
result Found two conditions for a polyhedron to be Rupert, simplifying the proof process.
For discrete data, the likelihood can be rewritten exactly and parametrized into if has enough capacity to put no probability mass on any for which , where is a deterministic discrete function. The log of the first factor giv…
Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basincs.LG
Quantization-aware training can recover accuracy lost by post-training quantization.
problem Post-training quantization (PTQ) can fail sharply at aggressive bitwidths.
method A unified geometric framework that explains PTQ failure and QAT recovery.
result QAT has a useful bias that steers iterates back into the basin.
Improves bit error tolerance in RRAM-based BNNs without overfitting.
problem Bit errors in RRAM-based BNNs reduce accuracy and overfit to training error rates.
method Proposes straight-through gradient approximation and a novel regularizer.
result Improves BNNs' robustness to bit errors without overfitting.
Graph neural network optimizes energy-efficient precoding for massive MIMO systems.
problem Energy bottleneck in massive MIMO systems due to high DAC complexity and power consumption.
method Proposes a graph neural network to directly output precoded quantized vectors from channel matrix and transmit symbols.
result Significant increase in achievable sum rate with reduced DAC power consumption.