Bayesian Bits unifies quantization and pruning through gradient optimization.
problem Joint mixed precision quantization and pruning for efficient neural networks.
method Gradient-based optimization with a novel bit width decomposition and learnable stochastic gates.
result Bayesian Bits achieves better accuracy vs. efficiency trade-off compared to static bit width networks.
Paper proposes a learning-based sparse Bayesian method for accurate off-grid DOA estimation.
problem One-bit off-grid direction of arrival (DOA) estimation in a single snapshot scenario.
method Formulated off-grid DOA estimation model, used Sparse Bayesian framework, proposed Learning-based Sparse Bayesian approach.
result Improved computational efficiency and accuracy in off-grid DOA estimation.
Study improves fractional posterior for 1-bit matrix completion.
problem Estimating a binary matrix from observed entries.
method Fractional posterior approach with low-rank factorization and spectral scaled Student priors.
result Concentration results for fractional posterior, demonstrating effectiveness in matrix recovery.
BITS for GAPS uses Bayesian methods to improve surrogate model accuracy in complex systems.
problem Improving surrogate model accuracy in complex physical systems with uncertainty.
method Bayesian Information-Theoretic Sampling for hierarchical Gaussian Process Surrogates.
result Increased expected information gain and predictive accuracy by targeting high-uncertainty regions.
This letter proposes a low-computational Bayesian algorithm for noisy sparse recovery in the context of one bit compressed sensing with sensing matrix perturbation. The proposed algorithm which is called BHT-MLE comprises a sparse support detector and an amplitude estimator. The support detector utilizes Bayesian hypot…
Bayesian approach fixes overconfidence in ReLU networks, even slightly.
problem Overconfidence in ReLU networks far from training data.
method Theoretical analysis of approximate Gaussian distributions on ReLU weights, and empirical validation.
result Even a simplistic Bayesian approximation fixes overconfidence issues.
The property of perfectness plays an important role in the theory of Bayesian networks. First, the existence of perfect distributions for arbitrary sets of variables and directed acyclic graphs implies that various methods for reading independence from the structure of the graph (e.g., Pearl, 1988; Lauritzen, Dawid, La…
We prove that the binary classifiers of bit strings generated by random wide deep neural networks with ReLU activation function are biased towards simple functions. The simplicity is captured by the following two properties. For any given input bit string, the average Hamming distance of the closest input bit string wi…
Transformers approximate Bayesian posteriors but not exactly.
problem Bayesian accounts of in-context learning face challenges due to task-preserving order changes in transformers.
method Showed that excess prequential code length is exactly cumulative predictive KL, decomposing expected regret into order-averaged predictor and order-averaging gain.
result Transformers approximate Bayesian posteriors but not exactly, priced by log loss.
OOD-trained Bayesian neural networks perform similarly to frequentist methods in uncertainty quantification.
problem Bayesian neural networks struggle in out-of-distribution (OOD) detection tasks.
method Incorporated out-of-distribution data into Bayesian inference through four different methods.
result OOD-trained Bayesian neural networks are competitive with frequentist baselines.
New techniques improve 16-bit training accuracy without 32-bit units.
problem Training deep learning models with only 16-bit floating-point units.
method Studied BFloat16 units and applied stochastic rounding and Kahan summation techniques.
result Up to 7% absolute validation accuracy gain in 16-bit-FPU training.
MsIGN tackles high-dimensional Bayesian inference using multiscale structure.
problem High-dimensional Bayesian inference challenges due to the curse of dimensionality.
method MsIGN generates samples from coarse to fine scale, minimizing Jeffreys divergence.
result MsIGN outperforms previous approaches in posterior approximation and mode capture.
Low-bit training framework reduces energy consumption in CNNs.
problem Reducing energy consumption in convolutional neural networks.
method Low-bit training framework using MLS tensor format with dynamic quantization.
result Achieves superior trade-off between accuracy and bit-width.
Due to challenging applications such as collaborative filtering, the matrix completion problem has been widely studied in the past few years. Different approaches rely on different structure assumptions on the matrix in hand. Here, we focus on the completion of a (possibly) low-rank matrix with binary entries, the so-c…
Paper improves DNN accelerator robustness against bit errors with energy savings.
problem Bit errors in quantized DNN weights reduce energy efficiency.
method Combines robust fixed-point quantization, weight clipping, and random bit error training.
result Significantly improves robustness against random bit errors with high energy savings.
The bits-back argument suggests that latent variable models can be turned into lossless compression schemes. Translating the bits-back argument into efficient and practical lossless compression schemes for general latent variable models, however, is still an open problem. Bits-Back with Asymmetric Numeral Systems (BB-A…
Paper proposes a QUBO formulation that reduces binary variables in Bayesian network learning.
problem Reducing the number of binary variables in QUBO formulations for Bayesian network learning.
method Proposes a new QUBO formulation that minimizes binary variables.
result Significantly reduces the number of binary variables required for Bayesian network structure learning.
Bit-slice sparsity improves ReRAM-based DNN acceleration.
problem Limited ADC power and area constraints in ReRAM-based DNN accelerators.
method Proposed bit-slice L1 algorithm to induce sparsity during training.
result 2x sparsity improvement compared to previous methods.
The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, un…
Majority bit estimation in noisy random recursive DAGs.
problem Estimating the majority bit in a noisy random recursive DAG.
method Majority rule among nodes, with bit flipping and noisy channel.
result Identification of the threshold for p at which majority rule yields errors. Paper proposes a CNN-based method for estimating intra frame bits and quality.
problem Efficient video delivery and bit allocation in video coding.
method Deep learning approach using CNNs trained on original frames and encoded distortions.
result Accurate estimation of intra frame bits and quality for better bit allocation.
Improves matrix multiplication throughput for asymmetric bit-width operands.
problem Matrix multiplications between asymmetric bit-width operands, especially 8- and 4-bit, are not efficiently handled by existing SIMD instructions.
method Proposes a new SIMD matrix multiplication instruction that uses mixed precision on inputs (8- and 4-bit) and accumulates into 16-bit output, improving throughput.
result Offers 2x improvement in throughput compared to existing symmetric-operand-size instructions, with negligible overflow.
Paper proposes a hybrid model-based and data-driven approach for one-bit compressive autoencoding.
problem Designing efficient one-bit compressive autoencoding models for complex systems.
method Hybrid model-based and data-driven methodology for one-bit sparse signal recovery.
result Significant improvement in one-bit compressive autoencoding compared to state-of-the-art algorithms.
Paper proposes a hybrid model-based and data-driven method for one-bit compressive variational autoencoding.
problem Designing efficient one-bit compressive sensing systems.
method Hybrid model-based and data-driven approach for one-bit compressive variational autoencoding.
result Significant improvement in one-bit compressive sensing compared to state-of-the-art methods.
Transformers mimic Bayesian reasoning in controlled settings, revealing geometric mechanisms.
problem Verifying if transformers perform Bayesian reasoning rigorously in natural data.
method Constructing Bayesian wind tunnels with known posteriors and proving memorization impossibility.
result Transformers achieve 10−3-10−4 bit accuracy in Bayesian posteriors, while MLPs fail. New protocols show 1-bit mean estimation can be order-optimal without interaction.
problem Can 1-bit mean estimation be optimal without interaction?
method Adaptive and non-adaptive threshold and interval queries, with one adaptive transition.
result Arbitrary non-adaptive quantizers can match the adaptive rate, suggesting interaction is not necessary.
Topological data analysis classifies encrypted bits with success.
problem Classifying encrypted data with traditional machine learning methods.
method Persistent homology for generating topological features, machine learning pipeline.
result Successfully classifies encrypted data, outperforming classical models.
We consider the problem of estimating the mean of a symmetric log-concave distribution under the constraint that only a single bit per sample from this distribution is available to the estimator. We study the mean squared error as a function of the sample size (and hence the number of bits). We consider three settings:…
Paper offers robust recovery for 1-bit sensing with partial Gaussian circulant matrices.
problem Accurately recovering vectors from 1-bit measurements using structured matrices.
method Correlation-based optimization with randomly signed partial Gaussian circulant matrices and generative models.
result Recovery guarantees match those for i.i.d. Gaussian matrices but with faster computation.
One-bit feedback suffices for a bandit problem's optimal strategy.
problem Optimal strategy for multi-armed bandit problem with limited feedback.
method Coding and decoding schemes for one-bit feedback to mimic full-reward feedback.
result Regret ratio approaches 1 with one-bit feedback.
New algorithm tackles batched stochastic linear bandits with 1-bit communication constraints.
problem Stochastic linear bandits with 1-bit communication constraints.
method Phased-elimination algorithms based on G-optimal designs and 1-bit mean estimation.
result Achieves near-optimal regret bounds for broad scaling regimes.
Implementing large-scale deep neural networks with high computational complexity on low-cost IoT devices may inevitably be constrained by limited computation resource, making the devices hard to respond in real-time. This disjunction makes the state-of-art deep learning algorithms, i.e. CNN (Convolutional Neural Networ…
Moniqua improves SGD convergence with quantized communication.
problem Efficiently communicating in decentralized SGD with limited bandwidth.
method Modulo quantized communication in decentralized SGD.
result Moniqua converges at the same rate as full-precision communication with less bits.
Two methods reduce BN and DNN complexity, balancing size and accuracy.
problem Balancing model size and prediction accuracy in Bayesian networks and deep neural networks.
method Quantization-aware training and tree-augmented naive Bayes structure learning extension.
result Pareto optimal models found for small-scale scenarios.
Study 1-bit compressive sensing with generative models, improving recovery accuracy.
problem Accurately recover sparse vectors from binary measurements with generative models.
method Analyzes noiseless and noisy 1-bit measurements with i.i.d.~Gaussian and Lipschitz continuous generative priors, proving sample complexity bounds and stability properties.
result Proves sample complexity bounds and stability properties for 1-bit compressive sensing with generative models.
It is of fundamental importance to find algorithms obtaining optimal performance for learning of statistical models in distributed and communication limited systems. Aiming at characterizing the optimal strategies, we consider learning of Gaussian Processes (GPs) in distributed systems as a pivotal example. We first ad…
ADD embeds a 48-bit message into images, achieving high accuracy and speed.
problem Embedding high-fidelity messages into images to detect authenticity and source.
method Two-stage process: linear combination and addition of watermark to image, followed by decoding.
result ADD achieves 100% decoding accuracy for 48-bit watermarking, with minimal performance drop under various distortions.
This letter proposes a dictionary learning algorithm for blind one bit compressed sensing. In the blind one bit compressed sensing framework, the original signal to be reconstructed from one bit linear random measurements is sparse in an unknown domain. In this context, the multiplication of measurement matrix $\Ab$ an…
FleXOR trains fractional quantization for neural networks, improving accuracy and size.
problem Quantization limits to integer bits restricts compression and accuracy.
method Encryption algorithm with XOR gates for fractional bits during inference.
result FleXOR achieves high accuracy with fractional sub-1-bit weights.
Generalizes bits back coding for time-series models with latent Markov structures.
problem Efficiently compressing time-series data with latent Markov structures.
method Extends bits back coding to time-series models with latent Markov structures, including HMMs and LGSSMs.
result Effective for small scale models, promising for larger scale settings like video compression.
Reduced precision computation for deep neural networks is one of the key areas addressing the widening compute gap driven by an exponential growth in model size. In recent years, deep learning training has largely migrated to 16-bit precision, with significant gains in performance and energy efficiency. However, attemp…
Paper studies distributed learning with limited communication bits, achieving optimal error exponents.
problem Distributed hypothesis testing with constant communication bits.
method Geometric approach in distribution spaces, encoding empirical distributions to transmission bits.
result Optimal achievable error exponents and coding schemes for various communication constraints.
WrapNet optimizes inference for low-resolution neural networks by using 8-bit additions.
problem Reducing multiplication complexity in low-resolution neural networks.
method Adapting neural networks to use low-resolution (8-bit) additions in accumulators, with a cyclic activation layer and overflow penalty regularizer.
result Achieves comparable classification accuracy to 32-bit counterparts using low-resolution additions.
The method of random projections has become a standard tool for machine learning, data mining, and search with massive data at Web scale. The effective use of random projections requires efficient coding schemes for quantizing (real-valued) projected data into integers. In this paper, we focus on a simple 2-bit coding …
A new algorithm for compressing latent representations in deep models.
problem Compressing continuous latent representations in deep models.
method Separates model design and training from quantization; uses adaptive quantization based on posterior uncertainty.
result Image compression with the proposed algorithm outperforms JPEG over a wide range of bit rates.
Kernel Quantization improves CNN compression without sacrificing performance.
problem Efficiently compressing CNN models without significant performance loss.
method Quantizes convolution kernels as the unit, learning a codebook for low-bit indexes.
result Significant compression ratio achieved with minimal accuracy loss.
Minwise hashing is the standard technique in the context of search and databases for efficiently estimating set (e.g., high-dimensional 0/1 vector) similarities. Recently, b-bit minwise hashing was proposed which significantly improves upon the original minwise hashing in practice by storing only the lowest b bits of e…
Paper proposes a 1-bit quantization scheme for high-dimensional statistical estimation.
problem High-dimensional statistical estimation with limited data.
method Uniformly dithered 1-bit quantization for sparse covariance matrix estimation, sparse linear regression, and matrix completion.
result Near minimax rates in sub-Gaussian regime and improved rates in heavy-tailed regime.