Channel pruning and weight binarization improve keyword spotting accuracy.
problem Improving accuracy of keyword spotting in neural networks.
method Group-wise splitting method using group Lasso penalty for channel sparsity, combined with 1-bit weight precision.
result Achieved over 50% channel sparsity with minimal accuracy loss.
Paper verifies properties of binarized neural networks using SAT solvers.
problem Verifying properties of deep neural networks.
method Exact Boolean encoding of binarized neural networks, SAT solvers, counterexample-guided search.
result Demonstrates scalability to medium-size deep neural networks for robustness verification.
Develops a binarized GNN for more efficient graph embeddings.
problem Real-valued GNN parameters limit efficiency and scalability.
method Integrates binarization into GNN-based graph embedding approaches.
result Binarized graph neural network (BGN) achieves state-of-the-art performance with significant efficiency gains.
A deep neural network improves document binarization accuracy.
problem Binarizing digital documents with historical degradations.
method Combines FCN with primal-dual network for end-to-end training.
result Achieves state-of-the-art binarization on four out of seven datasets.
Binarized CNNs improve GPU inference efficiency on resource-constrained devices.
problem Efficient inference on resource-constrained devices for image classification.
method Binarization of weights and computations in CNNs, implemented on GPUs.
result 7.4X speedup with 4.4% accuracy loss on embedded GPU platforms.
FPGAs accelerate DNNs with binarized weights, improving power and speed.
problem Power and speed limitations of GPU-accelerated DNNs.
method Binarized neural networks on FPGAs using OpenCL.
result Near state-of-the-art performance with >16x power savings.
TentacleNet improves binarized CNNs, reducing accuracy loss and memory usage.
problem Excessive accuracy loss in binarized CNNs.
method Parallelization inspired by ensemble learning theory, end-to-end trainable compact topology.
result Significant memory savings compared to state-of-the-art binary ensemble methods.
End-to-end auto-encoder for neural image compression with variable bit rates.
problem Efficient image compression with variable bit rates.
method Block-based auto-encoder system with novel contributions.
result Incremental performance improvement of each contribution.
New insights into BNN optimization redefine latent weights as inertia.
problem Optimizing Binarized Neural Networks (BNNs) with latent weights.
method Interpreted latent weights as inertia and introduced Binary Optimizer (Bop).
result Demonstrated improved performance of Bop on CIFAR-10 and ImageNet.
This work tackles catastrophic forgetting in neural networks by mimicking brain's metaplasticity.
problem Catastrophic forgetting in neural networks, where new tasks erase previously learned ones.
method Interpreting binarized neural networks as metaplastic systems, adjusting their training technique.
result Training technique reduces catastrophic forgetting without needing previously presented data.
New combinatorial attacks improve BNN robustness.
problem Improving BNN robustness against adversarial attacks.
method Mixed Integer Linear Programming (MILP) and decomposition-based algorithm (IProp).
result IProp outperforms FGSM in BNN robustness.
Binary representation is desirable for its memory efficiency, computation speed and robustness. In this paper, we propose adjustable bounded rectifiers to learn binary representations for deep neural networks. While hard constraining representations across layers to be binary makes training unreasonably difficult, we s…
01 loss neural network improves adversarial robustness on CIFAR10.
problem Improving machine learning models' resistance to adversarial attacks.
method Proposes a single hidden layer 01 loss neural network trained with stochastic coordinate descent.
result 01 loss network has the largest adversarial distortion compared to other models.
Pruned neural networks learn digital circuits with 99% weight reduction.
problem Efficiently train deep neural networks with minimal weights.
method Constrained binarized networks to zero or one weights.
result Pruned networks achieve similar performance to standard networks with 99% weight reduction.
Compact binary CNNs improve landmark localization on limited resources.
problem Designing lightweight architectures for landmark localization with limited resources.
method Binarization of neural networks, hierarchical, parallel, multi-scale residual architecture.
result Significant performance improvement over standard architectures.
Study on function sensitivity in random DNNs using large deviation theory.
problem Understanding function sensitivity in finite-size deep neural networks.
method Large deviation theory and path integral analysis applied to random DNNs with ReLU and sign activations.
result Random DNNs with ReLU activations are more robust to parameter perturbations.
High error rates improve neural network performance and reduce power consumption.
problem Training neural networks with high Bit Error Rates (BERs).
method Trained three Binarized Convolutional Neural Network architectures on various datasets with high BERs.
result High BERs do not significantly degrade test accuracy, enabling more efficient hardware.
QubitHD improves HD computing ML efficiency without sacrificing accuracy.
problem Trade-off between energy efficiency and classification accuracy in HD computing-based ML.
method Stochastically binarizes HD-based algorithms while maintaining comparable classification accuracies.
result 65% improvement in energy efficiency and 95% improvement in training time on FPGA.
Progressive stochastic binarization improves deep network inference efficiency.
problem Efficient inference of deep networks with reduced memory and computational resources.
method A progressive stochastic binarization scheme for deep networks that uses small integers and fixed shifts.
result Matches the accuracy of previous binarized approaches and reduces inference costs by up to 33%.
A new adaptive binarization technique using fuzzy integrals improves image quality.
problem Improving image thresholding quality.
method FLAT (Fuzzy Local Adaptive Thresholding) based on fuzzy integrals.
result The proposed FLAT method produces better image quality than traditional algorithms and neural networks.
A method for efficient approximate inference on discrete distributions.
problem Applying SVGD to discrete distributions.
method Transforming discrete distributions to piecewise continuous distributions for SVGD application.
result Outperforms traditional algorithms and ensemble methods on discrete graphical models.
This paper introduces GLT for better input data representation in BNN and proposes a compact topology with block pruning.
problem Improving input data representation for Binary Neural Networks (BNN).
method Generic Learned Thermometer (GLT) for encoding, block pruning and Knowledge Distillation for compact topology.
result Significant accuracy gains and lightweight fully-binarized models with limited accuracy degradation.
This work improves DNN weight quantization with ADMM, achieving lossless binarization and reduced search space.
problem Improving DNN model compression and accuracy with low bit quantization.
method Extending ADMM framework for DNN weight quantization with progressive multi-step approach.
result Achieved lossless and fully binarized DNNs with reduced accuracy loss.
LUTNet optimizes FPGA for neural network inference, achieving high efficiency.
problem Redundancy in neural networks leads to inefficient hardware implementations.
method Exploits LUTs' flexibility for efficient neural network inference on FPGAs.
result Significant area savings with comparable accuracy compared to state-of-the-art binarized neural networks.
Paper proposes efficient BNN inference techniques on FPGA.
problem Redundancy in BNN inference leading to high computation and data access costs.
method Analyzed image similarity and BNN kernel weights to exploit redundancy. Proposed two types of fast and energy-efficient architectures.
result 80% reduction in computation and 40% in buffer access, achieving 17% power reduction.
New approach quantizes neural nets with guaranteed convergence to loss-optimal states.
problem Optimizing deep neural net compression by quantizing weights.
method Model compression as constrained optimization framework, alternating learning and quantization.
result Guaranteed convergence to local optimum of loss for quantized nets, achieving high compression rates.
Gaudy images help train deep neural networks with less data.
problem Training deep neural networks with limited real data from visual cortex neurons.
method Used high-contrast binarized natural images (gaudy images) to train DNNs.
result Reduced training data needed for accurate DNN predictions of visual cortex neuron responses.
BCGD algorithm improves training of quantized neural networks.
problem Training quantized deep neural networks at low bit-widths.
method Introduces coarse gradient descent and blended correction for training.
result BCGD achieves high accuracy in quantized neural networks.
New model CRS combines transparency and high performance for classification.
problem Need models with transparent structure and high classification performance.
method Concept Rule Sets (CRS) with Multilayer Logical Perceptron (MLLP) and Random Binarization (RB).
result CRS outperforms state-of-the-art approaches and has low complexity.
Improves bit error tolerance in RRAM-based BNNs without overfitting.
problem Bit errors in RRAM-based BNNs reduce accuracy and overfit to training error rates.
method Proposes straight-through gradient approximation and a novel regularizer.
result Improves BNNs' robustness to bit errors without overfitting.
This paper justifies the use of straight-through estimator in training quantized neural nets.
problem Minimizing loss in quantized neural nets with vanishing gradients.
method Introduced straight-through estimator (STE) and proved its effectiveness in two-linear-layer network with binarized ReLU activations.
result Proved that the coarse gradient derived from STE is a descent direction for minimizing population loss.
NAF combines neural networks with autoregressive models for better density estimation.
problem Improving density estimation and speech synthesis speed.
method Generalizes autoregressive models using neural networks for invertible transformations.
result NAF is a universal approximator for continuous probability distributions and outperforms IAF.
Quantized neural networks can improve robustness against adversarial attacks.
problem Adversarial attacks on neural networks with low-precision weights and activations.
method Proposed a third benefit of very low-precision neural networks: improved robustness against some adversarial attacks. Focused on weights and activations quantized to ±1, and conducted black-box and white-box experiments.
result Non-scaled binary neural networks can reduce the impact of iterative attacks, but do not artificially mask gradients.
This research bridges binary and spiking neural networks for efficient on-chip AI.
problem Reducing compute requirements in machine learning frameworks.
method Training Spiking Neural Networks in extreme quantization regime and utilizing standard training techniques for conversion.
result Training Spiking Neural Networks in extreme quantization regime achieves near full precision accuracies.
New approach uses Boolean circuits to optimize neural networks.
problem Improving efficiency of neural network implementations on hardware accelerators.
method Formalized neural networks as Boolean circuits, showing binarized networks are functionally complete.
result Binarized neural networks are functionally complete, suggesting new possibilities for neural network accelerators.
TAPAS speeds up encrypted machine learning predictions.
problem Data privacy and computation time in encrypted machine learning.
method Combining binarization, sparsification, and homomorphic encryption techniques.
result Significantly reduces computation time for encrypted predictions.
LUTNet optimizes FPGA neural network accelerators by leveraging LUTs for inference, achieving significant area savings.
problem Redundancy in deep neural networks and inefficient use of FPGA resources.
method End-to-end hardware-software framework using LUTs to implement any K-input Boolean operation for inference.
result Significant area savings and comparable accuracy compared to state-of-the-art binarized neural networks.
Paper models entropy-based impact of soft errors on neural network inference.
problem Estimating impact of radiation-induced faults on neural network inference.
method Entropy-based statistical models for SEU and MBU across layers.
result Accurate models to evaluate error-resiliency of neural network topologies.
New framework reduces LLM complexity by directly finetuning in Boolean domain.
problem Reducing the complexity of large language models (LLMs) while maintaining performance.
method Proposes a novel framework using multi-kernel Boolean parameters for direct finetuning in the Boolean domain.
result Significantly reduces complexity during both finetuning and inference, outperforming recent techniques.
CNN detects phase transitions in Potts models without prior knowledge.
problem Detecting phase transitions in q-state Potts models using deep learning. method Trained a deep CNN on Ising model spin configurations and temperatures, then tested on Potts model images.
result Deep CNN accurately detects phase transitions in Potts models, including high- and low-temperature regions.
DFM binarizes feature embeddings for fast, accurate recommendation.
problem Expensive storage and computational cost due to large feature dimensions.
method DFM binarizes real-valued model parameters into binary codes for efficient storage and computation.
result DFM outperforms state-of-the-art binarized recommendation models and shows competitive performance compared to its real-valued version.
Sparse binary compression reduces communication costs in distributed deep learning.
problem Limited communication bandwidth in distributed deep learning.
method Combines gradient sparsification, binarization, and optimal weight update encoding.
result Reduces upstream communication by more than four orders of magnitude.
MeliusNet improves binary neural networks to match MobileNet-v1 accuracy.
problem Achieving high accuracy with binary neural networks on mobile devices.
method Alternating DenseBlocks and ImprovementBlocks to increase feature capacity and quality.
result MeliusNet matches MobileNet-v1 accuracy on ImageNet, improving binary network performance.
BinaryDuo improves BNNs by coupling binary activations, outperforming state-of-the-art models.
problem Gradient mismatch in BNNs due to binarizing activations.
method Using gradient of smoothed loss function to estimate gradient mismatch, proposing BinaryDuo scheme with coupled ternary activations.
result BinaryDuo outperforms state-of-the-art BNNs on various benchmarks.
New memory allocation scheme improves image generation performance.
problem Improving episodic and semantic memory representation in neural networks.
method Developed a hierarchical latent variable model with differentiable, locally block allocated latent memory.
result Improved conditional likelihood values on various datasets.
CodNN uses error-correcting codes to make neural networks more resilient to noise.
problem Neural networks are sensitive to noise, especially in critical applications.
method Construct robust neural networks by coding data or internal layers with error-correcting codes.
result Parity codes can guarantee robustness for a wide range of neural networks, including binarized networks.
This work tackles large action spaces in RL by binarizing actions.
problem Large action spaces in reinforcement learning cause significant challenges.
method Sequentializing actions and binarizing the action space.
result Binarizing the action space can significantly improve RL algorithms and reduce state space size.
BIL allows binary input data in CNNs, improving performance on multimodal datasets.
problem Efficient execution of CNNs on edge devices with reduced bit width.
method BIL concept that learns bit-specific binary weights for binary input data.
result BIL outperforms full precision weights by 1.92% on multimodal datasets.