Study binary activated deep neural networks using PAC-Bayesian theory.
problem Generalization bounds for binary activated deep neural networks.
method Developed an end-to-end framework and provided PAC-Bayesian generalization bounds.
result Nonvacuous PAC-Bayesian generalization bounds for binary activated deep neural networks.
G-Net constructs binary neural networks with high accuracy using randomized binary embeddings.
problem Creating high-accuracy binary neural networks with theoretical guarantees.
method Proposes a novel floating-point G-Net family with randomized binary embeddings and theoretical accuracy guarantees.
result Empirically, G-Net achieves almost 30% higher accuracy on CIFAR-10 compared to prior HDC models.
Improves training of binary neural networks for mobile devices.
problem Training accurate binary neural networks for mobile devices.
method Systematic evaluation of network architectures and hyperparameters.
result Increased accuracy by increasing the number of connections in the network.
This research bridges binary and spiking neural networks for efficient on-chip AI.
problem Reducing compute requirements in machine learning frameworks.
method Training Spiking Neural Networks in extreme quantization regime and utilizing standard training techniques for conversion.
result Training Spiking Neural Networks in extreme quantization regime achieves near full precision accuracies.
Binary neural networks trained from scratch achieve state-of-the-art results.
problem Training accurate binary neural networks from scratch is challenging.
method No prior knowledge and simple training strategy used.
result Achieved state-of-the-art results on standard benchmark datasets.
BENN improves binary neural networks by ensemble methods, boosting accuracy without sacrificing efficiency.
problem Inefficiency and accuracy degradation in binary neural networks.
method Proposes Binary Ensemble Neural Network (BENN) using ensemble techniques.
result BENN outperforms state-of-the-art binary networks and full-precision networks.
A new neural network using chi-square test for binary classification.
problem Improving binary classification accuracy.
method Backpropagation neural network with chi-square test redefined cost and error functions.
result Significantly improved classification accuracy compared to related approaches.
Stochastic binary synapses in neural networks lead to robust solutions.
problem Training low-precision neural networks with stochastic weights.
method Gradient descent on a probability distribution of binary synapses.
result Binary solutions are robust and generalize well, while typical solutions are isolated.
Probabilistic BLRNet uses binary weights and activations for efficient neural networks.
problem Efficiently training and deploying deep neural networks with limited memory and compute.
method Probabilistic training method for binary weights and activations, introducing stochastic operations.
result BLRNet achieves performance comparable to full-precision networks while using fewer bits.
New algorithms improve binary neural network configurations.
problem Training binary neural networks efficiently.
method Stochastic message passing algorithms (BP and SP) for discrete inference.
result Stochastic BP and SP find better BNN configurations.
Study shows proper initialisation of binary weights is crucial for deep neural networks.
problem Training stochastic binary neural networks with continuous surrogates is challenging.
method Developed new surrogates based on Markov chain theory and mean field analysis.
result Critical initialisations are necessary for training deep networks with binary weights.
Proposes JBNN for multi-label classification with improved efficiency and performance.
problem Multi-label classification with dependencies and heavy computational load.
method Joint Binary Neural Network (JBNN) that synchronously performs multiple binary classifications and captures label relations via joint binary cross entropy (JBCE) loss.
result Significantly better performance and computational efficiency compared to state-of-the-art methods.
Bayesian learning rule trains binary neural networks effectively.
problem Training binary neural networks is challenging due to discrete optimization.
method Proposes the Bayesian learning rule to estimate Bernoulli weights.
result Obtains state-of-the-art performance and enables uncertainty estimation.
Single neurons can perform as well as dense networks in binary and multi-class recognition tasks.
problem Designing efficient neural networks for recognition tasks.
method Investigated the use of single or multiple neurons in neural networks for binary and multi-class recognition tasks.
result Sparse networks can be as efficient as dense networks in both binary and multi-class tasks.
Paper benchmarks quantum neural networks against classical ones for binary classification tasks.
problem Comparing quantum neural networks with classical ones for binary classification.
method Evaluated with two toy examples, focusing on model complexity and training data size.
result EQNN and QNN outperform ENN and DNN for smaller parameter sets and training data samples.
MeliusNet improves binary neural networks to match MobileNet-v1 accuracy.
problem Achieving high accuracy with binary neural networks on mobile devices.
method Alternating DenseBlocks and ImprovementBlocks to increase feature capacity and quality.
result MeliusNet matches MobileNet-v1 accuracy on ImageNet, improving binary network performance.
A spiking neural network model for probabilistic inference of binary Markov random fields.
problem Implementing probabilistic inference in spiking neural networks.
method Designing a spiking recurrent neural network and proving its equivalence to mean-field inference of binary Markov random fields.
result The spiking neural network model can implement inference of arbitrary binary Markov random fields.
Paper proposes HTAF for stable training of binary neural networks.
problem Challenges in training binary neural networks with gradient-based optimization.
method HTAF is a smooth approximation to the Heaviside function that enables stable training.
result HTAF enables stable training of various binary neural networks with gradient-based optimization.
Binary encoding enables neural networks to extrapolate periodic functions.
problem Extrapolating periodic functions without prior knowledge of their form.
method Normalized Base-2 Encoding (NB2E) for continuous numerical values.
result MLPs using NB2E can successfully extrapolate diverse periodic signals.
Develops a binarized GNN for more efficient graph embeddings.
problem Real-valued GNN parameters limit efficiency and scalability.
method Integrates binarization into GNN-based graph embedding approaches.
result Binarized graph neural network (BGN) achieves state-of-the-art performance with significant efficiency gains.
Reintroduces straight-through estimators for binary neural networks.
problem Training neural networks with binary weights and activations is challenging due to gradient issues and discrete weight optimization.
method Derives ST methods as estimators in the SBN model, analyzes properties and estimation accuracy, explains latent weights and mirror descent method.
result Reintroduces ST methods as sound approximations and provides clearer application and improvements.
MCLNN improves audio classification with binary masks.
problem Improving audio classification accuracy.
method Binary mask applied to CLNN for feature preservation and combination exploration.
result Competitive recognition accuracies on GTZAN and ISMIR2004 datasets.
BinaryDuo improves BNNs by coupling binary activations, outperforming state-of-the-art models.
problem Gradient mismatch in BNNs due to binarizing activations.
method Using gradient of smoothed loss function to estimate gradient mismatch, proposing BinaryDuo scheme with coupled ternary activations.
result BinaryDuo outperforms state-of-the-art BNNs on various benchmarks.
This paper analyzes the training dynamics of binary neural networks using information bottleneck.
problem Training binary neural networks is challenging due to discontinuity in activation functions.
method The approach uses the Information Bottleneck principle to analyze BNN training dynamics.
result Training dynamics of BNNs are different from DNNs, with both phases occurring simultaneously.
Deep learning predicts phase segregation in binary mixtures.
problem Predicting phase segregation in binary mixtures.
method Conditional generative convolutional neural networks.
result Deep learning model accurately predicts phase segregation up to 98%.
New method trains neural networks with binary or ternary weights efficiently.
problem Training high-precision, real-time neural networks on limited hardware.
method Local reparameterization trick modification for discrete weights.
result State-of-the-art results on binary and ternary models on various benchmarks.
Paper proposes a new binary quantization method for faster DNN inference.
problem Accelerating deep neural network inference on resource-limited devices.
method Quantized Compressed Sensing (QCS) for binary quantization.
result The proposed method preserves benefits of standard methods while reducing quantization error.
Neural networks for binary classification have zero training error at all local minima under certain conditions.
problem Understanding the loss surface of neural networks for binary classification.
method Analyzing single-layered neural networks with smooth hinge loss function, providing conditions for zero training error at all local minima.
result Zero training error at all local minima is achieved under specific conditions (strict convexity of neurons and smooth hinge loss).
BatchNorm helps BNNs avoid exploding gradients during training.
problem Difficulty in training Binary Neural Networks (BNNs).
method Theoretical study and numerical experiments on the role of BatchNorm in BNNs.
result BatchNorm is crucial for avoiding exploding gradients in BNNs.
Non-negative constraints improve neural network defenses.
problem Effective defenses against adversarial attacks in neural networks.
method Non-negative weight constraints applied to binary and non-binary classification problems.
result Non-negative constraints can improve resistance to adversarial attacks, especially in binary classification with asymmetric costs.
Smart Quantization adapts binary and ternary quantization for neural networks.
problem Resource constraints in deploying neural networks on devices with limited resources.
method Adaptive combination of binary and ternary quantization with a regularization function.
result Adapts quantization depth during training to maintain high model accuracy.
A new method sparsifies neural networks using stochastic binary optimization.
problem Sparsifying neural networks to reduce computational cost and improve efficiency.
method Stochastic binary optimization with the Augment-Reinforce-Merge (ARM) estimator.
result ARM enables efficient network sparsification with comparable accuracy to baseline methods.
Foothill regularizer improves DNNs for edge computing by reducing accuracy gap.
problem Improving generalization error of DNNs for edge devices.
method Introducing foothill function as a quasiconvex regularizer.
result Foothill reduces accuracy gap between BNNs and full-precision DNNs.
New method for estimating gradients in stochastic binary networks.
problem Challenges in training neural networks with binary activations and weights.
method Combines sampling and analytic approximation steps to estimate gradients accurately.
result Significantly reduced variance at the cost of small bias, leading to practical tradeoffs.
Unified approach optimizes neural network training for various metrics.
problem Training and evaluation of neural network binary classifiers often use different metrics.
method Combines differentiable approximation and probabilistic soft sets.
result Effective in optimizing for metrics like F1-Score across various domains.
Study on double descent behavior in two-layer neural networks for binary classification.
problem Understanding the double descent phenomenon in model test error.
method Two-layer neural network with ReLU activation for binary classification. Quantified model size by sample-to-dimension ratio. Empirical risk minimization using Convex Gaussian Min Max Theorem.
result Observed and investigated the double descent behavior of model test error.
Quantized neural networks can improve robustness against adversarial attacks.
problem Adversarial attacks on neural networks with low-precision weights and activations.
method Proposed a third benefit of very low-precision neural networks: improved robustness against some adversarial attacks. Focused on weights and activations quantized to ±1, and conducted black-box and white-box experiments.
result Non-scaled binary neural networks can reduce the impact of iterative attacks, but do not artificially mask gradients.
StrNN uses neural network structures to learn conditional independencies.
problem Learning conditional independencies in neural networks.
method Designing masks for neural networks based on binary matrix factorization.
result StrNN improves density estimation and causal inference.
QNNs can't distinguish binary signals from their negations, revealing a new symmetry.
problem Understanding the behavior of QNNs in binary pattern classification.
method Presented and analyzed a new form of invariance (negational symmetry) in QNNs.
result QNNs cannot differentiate a quantum binary signal and its negational counterpart in binary classification tasks.
Modified AUC improves CNN training by considering model confidence.
problem Improving binary classifier performance metrics.
method Proposes a modified AUC metric that incorporates model confidence into BCE loss for CNN training.
result Demonstrates improved performance on three datasets: MNIST, prostate MRI, and brain MRI.
Neural network improves normality testing accuracy.
problem Testing for normality with small samples.
method Binary classification using a feedforward neural network.
result Neural network outperforms standard tests on small samples.
Model neural plasticity as binary optimization to dynamically activate or deactivate network units.
problem Dynamic learning and adaptability of neural networks.
method Model neural plasticity as an L0-norm regularized binary optimization problem, where units can be activated or deactivated based on a cost-benefit tradeoff. result Demonstrates that a single parameter k can modulate learning dynamics, unifying network sparsification and expansion. Feed-forward nets fail to learn equality relations, but adding DR units helps.
problem Feed-forward neural networks struggle to learn equality relations reliably.
method Introduced differential rectifier (DR) units to create an inductive bias.
result DR units enable feed-forward nets to learn equality relations reliably.
ARM policy gradient reduces variance for binary actions.
problem High variance in policy gradients for binary actions.
method Augment-Reinforce-Merge (ARM) policy gradient estimator.
result ARM estimator achieves significant variance reduction and faster convergence.
Binary representation is desirable for its memory efficiency, computation speed and robustness. In this paper, we propose adjustable bounded rectifiers to learn binary representations for deep neural networks. While hard constraining representations across layers to be binary makes training unreasonably difficult, we s…
Alternative to convolutions using decision trees for neural networks.
problem Replacing complex convolutions with simpler decision-based layers.
method Binary decisions as indices to conditional distributions, trained using backpropagation.
result Performance similar to conventional neural networks, with runtime improvements.
Paper introduces ABC-Net, a binary CNN that maintains high accuracy with reduced memory and power.
problem Accuracy loss in binary CNNs during inference.
method Approximating full-precision weights with binary bases and using multiple binary activations.
result ABC-Net achieves comparable prediction accuracy to full-precision CNNs, even on challenging datasets.
ECNN combines binary classifiers to protect neural networks from adversarial attacks.
problem Vulnerability of neural networks to adversarial inputs.
method Designs an error-correcting code matrix to maximize row and column distances, training end-to-end.
result ECNN effectively defends against adversarial attacks with good accuracy on normal examples.