BIL allows binary input data in CNNs, improving performance on multimodal datasets.
problem Efficient execution of CNNs on edge devices with reduced bit width.
method BIL concept that learns bit-specific binary weights for binary input data.
result BIL outperforms full precision weights by 1.92% on multimodal datasets.
Binary autoencoder with sparse hidden layer preserves information and zero reconstruction error.
problem Preserving information and zero reconstruction error in binary neural networks.
method Binary autoencoder with random binary weights, sparse hidden layer, and varying neuron thresholds.
result Zero reconstruction error for any input with a large hidden layer and varying neuron thresholds.
The study explores the compressive power of Boolean threshold autoencoders, finding that seven layers are necessary but three are not.
problem Understanding the compressive limits of Boolean threshold autoencoders.
method Investigation into the minimum number of layers and nodes required for autoencoders to accurately transform binary vectors.
result There exists a seven-layer autoencoder with a logarithmic middle layer size for any set of distinct vectors, but not a three-layer one.
DFS dynamically decides bitwidths for layers to balance accuracy and efficiency.
problem Balancing model accuracy and inference speed for deep networks.
method Dynamic Fractional Skipping (DFS) framework that assigns bitwidths to layers for input-adaptive inference.
result DFS achieves superior tradeoff between computational cost and model accuracy.
Dual neural network architecture improves accuracy and interpretability.
problem Improving neural network interpretability and accuracy.
method Stacked recurrent and feedforward layers, binary activation function.
result Binary activation leads to simpler, more interpretable models with higher accuracy.
Binary encoding enables neural networks to extrapolate periodic functions.
problem Extrapolating periodic functions without prior knowledge of their form.
method Normalized Base-2 Encoding (NB2E) for continuous numerical values.
result MLPs using NB2E can successfully extrapolate diverse periodic signals.
The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.
problem Establishing sample complexity lower bounds for neural network parameters and excess risk.
method Using information-theoretic tools, the paper proves lower bounds by constructing a generative network.
result Proves information-theoretic lower bounds for exact parameter recovery and positive excess risk.
We formulate learning of a binary autoencoder as a biconvex optimization problem which learns from the pairwise correlations between encoded and decoded bits. Among all possible algorithms that use this information, ours finds the autoencoder that reconstructs its inputs with worst-case optimal loss. The optimal decode…
BSF algorithm reduces neural network size and selects features efficiently.
problem Neural network size and feature selection optimization.
method Binary Stochastic Filtering (BSF) layer that penalizes information, stochastically passes or drops features.
result Multifold decrease in neural network size and optimal feature selection.
We propose Very Simple Classifier (VSC) a novel method designed to incorporate the concepts of subsampling and locality in the definition of features to be used as the input of a perceptron. The rationale is that locality theoretically guarantees a bound on the generalization error. Each feature in VSC is a max-margin …
A neural network with a single hidden layer can't represent certain multivariable functions.
problem Representing certain multivariable functions with a neural network having only one hidden layer.
method Developed a continuum version of a one-hidden-layer neural network with ReLU activation, and proved constraints on its parameters and second derivative.
result Existence of a smooth binary function that cannot be precisely represented by any such neural network.
Deep learning has become a powerful and popular tool for a variety of machine learning tasks. However, it is challenging to understand the mechanism of deep learning from a theoretical perspective. In this work, we propose a random active path model to study collective properties of deep neural networks with binary syn…
Unsupervised pre-training preserves multiple views from input distribution.
problem Learning from multiple views of supervision in unsupervised settings.
method Investigates unsupervised pre-training under information theory, focusing on disentanglement and multiple views.
result Unsupervised pre-training helps conserve multiple views from input distribution, improving model performance.
Paper proposes using blockchain for trustable machine learning with streaming layer and synthetic data.
problem Machine learning results are not fully trusted due to mutable data and difficulty in automation.
method Blockchain technology for immutable data storage and smart contracts for automation. Server, streaming, and smart contract implementations.
result A compact binary model format for streaming layer and synthetic data generation for limited training data.
Alternative to convolutions using decision trees for neural networks.
problem Replacing complex convolutions with simpler decision-based layers.
method Binary decisions as indices to conditional distributions, trained using backpropagation.
result Performance similar to conventional neural networks, with runtime improvements.
Detects adversarial inputs in deep learning models without modifying the main network.
problem Vulnerability of deep learning models to adversarial inputs.
method Augment main network with observer networks that classify inputs as clean or adversarial.
result 99.5% detection accuracy on MNIST and 97.5% on CIFAR-10 datasets.
Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.
problem Training two-layer networks for binary classification.
method Gradient descent with logistic loss applied to two-layer networks.
result Gradient descent can drive training loss to zero under certain conditions.
We establish upper bounds for the minimal number of hidden units for which a binary stochastic feedforward network with sigmoid activation probabilities and a single hidden layer is a universal approximator of Markov kernels. We show that each possible probabilistic assignment of the states of n output units, given t…
DIET-SNN optimizes SNNs for faster, lower-energy image classification.
problem High inference latency and inefficient input encoding in SNNs.
method End-to-end backpropagation to optimize membrane leak and firing threshold.
result Achieves top-1 accuracy of 69% on ImageNet with 5 timesteps and 12x less compute energy.
Proposes a new method to improve multiclass probability calibration.
problem Uncalibrated class probabilities in multiclass classifiers leading to over-confidence.
method Dirichlet calibration method applicable to any model class, derived from Dirichlet distributions.
result Improved probabilistic predictions across various datasets and classifiers.
Binary representation is desirable for its memory efficiency, computation speed and robustness. In this paper, we propose adjustable bounded rectifiers to learn binary representations for deep neural networks. While hard constraining representations across layers to be binary makes training unreasonably difficult, we s…
We have applied a little-known data transformation to subsets of the Surveillance, Epidemiology, and End Results (SEER) publically available data of the National Cancer Institute (NCI) to make it suitable input to standard machine learning classifiers. This transformation properly treats the right-censored data in the …
New conditions ensure deep neural networks can approximate any function on non-Euclidean spaces.
problem Understanding how to modify neural network architectures to approximate functions on non-Euclidean spaces.
method Developed conditions for feature and readout maps that preserve universal approximation capabilities.
result Modified architectures can deterministically approximate any classifier on non-Euclidean spaces.
BinConv improves time series forecasting by preserving ordinal information in a classification framework.
problem Lack of ordinal information in existing classification-based time series forecasting methods.
method Cumulative Binary Encoding (CBE) and BinConv architecture.
result BinConv achieves superior performance in time series forecasting compared to existing methods.
Simplified Bayesian neural networks reduce model complexity and improve interpretability.
problem Over-parameterization and interpretability issues in deep learning models.
method Input-skip Latent Binary Bayesian Neural Networks (LBBNNs) that allow covariates to skip layers or be excluded.
result Significant reduction in model complexity (over 99%) with minimal loss in accuracy and uncertainty.
Gradient descent biases neural networks to use an average of features, leading to non-robustness.
problem Non-robustness in neural networks due to feature averaging.
method Theoretical analysis and experiments on binary classification tasks.
result Gradient descent trains networks to rely on an average of features, making them vulnerable to adversarial attacks.
Study on double descent behavior in two-layer neural networks for binary classification.
problem Understanding the double descent phenomenon in model test error.
method Two-layer neural network with ReLU activation for binary classification. Quantified model size by sample-to-dimension ratio. Empirical risk minimization using Convex Gaussian Min Max Theorem.
result Observed and investigated the double descent behavior of model test error.
Stochastic binary hidden units in a multi-layer perceptron (MLP) network give at least three potential benefits when compared to deterministic MLP networks. (1) They allow to learn one-to-many type of mappings. (2) They can be used in structured prediction problems, where modeling the internal structure of the output i…
The muti-layer information bottleneck (IB) problem, where information is propagated (or successively refined) from layer to layer, is considered. Based on information forwarded by the preceding layer, each stage of the network is required to preserve a certain level of relevance with regards to a specific hidden variab…
Enhances quantum machine learning models using Fock states.
problem Data-embedding bottleneck in quantum machine learning.
method Photonic-based bosonic data-encoding scheme in Fock space.
result Controlled expressive power via photon number.
New findings on neural networks with non-negative weights and low training error.
problem Does a low training error imply a small outer norm for two-layer neural networks?
method Covering number argument and fat-shattering dimension analysis.
result For non-negative output weights, low training error guarantees a well-controlled outer norm.
New algorithm learns two-layer neural networks under symmetric input distributions.
problem Learning two-layer neural networks with symmetric inputs.
method Method-of-moments framework, spectral algorithms.
result Guaranteed to recover parameters of ground-truth network under certain conditions.
The study develops a supervised and unsupervised WTA model for sparse binary projections.
problem Sparse binary projections in high-dimensional spaces.
method Supervised and unsupervised WTA models with efficient algorithms.
result Significantly improved results in similarity search tasks.
New method approximates neural network training for robustness.
problem Training robust neural networks with adversarial input perturbations.
method Stochastic convex optimization approach to adversarial training.
result Method achieves better adversarial robustness and performance.
Binary testing for softmax models requires many samples, similar to leverage score models.
problem Binary hypothesis testing for softmax models and leverage score models.
method Analyzing sample complexity and drawing analogies between models.
result Sample complexity is asymptotically \(O(ε^{-2})\), where \(ε\) is the distance between model parameters.
Gradient descent amplifies random features in neural networks to useful ones.
problem Generalization in neural networks trained on corrupted data.
method Characterization of feature-learning process in two-layer ReLU networks trained by gradient descent.
result Gradient descent amplifies random features to useful ones, achieving near optimal generalization error.
New method improves accuracy of quantized neural networks.
problem Accuracy drop in quantized neural networks, especially MobileNet family.
method Weight equalizing shift scaler, binary shifting to recover output range.
result Top-1 accuracy improved from 0.1% to 69.78% ~ 70.96% in MobileNets.
A new method for imbalanced binary classification without resampling.
problem Imbalanced binary classification tasks where majority class under-representation leads to information loss.
method Layered learning approach with two stages: clustering and classification.
result The method outperforms state-of-the-art methods in 100 benchmark data sets.
Batch normalization biases linear models towards uniform margins, improving performance in binary classification.
problem Understanding the implicit bias of batch normalization in linear models and neural networks.
method Analyzing gradient descent convergence on linear models and two-layer CNNs with batch normalization.
result Gradient descent with batch normalization in linear models converges to a uniform margin classifier with an exponential convergence rate.
This research analyzes how input and output layers affect deep neural networks' resistance to adversarial attacks.
problem The vulnerability of deep neural networks to adversarial inputs, especially non-gradient based attacks.
method Analysis of three different fully connected dense network classes with manipulated input and output layers.
result Manipulating input and output layers can significantly enhance a deep neural network's robustness against adversarial attacks.
DCFS predicts stock indices using deep learning and fuzzy systems.
problem Predicting stock indices with high-dimensional inputs.
method Bottom-up layer-by-layer design of DCFS, parameter sharing.
result DCFS models outperform weak estimators and real-world data.
To backpropagate the gradients through stochastic binary layers, we propose the augment-REINFORCE-merge (ARM) estimator that is unbiased, exhibits low variance, and has low computational complexity. Exploiting variable augmentation, REINFORCE, and reparameterization, the ARM estimator achieves adaptive variance reducti…
Active learning improves EDFA model accuracy with binary features.
problem Lack of labeled training data for EDFA devices.
method Active learning strategy for binary features using sparse linear models.
result Improved prediction and accelerated query generation.
Our paper explains deep neural collapse in multiple layers.
problem Understanding deep neural collapse in multi-layered neural networks.
method Generalized unconstrained features model for deep networks.
result Deep unconstrained features model exhibits deep neural collapse.
A simple 2-layer linear network outperforms neural networks in learning sparse targets.
problem Learning sparse targets from a sparse input with gradient descent.
method A 2-layer linear network with fully connected input layer and sparse targets.
result The 2-layer linear network achieves a lower expected square loss than neural networks.
FedUA trains UA models privately without raw data.
problem Training UA models requires raw user data, compromising privacy.
method Federated learning framework for privacy-preserving UA model training.
result FedUA reliably rejects unseen user data at high true positive rates.
BatchNorm helps BNNs avoid exploding gradients during training.
problem Difficulty in training Binary Neural Networks (BNNs).
method Theoretical study and numerical experiments on the role of BatchNorm in BNNs.
result BatchNorm is crucial for avoiding exploding gradients in BNNs.
PLLay adds topological layers to deep learning models efficiently.
problem Efficiently incorporating topological features into deep learning models.
method Persistence landscapes for differentiable topological features.
result PLLay improves model learnability and robustness.