This work proposes a new quantum neural network model and training algorithm.
problem Classical neural networks are computationally expensive, especially for tasks like image classification.
method Defines quantum neural networks via quantum time evolution and Hamiltonian, proposing a quantum backpropagation algorithm.
result Validated the proposed quantum backpropagation algorithm on the MNIST dataset using a quantum computer simulation.
Adversarial learning is one of the most successful approaches to modelling high-dimensional probability distributions from data. The quantum computing community has recently begun to generalize this idea and to look for potential applications. In this work, we derive an adversarial algorithm for the problem of approxim…
Bayesian methods in machine learning, such as Gaussian processes, have great advantages com-pared to other techniques. In particular, they provide estimates of the uncertainty associated with a prediction. Extending the Bayesian approach to deep architectures has remained a major challenge. Recent results connected dee…
In Deep Learning, a well-known approach for training a Deep Neural Network starts by training a generative Deep Belief Network model, typically using Contrastive Divergence (CD), then fine-tuning the weights using backpropagation or other discriminative techniques. However, the generative training can be time-consuming…
Belief propagation recovers backpropagation results.
problem Connection between backpropagation and belief propagation poorly understood.
method Converted backpropagation input to belief propagation input and showed results.
result Backpropagation is a special case of belief propagation.
Paper proposes DAG-DB for learning discrete DAGs via backpropagation.
problem Learning Directed Acyclic Graphs (DAGs) from data.
method DAG-DB uses Discrete Backpropagation with I-MLE and Straight-Through Estimation.
result DAG-DB learns DAGs effectively using probabilistic sampling and backpropagation.
GAIT-prop derives a biologically plausible learning rule from backpropagation.
problem Biological implausibility in traditional backpropagation for neural networks.
method GAIT-prop uses a top-down model to convert output error into plausible targets for weight updates.
result GAIT-prop and backpropagation give identical weight updates under certain conditions.
This work shows synthetic gradients can outperform backpropagation in sample efficiency.
problem The efficiency of backpropagation in training neural networks.
method Unified vectorized feedback framework for loss-based and reward-based learning, introducing synthetic gradients.
result Synthetic gradients can achieve lower gradient-estimation mean squared error than backpropagation under certain conditions.
A new method computes gradients without backpropagation.
problem Optimization of machine learning models.
method Forward mode automatic differentiation to compute gradients.
result Forward gradient is an unbiased estimate of the gradient, eliminating the need for backpropagation.
Unified framework for faster neural network training with less information loss.
problem Time-consuming backpropagation and loss of unpropagated gradient information.
method Unified sparse backpropagation framework and memorized sparse backpropagation algorithm.
result Convergence in probability with certain conditions and effective information loss mitigation.
New algorithm shows neural networks can learn without full backpropagation.
problem Stochastic gradient descent with backpropagation is non-biologically plausible.
method Random and fixed backpropagation weights in a feedback alignment algorithm.
result Error converges to zero exponentially fast in overparameterized networks.
Skip connections improve biologically-inspired learning rules.
problem Biologically-inspired learning rules often underperform compared to backpropagation.
method Introduced skip connections between intermediate layers in biologically-motivated learning rules.
result Skip connections can match the performance of backpropagation and are robust to hyper-parameters.
Pipelined Backpropagation trains large models without batches efficiently.
problem Training large models efficiently on hardware with limited batch sizes.
method Fine-grained Pipelined Backpropagation with Spike Compensation and Linear Weight Prediction.
result Fine-grained Pipelined Backpropagation with a batch size of one matches the accuracy of SGD for multiple networks.
ZORB speeds up neural network training without sacrificing accuracy.
problem Slow and strenuous training of neural networks using gradient descent.
method Uses pseudoinverse of targets instead of gradients for backpropagation.
result ZORB converges 300 times faster than Adam on MNIST without hyperparameter tuning.
Backpropagation algorithm is indispensable for the training of feedforward neural networks. It requires propagating error gradients sequentially from the output layer all the way back to the input layer. The backward locking in backpropagation algorithm constrains us from updating network layers in parallel and fully l…
HSIC bottleneck trains deep networks without backpropagation.
problem Training deep neural networks with exploding and vanishing gradients.
method HSIC bottleneck, alternative to cross-entropy loss and backpropagation.
result HSIC bottleneck achieves comparable performance to backpropagation.
Mean field theory explains gradient backpropagation in deep dropout networks.
problem Understanding gradient backpropagation in deep dropout networks.
method Applied mean field theory to dropout networks, considering realistic training conditions.
result Gradient backpropagation length is limited by depth scales, not just independence assumption.
Generalizes double backpropagation for neural networks in Hilbert space.
problem Lack of a general description of derivatives in neural network loss functions.
method Develops a Hilbert space framework to cover various special cases of double backpropagation.
result Optimized backpropagation rules for real-world scenarios, reducing calculations by up to 33%.
Backpropagation-free RL method trains layers using local signals.
problem Vanishing or exploding gradients in backpropagation-based RL.
method Local pairwise distance matching for layer-wise training without backpropagation.
result Backpropagation-free method achieves competitive performance and stability.
In real-world scenarios, it is appealing to learn a model carrying out stochastic operations internally, known as stochastic computation graphs (SCGs), rather than learning a deterministic mapping. However, standard backpropagation is not applicable to SCGs. We attempt to address this issue from the angle of cost propa…
VSML unifies meta learning concepts and enables simple backpropagation.
problem Improving and unifying meta learning concepts for neural networks.
method Unified approach using variable shared meta learning and simple weight-sharing.
result Simple backpropagation can be implemented and meta learned without gradient calculation.
Sideways trains video models by overwriting activations as new frames arrive, potentially improving generalization.
problem Training deep video models synchronously slows down and requires storing activations, limiting parallelism.
method Sideways trains video models by overwriting activations as new frames arrive, breaking the precise correspondence between gradients and activations.
result Sideways training can converge and potentially generalize better than standard synchronized backpropagation.
Study shows Direct Feedback Alignment fails to offer more efficient scaling than backpropagation.
problem Understanding and optimizing training methods for neural networks.
method Use of scaling laws to compare Direct Feedback Alignment (DFA) and backpropagation.
result DFA fails to offer more efficient scaling than backpropagation.
Article presents QR and LQ decomposition algorithms for various matrix sizes and ranks.
problem Solving least squares problems in machine learning and computer vision.
method Developed novel matrix backpropagation algorithms for QR and LQ decompositions of different matrix sizes and ranks.
result Numerical stability and computational efficiency of the proposed methods.
A faster, more stable method for optimizing topological functions.
problem Optimizing topological functions is computationally expensive and unstable.
method Introduces a novel backpropagation scheme for faster and more robust optimization.
result Produces more robust optima and stable visualizations.
New learning rules for wide neural networks without backpropagation.
problem Training wide neural networks efficiently and without backpropagation.
method Input-weight alignment driven by gradient descent in the NTK regime.
result Biologically-motivated learning rules equivalent to backpropagation in wide networks.
Study proposes memory-efficient backpropagation for linear layers in neural networks.
problem Significant memory usage in backpropagation through linear layers in neural networks.
method Randomized matrix multiplications to reduce memory usage with a moderate decrease in test accuracy.
result Demonstrated benefits of the proposed method on fine-tuning pre-trained models.
We introduce the chi-square test neural network: a single hidden layer backpropagation neural network using chi-square test theorem to redefine the cost function and the error function. The weights and thresholds are modified using standard backpropagation algorithm. The proposed approach has the advantage of making co…
Most successful machine intelligence systems rely on gradient-based learning, which is made possible by backpropagation. Some systems are designed to aid us in interpreting data when explicit goals cannot be provided. These unsupervised systems are commonly trained by backpropagating through a likelihood function. We i…
Backpropagation-free trunk training improves model performance on various benchmarks.
problem Memory inefficiency and noisy gradient estimates in deep network training.
method Split Forward Gradient (Split-FG) method that splits network into trunk and head, estimating only trunk gradient.
result Split-FG achieves better performance than pure forward-gradient training and backpropagation on various benchmarks.
We consider time-domain digital backpropagation with chromatic dispersion filters jointly optimized and quantized using machine-learning techniques. Compared to the baseline implementations, we show improved BER performance and >40% power dissipation reductions in 28-nm CMOS.
New method reduces deep learning training costs by approximating vector-jacobian products.
problem Efficiently training deep neural networks with reduced computational and memory costs.
method Randomized, unbiased approximations of vector-jacobian products during backpropagation.
result Validated potential for reducing deep learning training costs through unbiased estimates.
A new model uses 'ghost units' to enable efficient backpropagation in deep neural networks.
problem How to achieve efficient backpropagation in deep neural networks with biological plausibility.
method Introduces 'ghost units' to cancel feedback, enabling efficient error backpropagation.
result Demonstrates that the model can approximate error gradients and achieve good performance on classification tasks.
Arguably the biggest challenge in applying neural networks is tuning the hyperparameters, in particular the learning rate. The sensitivity to the learning rate is due to the reliance on backpropagation to train the network. In this paper we present the first application of Implicit Stochastic Gradient Descent (ISGD) to…
We propose a low-complexity sub-banded DSP architecture for digital backpropagation where the walk-off effect is compensated using simple delay elements. For a simulated 96-Gbaud signal and 2500 km optical link, our method achieves a 2.8 dB SNR improvement over linear equalization.
Forward gradients improve neural network training without backpropagation issues.
problem Training neural networks without backpropagation's locking and memorization problems.
method Using directional derivatives in forward differentiation mode, with biased guesses based on feedback from small auxiliary networks.
result Using gradients from a local loss as a candidate direction improves Forward Gradient methods.
A major drawback of backpropagation through time (BPTT) is the difficulty of learning long-term dependencies, coming from having to propagate credit information backwards through every single step of the forward computation. This makes BPTT both computationally impractical and biologically implausible. For this reason,…
ARSM estimator improves gradient backpropagation for categorical variables.
problem Improving gradient backpropagation through categorical variables.
method ARSM combines variable augmentation, REINFORCE, Rao-Blackwellization, and variable swapping.
result ARSM outperforms existing estimators and provides variance reduction methods.
Large multilayer neural networks trained with backpropagation have recently achieved state-of-the-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic pr…
We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is accomplished by generalizing the gradient computation in stochastic backpropagation via a reparametriza…
DNArch learns CNN architectures by backpropagation.
problem Discovering optimal CNN architectures.
method Differentiable Neural Architectures (DNArch) learns CNN architectures by backpropagation, controlling kernel sizes, channels, downsampling positions, and depth.
result DNArch finds performant CNN architectures across various tasks.
AR algorithm simplifies backpropagation with improved scalability and biological plausibility.
problem Improving backpropagation algorithms for complex neural networks and biological plausibility.
method Introducing learnable backwards weights and avoiding nonlinear derivative computations; relaxing frozen feedforward pass assumption.
result Simplified AR algorithm maintains performance on complex CNN architectures and challenging datasets.
FPGA-based multi-layer equalizer adapts to changing channels.
problem Real-time adaptation to time-varying channel impairments.
method Multi-layer machine learning on FPGA with on-chip gradient backpropagation training.
result Real-time adaptation to changing channel conditions achieved.
LLT-ECG classifies ECG signals without backpropagation using linear laws.
problem ECG signal classification efficiency and verifiability.
method Forward linear law identification for ECG signal features.
result State-of-the-art performance on real-world ECG datasets.
We translate ML pipelines into neural networks to optimize multiple models together.
problem Isolated training of ML pipelines limits joint optimization of multiple models.
method Propose translating ML pipelines into neural networks and fine-tuning them jointly.
result Fine-tuning translated pipelines increases final accuracy.
AD-HOC simplifies high-order derivative calculations in C++.
problem Efficiently computing high-order derivatives in C++.
method A C++ package that calculates derivatives of arbitrary order without code generation.
result Derivatives of arbitrary order computed in a single pass.
We provide a proof of backpropagation algorithm in matrix notation.
problem The lack of a full induction proof of backpropagation algorithm in matrix notation.
method We provide a full induction proof of the BP algorithm in matrix notation, situating it in the framework of matrix differential calculus.
result We prove the validity of the backpropagation algorithm in inductive form.
By and large, Backpropagation (BP) is regarded as one of the most important neural computation algorithms at the basis of the progress in machine learning, including the recent advances in deep learning. However, its computational structure has been the source of many debates on its arguable biological plausibility. In…