Unified backpropagation improves multi-objective deep learning performance.
problem Improving classification performance in hybrid neural networks.
method Linking hybrid loss functions through unified backpropagation.
result Consistent improvements in deep convolutional neural network classification performance.
Unified framework for faster neural network training with less information loss.
problem Time-consuming backpropagation and loss of unpropagated gradient information.
method Unified sparse backpropagation framework and memorized sparse backpropagation algorithm.
result Convergence in probability with certain conditions and effective information loss mitigation.
VSML unifies meta learning concepts and enables simple backpropagation.
problem Improving and unifying meta learning concepts for neural networks.
method Unified approach using variable shared meta learning and simple weight-sharing.
result Simple backpropagation can be implemented and meta learned without gradient calculation.
This work shows synthetic gradients can outperform backpropagation in sample efficiency.
problem The efficiency of backpropagation in training neural networks.
method Unified vectorized feedback framework for loss-based and reward-based learning, introducing synthetic gradients.
result Synthetic gradients can achieve lower gradient-estimation mean squared error than backpropagation under certain conditions.
Unified framework reduces NFEs for inverse problems.
problem High computational costs and degraded reconstruction quality in existing LDM-based inverse solvers.
method Consistency Regularised Gradient Flows for posterior sampling and prompt optimization.
result Significantly reduced computational cost with state-of-the-art performance.
Unified approach to multiclass classification using Gabriel graphs.
problem Improving multiclass classification accuracy and efficiency.
method Integrates Gabriel graphs for binary and multiclass classification, proposing new activation functions and support edge neurons.
result Experimental results show superior performance compared to previous GG-based classifiers.
Dynamic Sparse Training finds efficient sparse networks from scratch.
problem Finding efficient sparse neural networks.
method Jointly optimizes network parameters and sparsity with trainable thresholds.
result Achieves state-of-the-art performance with minimal performance loss.
Unified deep learning framework solves various optimal transport problems.
problem Solving variational problems in optimal transport with computational challenges.
method Unified deep learning framework leveraging dual formulation of Lagrangians.
result Outperforms previous approaches in single-cell trajectory inference.
Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods.
problem Optimizing neural networks using standard methods like Adam and RMSprop.
method Formulated as Bayesian filtering, accounting for temporal dynamics of all parameters.
result Recover Adam and AdamW optimizers with competitive generalization performance.
Belief propagation recovers backpropagation results.
problem Connection between backpropagation and belief propagation poorly understood.
method Converted backpropagation input to belief propagation input and showed results.
result Backpropagation is a special case of belief propagation.
Paper proposes DAG-DB for learning discrete DAGs via backpropagation.
problem Learning Directed Acyclic Graphs (DAGs) from data.
method DAG-DB uses Discrete Backpropagation with I-MLE and Straight-Through Estimation.
result DAG-DB learns DAGs effectively using probabilistic sampling and backpropagation.
Unified approach to guided generation techniques.
problem Controlling the generative process of flow/diffusion models.
method Unified posterior and end-to-end guidance techniques.
result Unified posterior guidance as a greedy strategy of end-to-end guidance.
GAIT-prop derives a biologically plausible learning rule from backpropagation.
problem Biological implausibility in traditional backpropagation for neural networks.
method GAIT-prop uses a top-down model to convert output error into plausible targets for weight updates.
result GAIT-prop and backpropagation give identical weight updates under certain conditions.
Improved BER with reduced power in time-domain digital backpropagation.
problem Improving BER performance in time-domain digital backpropagation.
method Jointly optimized and quantized chromatic dispersion filters using machine learning.
result Improved BER performance and power dissipation reductions.
Proposes a new backpropagation algorithm for deep learning with guaranteed convergence.
problem Backward locking in backpropagation limits parallel updates in deep neural networks.
method Decouples gradients and splits the network into modules for parallel updates, proving convergence for non-convex problems.
result The proposed algorithm achieves significant speedup without accuracy loss in training deep convolutional neural networks.
A new method computes gradients without backpropagation.
problem Optimization of machine learning models.
method Forward mode automatic differentiation to compute gradients.
result Forward gradient is an unbiased estimate of the gradient, eliminating the need for backpropagation.
Backprop-Q extends standard backpropagation for stochastic computation graphs.
problem Applying standard backpropagation to stochastic computation graphs is challenging.
method Construct Q-functions for each stochastic node and use them to train the SCG with standard backpropagation.
result Generalized backpropagation for stochastic computation graphs is feasible and extends learning signals beyond gradients.
New algorithm shows neural networks can learn without full backpropagation.
problem Stochastic gradient descent with backpropagation is non-biologically plausible.
method Random and fixed backpropagation weights in a feedback alignment algorithm.
result Error converges to zero exponentially fast in overparameterized networks.
Unified geometric principles unify neural network architectures.
problem High-dimensional learning tasks with underlying low-dimensionality and structure.
method Unified geometric principles applied to neural network architectures.
result Unified mathematical framework for neural network architectures.
Skip connections improve biologically-inspired learning rules.
problem Biologically-inspired learning rules often underperform compared to backpropagation.
method Introduced skip connections between intermediate layers in biologically-motivated learning rules.
result Skip connections can match the performance of backpropagation and are robust to hyper-parameters.
Pipelined Backpropagation trains large models without batches efficiently.
problem Training large models efficiently on hardware with limited batch sizes.
method Fine-grained Pipelined Backpropagation with Spike Compensation and Linear Weight Prediction.
result Fine-grained Pipelined Backpropagation with a batch size of one matches the accuracy of SGD for multiple networks.
Spectral backpropagation optimizes implicit likelihoods without explicit assumptions.
problem Training systems without explicit goals and likelihood functions.
method Spectral backpropagation for implicit likelihood optimization.
result Identified two novel properties of GAN generators: aberrant outputs and quasi-disentangled factors.
ZORB speeds up neural network training without sacrificing accuracy.
problem Slow and strenuous training of neural networks using gradient descent.
method Uses pseudoinverse of targets instead of gradients for backpropagation.
result ZORB converges 300 times faster than Adam on MNIST without hyperparameter tuning.
HSIC bottleneck trains deep networks without backpropagation.
problem Training deep neural networks with exploding and vanishing gradients.
method HSIC bottleneck, alternative to cross-entropy loss and backpropagation.
result HSIC bottleneck achieves comparable performance to backpropagation.
FlowChef steers RFMs to efficiently guide image generation tasks.
problem Efficiently guiding image generation tasks with RFMs.
method Developed a theoretical and empirical understanding of RFMs' vector field dynamics, proposing FlowChef for gradient-free navigation.
result FlowChef significantly outperforms baselines in performance, memory, and time requirements.
Mean field theory explains gradient backpropagation in deep dropout networks.
problem Understanding gradient backpropagation in deep dropout networks.
method Applied mean field theory to dropout networks, considering realistic training conditions.
result Gradient backpropagation length is limited by depth scales, not just independence assumption.
Generalizes double backpropagation for neural networks in Hilbert space.
problem Lack of a general description of derivatives in neural network loss functions.
method Develops a Hilbert space framework to cover various special cases of double backpropagation.
result Optimized backpropagation rules for real-world scenarios, reducing calculations by up to 33%.
Paper proposes using backpropagation for probabilistic program learning.
problem Difficult to learn probabilistic models from data.
method Learning parameters of a probabilistic program using backpropagation.
result Trains probabilistic models similar to neural networks.
Backpropagation-free RL method trains layers using local signals.
problem Vanishing or exploding gradients in backpropagation-based RL.
method Local pairwise distance matching for layer-wise training without backpropagation.
result Backpropagation-free method achieves competitive performance and stability.
Sparse Attentive Backtracking selectively backpropagates long-term dependencies in recurrent networks.
problem Difficulty in learning long-term dependencies in BPTT due to computational impracticality and biased gradient estimates.
method Sparse Attentive Backtracking learns an attention mechanism over past hidden states and selectively backpropagates through high-weight paths.
result Model learns long-term dependencies with fewer backpropagation steps, addressing biased gradient issues.
A new neural network using chi-square test for binary classification.
problem Improving binary classification accuracy.
method Backpropagation neural network with chi-square test redefined cost and error functions.
result Significantly improved classification accuracy compared to related approaches.
Improved signal processing for long-distance optical signals.
problem Compensating walk-off effect in long-distance optical signals.
method Sub-banded DSP architecture with deep learning for walk-off compensation.
result 2.8 dB SNR improvement over linear equalization.
Sideways trains video models by overwriting activations as new frames arrive, potentially improving generalization.
problem Training deep video models synchronously slows down and requires storing activations, limiting parallelism.
method Sideways trains video models by overwriting activations as new frames arrive, breaking the precise correspondence between gradients and activations.
result Sideways training can converge and potentially generalize better than standard synchronized backpropagation.
Study shows Direct Feedback Alignment fails to offer more efficient scaling than backpropagation.
problem Understanding and optimizing training methods for neural networks.
method Use of scaling laws to compare Direct Feedback Alignment (DFA) and backpropagation.
result DFA fails to offer more efficient scaling than backpropagation.
SLAYER improves SNN training by backpropagating spike errors.
problem Non-differentiability of spike function in SNNs.
method Introduces SLAYER for learning weights and delays, using temporal credit assignment.
result SLAYER achieves state-of-the-art performance on various datasets.
A method for learning sparse transformations through backpropagation.
problem Sparse transformations in deep learning architectures are hard to design and often represented densely during learning.
method Adaptive, sparse hyperlayer with randomly sampled connections to overcome gradient issues.
result Trained models achieve competitive performance on real data.
Paper introduces robust method for training neural networks.
problem Neural network hyperparameter tuning, particularly learning rate sensitivity.
method Implicit Stochastic Gradient Descent (ISGD) layer-wise approximation for neural networks.
result Our method is more robust to high learning rates and generally outperforms standard backpropagation.
Article presents QR and LQ decomposition algorithms for various matrix sizes and ranks.
problem Solving least squares problems in machine learning and computer vision.
method Developed novel matrix backpropagation algorithms for QR and LQ decompositions of different matrix sizes and ranks.
result Numerical stability and computational efficiency of the proposed methods.
A faster, more stable method for optimizing topological functions.
problem Optimizing topological functions is computationally expensive and unstable.
method Introduces a novel backpropagation scheme for faster and more robust optimization.
result Produces more robust optima and stable visualizations.
New learning rules for wide neural networks without backpropagation.
problem Training wide neural networks efficiently and without backpropagation.
method Input-weight alignment driven by gradient descent in the NTK regime.
result Biologically-motivated learning rules equivalent to backpropagation in wide networks.
Study proposes memory-efficient backpropagation for linear layers in neural networks.
problem Significant memory usage in backpropagation through linear layers in neural networks.
method Randomized matrix multiplications to reduce memory usage with a moderate decrease in test accuracy.
result Demonstrated benefits of the proposed method on fine-tuning pre-trained models.
Unified framework for subgraph-enhanced GNNs, improving prediction accuracy and reducing computation time.
problem Limited understanding of subgraph-enhanced GNNs and their relation to the Weisfeiler-Leman hierarchy.
method Theoretical framework, theoretical expressivity results, and data-driven subgraph sampling methods.
result Data-driven subgraph-enhanced GNNs outperform non-data-driven methods in predictive performance.
Backpropagation-free trunk training improves model performance on various benchmarks.
problem Memory inefficiency and noisy gradient estimates in deep network training.
method Split Forward Gradient (Split-FG) method that splits network into trunk and head, estimating only trunk gradient.
result Split-FG achieves better performance than pure forward-gradient training and backpropagation on various benchmarks.
Backdrop uses dropout-like masking in backpropagation for multi-scale data.
problem Improving generalization in multi-scale data.
method Inserting masking layers after convolutional layers to mask backward gradients.
result Backdrop leads to significant improvements in generalization.
New method reduces deep learning training costs by approximating vector-jacobian products.
problem Efficiently training deep neural networks with reduced computational and memory costs.
method Randomized, unbiased approximations of vector-jacobian products during backpropagation.
result Validated potential for reducing deep learning training costs through unbiased estimates.
Unified framework for sparse alternatives to softmax with control over sparsity.
problem Lack of understanding and explicit control over sparsity in probability mapping functions.
method Unified framework encompassing softmax, sum-normalization, spherical softmax, and sparsemax. Two novel sparse formulations (sparsegen-lin and sparsehourglass) and convex loss functions developed.
result Improved performance in multilabel classification and seq2seq tasks like neural machine translation and abstractive summarization.
Paper explores BP's biological plausibility and alternative learning algorithms.
problem Backpropagation's biological plausibility in neural networks.
method Framing supervised learning in the Lagrangian framework to devise biologically plausible local algorithms.
result Biologically plausible learning algorithms can be devised based on saddle point search in the learning adjoint space.
A new model uses 'ghost units' to enable efficient backpropagation in deep neural networks.
problem How to achieve efficient backpropagation in deep neural networks with biological plausibility.
method Introduces 'ghost units' to cancel feedback, enabling efficient error backpropagation.
result Demonstrates that the model can approximate error gradients and achieve good performance on classification tasks.