Paper proposes an alternating back-propagation algorithm for generator networks.
problem Learning realistic generator models of natural images, video sequences, and sounds.
method Alternating back-propagation algorithm that iterates inferential and learning steps.
result The alternating back-propagation algorithm can learn realistic generator models of natural images, video sequences, and sounds.
Paper proposes new neural network learning algorithms inspired by predictive coding.
problem Finding biologically plausible alternatives to back-propagation of errors.
method Error-driven Local Representation Alignment (LRA-E) and Difference Target Propagation.
result Both proposed algorithms yield stable performance and strong generalization in training deeper, highly nonlinear networks.
Paper learns dynamic generator models for video sequences.
problem Modeling spatial-temporal processes like dynamic textures and actions.
method Alternating back-propagation through time algorithm to learn latent state vectors and generator model.
result Trains realistic models for dynamic textures and actions.
Functional transfer matrices replace weights in neural networks, achieving high accuracy.
problem Representing connections in neural networks with functions instead of weights.
method Developed functional transfer matrices, stacked them with bias vectors and activations, and trained them using back-propagation.
result Deep functional transfer neural networks can be trained to achieve high test accuracies on the MNIST database.
New back-propagation rules discovered through evolutionary methods.
problem Improving the back-propagation algorithm for faster training.
method Evolutionary approach to discover new update equations.
result Several new update equations that train faster and perform similarly at convergence.
ProxQuant improves quantized neural networks using proximal operators.
problem Making neural networks work on devices with limited resources.
method Formulates quantized network training as a regularized learning problem and optimizes it via the prox-gradient method.
result ProxQuant outperforms state-of-the-art results on binary quantization and is on par with state-of-the-art on multi-bit quantization.
Truncated back-propagation improves hyperparameter tuning and meta learning efficiency.
problem Computational challenges in evaluating exact gradients for high-dimensional bilevel optimization problems.
method Use truncated back-propagation to approximate gradients for the lower-level problem.
result Optimization with few-step back-propagation approximations often performs comparably to exact gradients, but with less memory and computation.
LRA trains deep networks robustly with less sensitivity to initial weights.
problem Training deep networks is challenging due to issues like exploding and vanishing gradients.
method Local Representation Alignment (LRA) is a training procedure less sensitive to initial weights.
result LRA can train networks robustly, even with null initial weights, and outperforms other methods.
The back-propagation algorithm is widely used for learning in artificial neural networks. A challenge in machine learning is to create models that generalize to new data samples not seen in the training data. Recently, a common flaw in several machine learning algorithms was discovered: small perturbations added to the…
A new method reduces backpropagation in CNNs by passing only a small subset of gradients.
problem High computational resources in traditional CNN backpropagation.
method Minimal Effort Back Propagation (MEBP) for CNNs, selecting only a small subset of gradients.
result Only 5% of gradients are passed back, achieving the same or better results as traditional CNNs.
Proposes a semi-implicit back propagation method for neural networks.
problem Challenges in training neural networks, especially gradient vanishing and small step sizes.
method Proposes a semi-implicit back propagation method using error back propagation and proximal methods.
result The proposed method leads to better performance in terms of loss decreasing and training/validation accuracy compared to SGD and ProxBP.
Deep networks trained with Hebbian updates perform similarly to back-propagation on image datasets.
problem Training deep networks with realistic asymmetric connections and updates.
method Use Hebbian updates with separate feedforward and feedback weights, and local rule for updates.
result Similar performance to back-propagation achieved with Hebbian updates on challenging image datasets.
SPIGOT bypasses gradients of argmax functions in neural nets with discrete latent variables.
problem Training neural networks with discrete latent variables.
method Structured projected intermediate gradient optimization technique (SPIGOT).
result SPIGOT bypasses gradients of argmax functions effectively.
The paper introduces various gradient descent algorithms for training deep learning models.
problem Training deep neural networks is challenging due to their complexity.
method Gradient descent and its variants are discussed for optimizing deep learning models.
result Gradient descent and its variants improve the training performance of deep learning models.
The study learns neural update rules by remembering past experiences.
problem Developing efficient online learning rules for neural networks.
method Representing neurons with vectors, using meta-neural networks for updates, and training for remembering past experiences.
result The approach reveals insights into learning rules and could be used for complex tasks like episodic memory.
NoProp learns neural networks without full back-propagation or forward-propagation.
problem Learning hierarchical representations in neural networks.
method NoProp independently learns each block to denoise a noisy target using local targets and back-propagation within the block.
result NoProp is a viable learning algorithm that is easy to use and computationally efficient.
HSIC bottleneck trains deep networks without backpropagation.
problem Training deep neural networks with exploding and vanishing gradients.
method HSIC bottleneck, alternative to cross-entropy loss and backpropagation.
result HSIC bottleneck achieves comparable performance to backpropagation.
A simplified backpropagation method reduces model complexity and computational cost.
problem Complexity and computational inefficiency in deep learning models.
method Sparsification of gradient vectors and adaptive model simplification.
result Models can be simplified by updating only a small fraction of weights, improving accuracy.
New method trains deep neural networks without backpropagation.
problem Training deep neural networks efficiently and without backpropagation.
method Direct feedback alignment using random weights for error propagation.
result Achieves zero training error in convolutional and deep networks.
Back-propagation learns camera sensor design for color images.
problem Designing efficient color camera sensors for deep learning.
method Jointly learns sensor design and image reconstruction networks.
result Significant accuracy improvements over traditional Bayer pattern.
Paper uses AI to predict stock market volatility with neural networks and genetic algorithms.
problem Traditional methods for predicting stock market volatility have high errors.
method Back-propagation neural network and genetic algorithm integrated model.
result The model predicts future volatility with low errors and high accuracy.
The paper improves RBP for stable and efficient training of recurrent neural networks.
problem Stability issues in RBP for training recurrent neural networks.
method Proposed two variants of RBP: CG-RBP and Neumann-RBP, and compared them with BPTT and TBPTT.
result Neumann-RBP is more efficient in terms of memory usage compared to TBPTT.
Defines a calculus for integrating Moreau envelopes in differentiable programming.
problem Lack of a mathematical framework for applying Moreau envelopes to deep networks and machine learning systems.
method Develops a compositional calculus adapted to Moreau envelopes and integrates it into differentiable programming.
result Integrates Moreau envelopes into differentiable programming, enabling new gradient back-propagation methods.
A new backprop method reduces memory usage for deep neural networks.
problem Memory-intensive backpropagation in deep neural networks.
method Uses approximations of activations in buffers to reduce memory usage.
result Performance close to exact training with 4-bit precision activations.
A new method improves fitting neural data with spiking network models.
problem Fitting spiking network models to neural activity does not produce realistic data.
method Augment log-likelihood with dissimilarity terms measured by summary statistics and optimized via back-propagation.
result The new method generates more realistic neural activity statistics and improves network connectivity inference.
We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and learning. Our algorithm introduces a recognition model to represent approximate posterior distributions, and that acts as…
A new method for training CNNs that decouples layer updates.
problem Update locking inefficiency in neural network training.
method Decoupled Greedy Learning (DGL) that relaxes joint training objective.
result DGL leads to better generalization than sequential greedy optimization.
New method adapts without backprop, faster and better.
problem Efficient domain adaptation without source data.
method Computing class prototypes from pre-trained model.
result Significant accuracy improvements over pre-trained model.
Optimization-based pruning eliminates backpropagation for large language models.
problem Suboptimal pruning performance due to heuristic metrics.
method Optimization of Bernoulli distribution to learn pruning masks without backpropagation.
result Efficient pruning of large language models with improved performance.
EXODUS improves training of SNNs by stabilizing gradients and reducing complexity.
problem Training SNNs using BPTT is time-consuming and numerically unstable.
method EXODUS modifies SLAYER to account for neuron reset and uses IFT for correct gradient calculation, eliminating manual scaling.
result EXODUS achieves comparable or better performance than SLAYER, especially in tasks with temporal features.
New neural method for inferring Markov jump processes.
problem Inference in Markov jump processes is challenging.
method Variational inference using neural ODEs and backpropagation.
result Trains neural representations of data to approximate process rates.
Paper justifies ST estimator using pWGF and proposes an improved variant.
problem Theoretical justification for ST estimator for discrete variables.
method Interpreted ST as pWGF simulation and proposed an improved estimator.
result Established theoretical foundation for ST estimator and improved variant.
ILM-Norm normalizes instances individually for better performance.
problem Learning to normalize parameters for improved model performance.
method ILM-Norm learns normalization parameters via feature feed-forward and gradient back-propagation.
result ILM-Norm consistently improves model performance across different architectures and tasks.
LDR neural networks reduce space and complexity with high accuracy.
problem Reducing space and computational complexity in large-scale neural networks.
method Formal study of LDR matrices, proving approximation property, error bounds, and proposing training algorithm.
result LDR neural networks achieve high accuracy with significant reduction in space and computational complexity.
A new adversarial DBN framework for deep learning.
problem Training deep belief networks (DBNs) efficiently and with scalability.
method Replacing generator in GANs with DBN and developing a parallelizable training algorithm.
result Framework can be applied to general DBNs without backpropagation.
FrequentNet uses frequency domain basis vectors for image classification, making models more interpretable and efficient.
problem Image classification models are often complex and hard to interpret.
method FrequentNet selects filter vectors from frequency domain basis vectors instead of training them with back propagation.
result The method improves interpretability and efficiency of image classification models.
A new method prunes activation gradients to speed up CNN training.
problem Challenges in accelerating CNN training using sparsity.
method Randomly prunes small activation gradients in back-propagation.
result Substantial speedups (up to 5.92x) with minimal accuracy loss.
PLIs improve classifier performance by fine-tuning latent representations.
problem Difficult interpretation of high-dimensional latent representations in neural networks.
method Back-propagation of manual changes to low-dimensional embeddings using t-distributed stochastic neighbourhood embeddings.
result Manual separation of class clusters in latent space enhances classifier performance.
A new probabilistic framework improves deep learning performance.
problem Improving deep learning models and understanding their limitations.
method Developed a probabilistic framework based on DRMM, a generative model capturing latent variables.
result DRMM outperforms DCNs in classification tasks, achieving state-of-the-art results.
Neural model parses non-projective dependency trees efficiently.
problem Parsing non-projective dependency trees.
method Probabilistic parsing model using neural representations and Kirchhoff's Matrix-Tree Theorem.
result State-of-the-art parsing performance on nine datasets.
PredRNN++ improves video predictive learning by alleviating gradient issues.
problem Gradient vanishing problem in deep spatiotemporal models.
method Causal LSTM and Gradient Highway architecture to manage gradients.
result PredRNN++ yields state-of-the-art prediction results even in occlusion scenarios.
A new learning framework mimics biological STDP for neural networks.
problem To create a neural network that learns like biological systems.
method Developed MSTDP framework using Spike-timing dependent plasticity rules.
result Framework can learn and generate patterns without additional supervision.
New method trains neural networks with local error signals, outperforming global methods.
problem Training neural networks with global error signals.
method Layer-wise training with local error signals.
result Layer-wise training with local error signals can approach state-of-the-art performance.
Backpropagation rules emerge from weighted automata combinations.
problem Understanding the effectiveness of deep learning.
method Combining finite automata using a weighted majority algorithm and simplifying the result to a neural network.
result Backpropagation is equivalent to a neural network's learning process.
A new gradient method reduces variance for non-reparameterizable distributions.
problem Efficient calculation of unbiased gradients for expectation-based objectives.
method GO Gradient, which applies to non-reparameterizable distributions and has low variance.
result GO Gradient reduces variance to the same level as reparameterization trick with one sample.
This paper proposes an alternative to E2E training for deep networks, reducing memory footprint.
problem High GPUs memory footprint in end-to-end training of deep networks.
method Locally supervised learning with information propagation loss to avoid information collapse.
result The proposed method achieves competitive performance with less than 40% memory footprint compared to E2E training.
Formula for complex SVD backpropagation developed.
problem No specific problem stated; focuses on complex SVD.
method Back propagation formula for complex SVD developed.
result Back propagation formula for complex SVD created.
Paper visualizes and analyzes deep neural networks for CTR prediction.
problem Interpreting deep neural networks for CTR prediction is challenging.
method Visualizing and analyzing deep neural networks at neuron level, measuring layer-wise performance, and calculating saliency scores.
result Insights into the model's performance and mechanism.