Paper proposes an alternating back-propagation algorithm for generator networks.
problem Learning realistic generator models of natural images, video sequences, and sounds.
method Alternating back-propagation algorithm that iterates inferential and learning steps.
result The alternating back-propagation algorithm can learn realistic generator models of natural images, video sequences, and sounds.
New back-propagation rules discovered through evolutionary methods.
problem Improving the back-propagation algorithm for faster training.
method Evolutionary approach to discover new update equations.
result Several new update equations that train faster and perform similarly at convergence.
New algorithm trains deep neural networks with adaptive learning rates.
problem Inconsistent gradient magnitudes across layers in SGD.
method Back-matching propagation with approximations for layer-wise adaptive learning rates.
result Achieves favorable results over standard SGD in training deep neural networks.
Truncated back-propagation improves hyperparameter tuning and meta learning efficiency.
problem Computational challenges in evaluating exact gradients for high-dimensional bilevel optimization problems.
method Use truncated back-propagation to approximate gradients for the lower-level problem.
result Optimization with few-step back-propagation approximations often performs comparably to exact gradients, but with less memory and computation.
A new method reduces backpropagation in CNNs by passing only a small subset of gradients.
problem High computational resources in traditional CNN backpropagation.
method Minimal Effort Back Propagation (MEBP) for CNNs, selecting only a small subset of gradients.
result Only 5% of gradients are passed back, achieving the same or better results as traditional CNNs.
Proposes a semi-implicit back propagation method for neural networks.
problem Challenges in training neural networks, especially gradient vanishing and small step sizes.
method Proposes a semi-implicit back propagation method using error back propagation and proximal methods.
result The proposed method leads to better performance in terms of loss decreasing and training/validation accuracy compared to SGD and ProxBP.
Paper proposes new neural network learning algorithms inspired by predictive coding.
problem Finding biologically plausible alternatives to back-propagation of errors.
method Error-driven Local Representation Alignment (LRA-E) and Difference Target Propagation.
result Both proposed algorithms yield stable performance and strong generalization in training deeper, highly nonlinear networks.
The back-propagation algorithm is widely used for learning in artificial neural networks. A challenge in machine learning is to create models that generalize to new data samples not seen in the training data. Recently, a common flaw in several machine learning algorithms was discovered: small perturbations added to the…
Deep networks trained with Hebbian updates perform similarly to back-propagation on image datasets.
problem Training deep networks with realistic asymmetric connections and updates.
method Use Hebbian updates with separate feedforward and feedback weights, and local rule for updates.
result Similar performance to back-propagation achieved with Hebbian updates on challenging image datasets.
NoProp learns neural networks without full back-propagation or forward-propagation.
problem Learning hierarchical representations in neural networks.
method NoProp independently learns each block to denoise a noisy target using local targets and back-propagation within the block.
result NoProp is a viable learning algorithm that is easy to use and computationally efficient.
New method trains deep neural networks without backpropagation.
problem Training deep neural networks efficiently and without backpropagation.
method Direct feedback alignment using random weights for error propagation.
result Achieves zero training error in convolutional and deep networks.
Back-propagation learns camera sensor design for color images.
problem Designing efficient color camera sensors for deep learning.
method Jointly learns sensor design and image reconstruction networks.
result Significant accuracy improvements over traditional Bayer pattern.
The paper introduces various gradient descent algorithms for training deep learning models.
problem Training deep neural networks is challenging due to their complexity.
method Gradient descent and its variants are discussed for optimizing deep learning models.
result Gradient descent and its variants improve the training performance of deep learning models.
The study learns neural update rules by remembering past experiences.
problem Developing efficient online learning rules for neural networks.
method Representing neurons with vectors, using meta-neural networks for updates, and training for remembering past experiences.
result The approach reveals insights into learning rules and could be used for complex tasks like episodic memory.
Defines a calculus for integrating Moreau envelopes in differentiable programming.
problem Lack of a mathematical framework for applying Moreau envelopes to deep networks and machine learning systems.
method Develops a compositional calculus adapted to Moreau envelopes and integrates it into differentiable programming.
result Integrates Moreau envelopes into differentiable programming, enabling new gradient back-propagation methods.
A simplified backpropagation method reduces model complexity and computational cost.
problem Complexity and computational inefficiency in deep learning models.
method Sparsification of gradient vectors and adaptive model simplification.
result Models can be simplified by updating only a small fraction of weights, improving accuracy.
A new method for target propagation using iterative approximations converges fast and is more biologically plausible.
problem Improving target propagation methods for neural networks.
method Iterative approximate inverses and local auto-encoders.
result The method converges exponentially fast under certain conditions.
Paper uses AI to predict stock market volatility with neural networks and genetic algorithms.
problem Traditional methods for predicting stock market volatility have high errors.
method Back-propagation neural network and genetic algorithm integrated model.
result The model predicts future volatility with low errors and high accuracy.
Functional transfer matrices replace weights in neural networks, achieving high accuracy.
problem Representing connections in neural networks with functions instead of weights.
method Developed functional transfer matrices, stacked them with bias vectors and activations, and trained them using back-propagation.
result Deep functional transfer neural networks can be trained to achieve high test accuracies on the MNIST database.
The paper improves RBP for stable and efficient training of recurrent neural networks.
problem Stability issues in RBP for training recurrent neural networks.
method Proposed two variants of RBP: CG-RBP and Neumann-RBP, and compared them with BPTT and TBPTT.
result Neumann-RBP is more efficient in terms of memory usage compared to TBPTT.
A new backprop method reduces memory usage for deep neural networks.
problem Memory-intensive backpropagation in deep neural networks.
method Uses approximations of activations in buffers to reduce memory usage.
result Performance close to exact training with 4-bit precision activations.
Alternative method proposed for handling uncertainty in i-vector extraction.
problem Uncertainty in i-vector extraction for spoken language recognition.
method Proposes an alternative method to propagate uncertainty into the Gaussian back-end.
result Alternative method effectively handles uncertainty in i-vector extraction.
A new method improves fitting neural data with spiking network models.
problem Fitting spiking network models to neural activity does not produce realistic data.
method Augment log-likelihood with dissimilarity terms measured by summary statistics and optimized via back-propagation.
result The new method generates more realistic neural activity statistics and improves network connectivity inference.
Paper learns dynamic generator models for video sequences.
problem Modeling spatial-temporal processes like dynamic textures and actions.
method Alternating back-propagation through time algorithm to learn latent state vectors and generator model.
result Trains realistic models for dynamic textures and actions.
We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and learning. Our algorithm introduces a recognition model to represent approximate posterior distributions, and that acts as…
New method adapts without backprop, faster and better.
problem Efficient domain adaptation without source data.
method Computing class prototypes from pre-trained model.
result Significant accuracy improvements over pre-trained model.
Optimization-based pruning eliminates backpropagation for large language models.
problem Suboptimal pruning performance due to heuristic metrics.
method Optimization of Bernoulli distribution to learn pruning masks without backpropagation.
result Efficient pruning of large language models with improved performance.
EXODUS improves training of SNNs by stabilizing gradients and reducing complexity.
problem Training SNNs using BPTT is time-consuming and numerically unstable.
method EXODUS modifies SLAYER to account for neuron reset and uses IFT for correct gradient calculation, eliminating manual scaling.
result EXODUS achieves comparable or better performance than SLAYER, especially in tasks with temporal features.
A new method backtracks through a few key past states to speed up credit assignment in long sequences.
problem Computational inefficiency of back-propagation through time for long sequences.
method Sparse attentive backtracking using learned attention mechanisms to skip connections.
result Matches or outperforms regular BPTT and truncated BPTT in tasks with long-term dependencies.
Paper introduces MNNs for efficient neural network training.
problem Training large-scale sparse neural networks efficiently.
method Formal derivation of MNN architecture with forward-only gradient propagation.
result MNNs can efficiently compute gradients without backward computation.
A new gradient method reduces variance for non-reparameterizable distributions.
problem Efficient calculation of unbiased gradients for expectation-based objectives.
method GO Gradient, which applies to non-reparameterizable distributions and has low variance.
result GO Gradient reduces variance to the same level as reparameterization trick with one sample.
This thesis investigates belief propagation's performance in graphical models with loops.
problem Belief propagation's performance and convergence guarantees in models with loops are uncertain.
method Investigates how model parameters affect belief propagation's performance, convergence, and approximation quality.
result Model parameters influence the number of fixed points, convergence properties, and approximation quality of belief propagation.
ILM-Norm normalizes instances individually for better performance.
problem Learning to normalize parameters for improved model performance.
method ILM-Norm learns normalization parameters via feature feed-forward and gradient back-propagation.
result ILM-Norm consistently improves model performance across different architectures and tasks.
LDR neural networks reduce space and complexity with high accuracy.
problem Reducing space and computational complexity in large-scale neural networks.
method Formal study of LDR matrices, proving approximation property, error bounds, and proposing training algorithm.
result LDR neural networks achieve high accuracy with significant reduction in space and computational complexity.
Paper closes neural-symbolic learning loop with grammar model and back-search algorithm.
problem Slow convergence in neural-symbolic learning due to error propagation issues.
method Introduces grammar model as symbolic prior and back-search algorithm for efficient error propagation.
result Significantly outperforms RL methods in performance, converging speed, and data efficiency.
A new adversarial DBN framework for deep learning.
problem Training deep belief networks (DBNs) efficiently and with scalability.
method Replacing generator in GANs with DBN and developing a parallelizable training algorithm.
result Framework can be applied to general DBNs without backpropagation.
LRA trains deep networks robustly with less sensitivity to initial weights.
problem Training deep networks is challenging due to issues like exploding and vanishing gradients.
method Local Representation Alignment (LRA) is a training procedure less sensitive to initial weights.
result LRA can train networks robustly, even with null initial weights, and outperforms other methods.
FrequentNet uses frequency domain basis vectors for image classification, making models more interpretable and efficient.
problem Image classification models are often complex and hard to interpret.
method FrequentNet selects filter vectors from frequency domain basis vectors instead of training them with back propagation.
result The method improves interpretability and efficiency of image classification models.
A new method prunes activation gradients to speed up CNN training.
problem Challenges in accelerating CNN training using sparsity.
method Randomly prunes small activation gradients in back-propagation.
result Substantial speedups (up to 5.92x) with minimal accuracy loss.
PLIs improve classifier performance by fine-tuning latent representations.
problem Difficult interpretation of high-dimensional latent representations in neural networks.
method Back-propagation of manual changes to low-dimensional embeddings using t-distributed stochastic neighbourhood embeddings.
result Manual separation of class clusters in latent space enhances classifier performance.
Neural model parses non-projective dependency trees efficiently.
problem Parsing non-projective dependency trees.
method Probabilistic parsing model using neural representations and Kirchhoff's Matrix-Tree Theorem.
result State-of-the-art parsing performance on nine datasets.
A new learning framework mimics biological STDP for neural networks.
problem To create a neural network that learns like biological systems.
method Developed MSTDP framework using Spike-timing dependent plasticity rules.
result Framework can learn and generate patterns without additional supervision.
Proposes dynamic channel pruning during neural network training.
problem Pruning neural networks during training to reduce computational cost and improve efficiency.
method Dynamic channel propagation to update channel utility values and selectively prune channels.
result Our scheme trains and prunes neural networks simultaneously, achieving superior performance.
Backpropagation rules emerge from weighted automata combinations.
problem Understanding the effectiveness of deep learning.
method Combining finite automata using a weighted majority algorithm and simplifying the result to a neural network.
result Backpropagation is equivalent to a neural network's learning process.
Formula for complex SVD backpropagation developed.
problem No specific problem stated; focuses on complex SVD.
method Back propagation formula for complex SVD developed.
result Back propagation formula for complex SVD created.
Paper visualizes and analyzes deep neural networks for CTR prediction.
problem Interpreting deep neural networks for CTR prediction is challenging.
method Visualizing and analyzing deep neural networks at neuron level, measuring layer-wise performance, and calculating saliency scores.
result Insights into the model's performance and mechanism.
Unified framework for ternary neural networks reduces memory and computation.
problem Training deep neural networks with limited precision and memory.
method Discretization of activations and weights, derivative approximation, and state transition constraints.
result Ternary networks can be reduced to sparse binary networks, termed GXNOR-Nets.
Visualizes ConvNets without confounding effects.
problem Misinterpretation of saliency maps due to confounding variables.
method Univariate statistical tests and partial back-propagation to remove confounding effects.
result Visualization of confounder-free saliency maps.