arXiv research

Paper refines RNN training by analyzing smoothness and attractors.

New neural network approach mitigates vanishing/exploding gradients.

TSGO optimizes gradients in tensor networks to avoid vanishing/exploding issues.

Blind Descent avoids gradient issues, using a different learning approach.

Vanishing nodes cause hidden nodes to behave similarly, complicating deep neural network training.

Maxout networks study gradients and propose initialization strategies.

ERNNs evolve hidden states on an ODE's equilibrium manifold to mitigate vanishing and exploding gradients.

Volume-preserving neural networks prevent gradient issues.

ADMMiRNN solves RNN training issues with stable convergence.

A new RNN model tackles long-time dependencies with fast, invertible, and memory-efficient hidden states.

Channel normalization prevents vanishing gradients in convolutional neural networks.

ENRNN uses eigenvalue normalization for short-term memory in RNNs.

A simple gating mechanism improves deep learning convergence.

Non-normal RNNs outperform orthogonal ones in sequential tasks.

Hamiltonian RNN controls hidden states gradient for long-term dependencies.

Novel BSG method for efficient stochastic optimization.

Gradient flossing stabilizes RNN training by controlling Lyapunov exponents.

New RNN model handles long-term dependencies in irregularly-sampled time series.

Stable ResNet stabilizes gradients in deep networks.

Vanishing and exploding gradients are two of the main obstacles in training deep neural networks, especially in capturing long range dependencies in recurrent neural networks~(RNNs). In this paper, we present an efficient parametrization of the transition matrix of an RNN that allows us to stabilize the gradients that …

This paper analyzes challenges and solutions in deep learning optimization.

Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), and Memory Networks which contain memory are popularly used to learn patterns in sequential data. Sequential data has long sequences that hold relationships. RNN can handle long sequences but suffers from the vanishing and exploding gradient probl…

Scaling ResNets requires careful consideration of the layer depth and output scaling factors.

We give a rigorous analysis of the statistical behavior of gradients in a randomly initialized fully connected network N with ReLU activations. Our results show that the empirical variance of the squares of the entries in the input-output Jacobian of N is exponential in a simple architecture-dependent constant beta, gi…

Spectral normalization stabilizes GANs by controlling gradient explosion and vanishing.

Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that good…

New method bounds singular values of convolutional kernels to stabilize gradients.

Gradient control plays an important role in feed-forward networks applied to various computer vision tasks. Previous work has shown that Recurrent Highway Networks minimize the problem of vanishing or exploding gradients. They achieve this by setting the eigenvalues of the temporal Jacobian to 1 across the time steps. …

Novel method improves training RNNs by accelerating gradient descent.

The vanishing gradient problem was a major obstacle for the success of deep learning. In recent years it was gradually alleviated through multiple different techniques. However the problem was not really overcome in a fundamental way, since it is inherent to neural networks with activation functions based on dot produc…

URNNs are as expressive as general RNNs with ReLU activations.

RNNs struggle with chaotic dynamics due to exploding gradients, but we found a way to optimize training.

Deep neural networks' Jacobian spectrum becomes well-conditioned with orthogonal weights.

A novel RNN model with shuffled hidden states.

Monte Carlo method trains deep neural networks without gradients.

A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than …

Proposes a method to constrain singular values of convolutional kernels in neural networks.

It is a known fact that training recurrent neural networks for tasks that have long term dependencies is challenging. One of the main reasons is the vanishing or exploding gradient problem, which prevents gradient information from propagating to early layers. In this paper we propose a simple recurrent architecture, th…

Guarantees for tuning step size using meta-gradient descent.

Proposes methods to add constraints to neural networks to improve stability and generalization.

pLSTM tackles long-range language modeling and computer vision tasks with parallelizable linear source transition mark networks.

HSIC bottleneck trains deep networks without backpropagation.

A new RNN model based on coupled oscillators mitigates gradient issues.

Paper proves multiplicative weight updates can train neural networks without learning rate tuning.

BatchNorm helps BNNs avoid exploding gradients during training.

LEM efficiently models long-term sequences with gradients.

This work tackles exploding inverses in INNs, revealing and mitigating their numerical non-invertibility.

Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately …