Paper refines RNN training by analyzing smoothness and attractors.
problem Exploding and vanishing gradients in RNNs.
method Refined concept of exploding gradients using cost function smoothness.
result RNNs need to learn attractors to fully use their power.
New neural network approach mitigates vanishing/exploding gradients.
problem Vanishing and exploding gradients in neural networks.
method Gaussian-Poincaré normalized functions and orthogonal weight matrices.
result High-dimensional probability theory shows gradients disappear with high probability in wide neural networks.
TSGO optimizes gradients in tensor networks to avoid vanishing/exploding issues.
problem Gradient vanishing and exploding problems in deep learning models.
method TSGO rotates parameters towards gradient direction in tangent space of normalized state.
result TSGO naturally determines learning rate based on angle between parameters and gradient.
Analysis shows exploding and vanishing gradients in neural nets with ReLU activations.
problem Exploding and vanishing gradients in neural nets with ReLU activations.
method Rigorous analysis of gradient behavior in randomly initialized fully connected networks with ReLU activations.
result Empirical variance of gradients is exponential in beta, the sum of reciprocals of hidden layer widths.
Vanishing nodes cause hidden nodes to behave similarly, complicating deep neural network training.
problem Complicating deep neural network training due to hidden nodes behaving similarly.
method Introduced vanishing node indicator (VNI) to quantify node redundancy and correlated behavior.
result The effective number of nodes vanishes to one as VNI increases, indicating a challenging training scenario.
Solves exploding and vanishing gradient problem in LSTMs.
problem Exploding and vanishing gradient problem in LSTM optimization.
method Introduces a simple stochastic algorithm (h-detach) to prevent suppression of gradient components through the cell state path in LSTM.
result Significant improvements in convergence speed, robustness, and generalization over vanilla LSTM training.
Blind Descent avoids gradient issues, using a different learning approach.
problem Gradient issues like exploding and vanishing gradients.
method Does not use gradients to guide learning; instead, it is a more fundamental learning process.
result Gradient descent is a specific case of Blind Descent.
ERNNs evolve hidden states on an ODE's equilibrium manifold to mitigate vanishing and exploding gradients.
problem Vanishing and exploding gradients in RNNs.
method Develop a novel family of RNNs (ERNNs) that evolve hidden states on the equilibrium manifold of an ODE.
result ERNNs achieve state-of-the-art accuracy with 3-10x speedups and 1.5-3x model size reduction.
LTM tackles long sequence language modeling by avoiding vanishing and exploding gradients.
problem Language models struggle with long sequences due to gradient issues.
method Introduces Long Term Memory network (LTM) that scales memory and weights input, avoiding overfitting.
result LTM achieves state-of-the-art perplexity results with fewer cells than previous models.
Maxout networks study gradients and propose initialization strategies.
problem Complexity in input-output Jacobian distribution complicates stable parameter initialization.
method Obtained bounds on moments of gradients and formulated initialization strategies.
result Parameter initialization strategies improve training of deep maxout networks.
Batch-normalized RHN improves gradient control in recurrent networks.
problem Gradient vanishing or exploding in recurrent networks.
method Batch normalization applied at each recurrence loop in RHN.
result Batch-normalized RHN converges faster and performs better.
Volume-preserving neural networks prevent gradient issues.
problem Vanishing and exploding gradients in deep neural networks.
method A new neural network architecture with volume-preserving sublayers.
result Volume-preserving neural networks maintain gradient stability.
A new RNN model tackles long-time dependencies with fast, invertible, and memory-efficient hidden states.
problem Challenges in processing sequential inputs with long-time dependencies in RNNs.
method A novel RNN architecture based on a Hamiltonian system of oscillators.
result The proposed RNN mitigates exploding and vanishing gradient problems, providing state-of-the-art performance.
ADMMiRNN solves RNN training issues with stable convergence.
problem Training RNN with stable convergence and avoiding gradient issues.
method Built ADMMiRNN framework on unfolded RNN, providing novel update rules and theoretical analysis.
result ADMMiRNN achieves convergent results and outperforms baselines.
Fixup replaces normalization in deep networks, achieving similar stability and performance.
problem The effectiveness of normalization layers in deep neural networks.
method Fixed-update initialization (Fixup) to solve exploding and vanishing gradient problems.
result Residual networks trained with Fixup achieve state-of-the-art performance without normalization.
AntisymmetricRNN improves RNN trainability without extra computation.
problem Difficulty in learning long-term dependencies in RNNs.
method Connecting RNNs to ordinary differential equations and proposing AntisymmetricRNN.
result AntisymmetricRNN captures long-term dependencies more predictably and efficiently.
Non-normal RNNs outperform orthogonal ones in sequential tasks.
problem Vanishing/exploding gradients in RNNs training.
method Investigate non-normal RNNs with non-normal recurrent connectivity matrix.
result Non-normal RNNs outperform orthogonal ones in various benchmarks.
ENRNN uses eigenvalue normalization for short-term memory in RNNs.
problem Vanishing/exploding gradient problem and long-term dependency modeling.
method Eigenvalue normalization of recurrent matrix to simulate short-term memory.
result ENRNN outperforms existing RNN variants in experiments.
A simple gating mechanism improves deep learning convergence.
problem Vanishing or exploding gradients in deep networks.
method Introducing a zero-initialized parameter to each residual connection.
result Training deep networks (up to 120 layers) with fast convergence and better performance.
Analysis shows BN prevents gradient vanishing/explosion in residual networks.
problem Gradient vanishing/explosion problem in residual networks.
method Mathematical analysis of BN and residual network training.
result BN confounds gradient variance, preventing vanishing/explosion.
Channel normalization prevents vanishing gradients in convolutional neural networks.
problem Vanishing gradients in convolutional neural networks during optimization.
method Channel normalization, which centers and normalizes each channel individually.
result Channel normalization avoids vanishing gradients, enabling efficient optimization.
Study on preventing early training failure modes in deep ReLU nets.
problem Early training failure modes in deep ReLU nets.
method Proved and avoided two failure modes: exploding/vanishing mean activation length and exponentially large variance of activation length.
result Correct initialization and architecture can prevent early training failure modes in deep ReLU nets.
LRA trains deep networks robustly with less sensitivity to initial weights.
problem Training deep networks is challenging due to issues like exploding and vanishing gradients.
method Local Representation Alignment (LRA) is a training procedure less sensitive to initial weights.
result LRA can train networks robustly, even with null initial weights, and outperforms other methods.
New RNN model handles long-term dependencies in irregularly-sampled time series.
problem Handling long-term dependencies in irregularly-sampled time series data.
method Designing ODE-LSTMs that separate memory from continuous-time state.
result ODE-LSTMs outperform other RNN-based models on non-uniformly sampled data with long-term dependencies.
Hamiltonian RNN controls hidden states gradient for long-term dependencies.
problem Challenges in learning long-term dependencies in RNNs.
method Symplectic discretization of Hamiltonian system to control gradient.
result Hamiltonian RNN outperforms other RNNs without hyperparameter optimization.
New method optimizes deep neural network weights using data statistics.
problem Vanishing/exploding gradient problem and non-convex objective function in DNNs.
method Data-dependent initialization of network weights.
result Our algorithm achieves superior classification accuracy on practical datasets.
Deep CNNs can be trained without special architectures.
problem Training extremely deep CNNs (10,000 layers) is challenging due to vanishing/exploding gradients.
method Developed a mean field theory for signal propagation and conditions for dynamical isometry. Derived an algorithm for generating random initial orthogonal convolution kernels.
result Vanilla CNNs with ten thousand layers can be efficiently trained using appropriate initialization schemes.
Scaling ResNets requires careful consideration of the layer depth and output scaling factors.
problem Avoiding vanishing or exploding gradients in deep ResNets as depth increases.
method Probabilistic analysis and continuous-time limit interpretation of ResNets.
result The optimal scaling factor is α L = 1 L α_L = \frac{1}{\sqrt{L}} α L = L 1 for standard i.i.d. initializations. Novel BSG method for efficient stochastic optimization.
problem Efficient optimization of non-convex surfaces in stochastic settings.
method Binary search combined with first order gradient optimization.
result BSG produces more promising results and better generalization than other methods.
Deep neural networks' Jacobian spectrum becomes well-conditioned with orthogonal weights.
problem Understanding and handling the Jacobian spectrum of deep neural networks.
method Applying free probability theory to show almost sure asymptotic freeness of Jacobians in the wide limit.
result Layer-wise Jacobians of deep neural networks with orthogonal weights are almost surely asymptotically free.
This paper extends de Rham theory of smooth manifolds to exploded manifolds. Included are versions of Stokes' theorem, De Rham cohomology, Poincare duality, and integration along the fiber. The resulting cohomology theory is used to define Gromov Witten invariants of exploded manifolds in a separate paper.
Gradient flossing stabilizes RNN training by controlling Lyapunov exponents.
problem Gradient instability in RNNs leading to exploding and vanishing gradients.
method Regularizing Lyapunov exponents through backpropagation using differentiable linear algebra.
result Gradient flossing improves RNN training success rate and convergence speed.
Stable ResNet stabilizes gradients in deep networks.
problem Gradient vanishing and exploding in deep ResNet architectures.
method Introducing Stable ResNet architectures with gradient stabilization and infinite depth expressivity.
result Stable ResNet maintains gradient stability and expressivity in deep networks.
This paper analyzes challenges and solutions in deep learning optimization.
problem Gradient vanishing and exploding issues in deep learning.
method Improvement of gradient flow and constraints on Lipschitz constant.
result Enhanced understanding of Jacobian matrices and Lipschitz constants in deep learning modules.
ResNets are effective due to their insensitivity to weight initialization.
problem Training deep neural networks with vanishing/exploding gradients.
method Simplified analysis of ResNet models, demonstrating the effect of batch normalization.
result ResNets are insensitive to weight initialization, improving training stability.
This paper stabilizes deep neural networks by efficiently parameterizing RNNs with SVD.
problem Vanishing and exploding gradients in deep neural networks, especially RNNs.
method Parameterize RNN transition matrix using SVD and Householder reflectors for efficiency.
result Spectral-RNN method stabilizes gradients and improves generalization, especially for long-range dependencies.
pLSTM tackles long-range language modeling and computer vision tasks with parallelizable linear source transition mark networks.
problem Challenges of existing recurrent architectures in handling sequences and multi-dimensional data.
method Introduces pLSTM, a parallelizable linear source transition mark network for linear graphs and DAGs, addressing vanishing/exploding activation/gradient issues.
result pLSTM outperforms Transformers in long-range tasks like arrow-pointing extrapolation and image size extrapolation.
URNNs are as expressive as general RNNs with ReLU activations.
problem Expressiveness of URNNs compared to general RNNs.
method Input-output equivalence between URNNs and contractive RNNs with ReLU activations.
result URNNs are as expressive as general RNNs with ReLU activations.
Spectral normalization stabilizes GANs by controlling gradient explosion and vanishing.
problem Stability and sample quality issues in GAN training.
method Spectral normalization controls gradient explosion and vanishing, improving GAN training stability and sample quality.
result Bidirectional Scaled Spectral Normalization (BSSN) outperforms standard spectral normalization in sample quality and training stability.
New neural networks use distance measures and RMSProp for better training.
problem Vanishing gradient problem in deep neural networks.
method Distance measure-based neural networks with RMSProp optimization.
result RMSProp enables efficient learning of multi-layer neural networks.
Notes for a short lecture series, covering exploded manifolds, the moduli stack of curves in exploded manifolds, and a tropical gluing formula for Gromov-Witten invariants: a gluing formula providing a degeneration formula for Gromov-Witten invariants in normal-crossing degenerations. I gave the original lecture series…
scoRNN improves RNN performance with simpler orthogonal weight matrices.
problem Vanishing and exploding gradients in RNNs.
method Parametrizing orthogonal recurrent weight matrices with a scaled Cayley transform.
result scoRNN achieves superior results with fewer parameters than other unitary RNNs.
A novel RNN model with shuffled hidden states.
problem Improving recurrent neural network performance.
method Permuting hidden states and applying learned functions.
result Competitive results compared to literature baselines.
Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent work on advancing the state of the art has been focused on the optimization or modelling of RNNs, mostly motivated by adressing the problems of the vanishing and exploding gradients. The control of overfitting has seen consid…
Novel method improves training RNNs by accelerating gradient descent.
problem Vanishing and exploding gradient problems in RNNs training.
method Adaptive stochastic Nesterov accelerated quasi-Newton method.
result Improved performance in training RNNs with low per-iteration cost.
New method bounds singular values of convolutional kernels to stabilize gradients.
problem Stable gradients in convolutional neural networks.
method Frobenius norm regularization for convolutional kernels.
result Bounded singular values of transformation matrices.
HSIC bottleneck trains deep networks without backpropagation.
problem Training deep neural networks with exploding and vanishing gradients.
method HSIC bottleneck, alternative to cross-entropy loss and backpropagation.
result HSIC bottleneck achieves comparable performance to backpropagation.
We present a gluing formula for Gromov-Witten invariants in the case of a triple product. This gluing formula is a simple case of a much more general gluing formula proved and stated using exploded manifolds. We present this simple case because it is relatively easy to explain without any knowledge of exploded manifold…