The paper analyzes GANs focusing on Inception Score, label smoothing, gradient vanishing, and -log(D(x)).
problem Improving the quality and diversity of generated images in Generative Adversarial Networks (GANs).
method Mathematical analysis of Inception Score, label smoothing, gradient vanishing, and -log(D(x)) alternative in the context of GANs.
result Detailed mathematical insights into Inception Score, label smoothing, gradient vanishing, and -log(D(x)) alternative.
Channel normalization prevents vanishing gradients in convolutional neural networks.
problem Vanishing gradients in convolutional neural networks during optimization.
method Channel normalization, which centers and normalizes each channel individually.
result Channel normalization avoids vanishing gradients, enabling efficient optimization.
In this paper, we will give a local version of the Hamilton-Ivey type pinching estimate of the gradient shrinking soliton with vanishing Weyl tensor, and then give a complete classification on gradient shrinking solitons with vanishing Weyl tensor.
New classification of gradient steady Ricci solitons with vanishing D-tensor.
problem Classifying gradient steady Ricci solitons with specific properties.
method Extending Cao-Chen's work on Bach-flat gradient Ricci solitons, proving properties for D-flat solitons. result Any n-dimensional complete noncompact gradient steady Ricci soliton with vanishing D-tensor is either Ricci-flat or isometric to the Bryant soliton. Sigmoid-type networks avoid vanishing gradients with regularization and rescaling.
problem Vanishing gradients in sigmoid-type networks.
method Mathematical arguments and two remedies: regularization and rescaling.
result Demonstrates effectiveness of regularization and rescaling in practice.
Analysis shows BN prevents gradient vanishing/explosion in residual networks.
problem Gradient vanishing/explosion problem in residual networks.
method Mathematical analysis of BN and residual network training.
result BN confounds gradient variance, preventing vanishing/explosion.
A new GAN framework GAN-QP avoids gradient vanishing and 1-Lipschitz constraint.
problem Gradient vanishing and 1-Lipschitz constraint in GANs.
method Construct a new GAN framework GAN-QP by eliminating the first step of divergence conversion.
result GAN-QP outperforms WGAN in theory and practice.
Vanishing gradients hinder reinforcement finetuning of language models.
problem Vanishing gradients impede the optimization of language models using reinforcement finetuning.
method The study identifies vanishing gradients as a fundamental optimization obstacle in reinforcement finetuning and proposes an initial supervised finetuning phase to mitigate this issue.
result An initial supervised finetuning phase is crucial for successful reinforcement finetuning of language models, as it helps prevent vanishing gradients and maximizes rewards.
Vanishing nodes cause hidden nodes to behave similarly, complicating deep neural network training.
problem Complicating deep neural network training due to hidden nodes behaving similarly.
method Introduced vanishing node indicator (VNI) to quantify node redundancy and correlated behavior.
result The effective number of nodes vanishes to one as VNI increases, indicating a challenging training scenario.
Paper refines RNN training by analyzing smoothness and attractors.
problem Exploding and vanishing gradients in RNNs.
method Refined concept of exploding gradients using cost function smoothness.
result RNNs need to learn attractors to fully use their power.
TSGO optimizes gradients in tensor networks to avoid vanishing/exploding issues.
problem Gradient vanishing and exploding problems in deep learning models.
method TSGO rotates parameters towards gradient direction in tangent space of normalized state.
result TSGO naturally determines learning rate based on angle between parameters and gradient.
Analysis shows exploding and vanishing gradients in neural nets with ReLU activations.
problem Exploding and vanishing gradients in neural nets with ReLU activations.
method Rigorous analysis of gradient behavior in randomly initialized fully connected networks with ReLU activations.
result Empirical variance of gradients is exponential in beta, the sum of reciprocals of hidden layer widths.
The natural gradient of ELBO vanishes in unconstrained optimization, simplifying learning.
problem The gap between evidence and ELBO has a vanishing natural gradient.
method Analyzes the Fisher-Rao gradient of ELBO and its implications for learning.
result Maximizing ELBO is equivalent to minimizing KL divergence, simplifying learning.
New neural network approach mitigates vanishing/exploding gradients.
problem Vanishing and exploding gradients in neural networks.
method Gaussian-Poincaré normalized functions and orthogonal weight matrices.
result High-dimensional probability theory shows gradients disappear with high probability in wide neural networks.
A new recurrent unit alleviates vanishing gradients for long-term dependencies.
problem Vanishing gradients in recurrent neural networks make long-term dependencies hard to model.
method Proposes a new NRU architecture that avoids saturating activation functions and gates.
result Demonstrates superior performance across various tasks with and without long-term dependencies.
Analyzes self-attention in recurrent networks, proving it mitigates vanishing gradients.
problem Vanishing gradients in recurrent networks when capturing long-term dependencies.
method Formal analysis of self-attention's effect on gradient propagation, proposing a relevancy screening mechanism.
result Self-attention mitigates vanishing gradients in recurrent networks, providing guarantees.
Gradient amplification boosts deep learning model performance without increasing training time.
problem Vanishing gradients in deep neural networks.
method Gradient amplification approach to prevent vanishing gradients and training strategy to enable/disable across epochs.
result Improves performance of deep learning models with reduced training time.
We classify complete gradient Ricci solitons satisfying a fourth-order vanishing condition on the Weyl tensor, improving previously known results. More precisely, we show that any n-dimensional (n≥4) gradient shrinking Ricci soliton with fourth order divergence-free Weyl tensor is either Einstein, or a finite q…
Gradient boosts monomial-order-free basis construction algorithms.
problem Lack of theoretical properties in monomial-order-free basis construction algorithms.
method Exploits gradient to sidestep spurious vanishing, achieve consistent output, and remove redundant bases.
result Proposes methods that equip monomial-order-free algorithms with theoretical properties.
Graph manifolds' simplicial volume approximated by covering volumes.
problem Proving simplicial volume of graph manifolds is zero.
method Approximating simplicial volume of graph manifolds by integral simplicial volumes of their finite coverings.
result Uniform proof of rank, Betti number, and torsion homology gradients for graph manifolds.
Blind Descent avoids gradient issues, using a different learning approach.
problem Gradient issues like exploding and vanishing gradients.
method Does not use gradients to guide learning; instead, it is a more fundamental learning process.
result Gradient descent is a specific case of Blind Descent.
Generative memory model avoids vanishing gradients to robustly retrieve patterns.
problem Robust retrieval of stored patterns in the presence of interference and noise.
method Training a generative distributed memory without explicitly simulating attractor dynamics, using a likelihood-based Lyapunov function.
result The model converges to correct patterns upon iterative retrieval and achieves competitive performance as a memory model and a generative model.
Paper proves properties of minimal hypersurfaces in specific solitons.
problem Characterizing minimal hypersurfaces in shrinking gradient Ricci solitons.
method Analyzes stable minimal hypersurfaces with specific curvature conditions.
result Minimal hypersurfaces in these solitons have zero second fundamental form and normal Ricci curvature.
Solves exploding and vanishing gradient problem in LSTMs.
problem Exploding and vanishing gradient problem in LSTM optimization.
method Introduces a simple stochastic algorithm (h-detach) to prevent suppression of gradient components through the cell state path in LSTM.
result Significant improvements in convergence speed, robustness, and generalization over vanilla LSTM training.
Improved perceptron design mitigates vanishing gradient problem.
problem Vanishing gradient problem in deep multilayer perceptrons.
method Auto-rotating perceptron (ARP) design with dynamic activation region.
result Neural networks with ARP units achieve better learning performance.
Paper analyzes if GANs with adversarial features outperform standard supervised learning.
problem Whether adversarial features improve performance over standard supervised learning.
method Theoretical analysis and empirical risk comparison.
result Supervised learning with adversarial features can outperform sole supervision under certain conditions.
This paper classifies solitons under specific tensor conditions.
problem Classifying solitons under vanishing conditions on the Weyl, Cotton, and Cao-Chen tensors.
method Analyzing complete conformal gradient solitons and using tensor conditions.
result Classification of complete nontrivial locally conformally flat conformal gradient solitons.
Noise causes learning plateaus in neural networks.
problem Plateau phenomena in online learning due to vanishing gradients.
method Analysis of stochastic gradient descent in multi-layer perceptrons.
result Noise induces synchronisation leading to strong plateaus.
A new GAN model α-GAN with tunable loss function addresses gradient vanishing and mode collapse issues.
problem Addressing vanishing gradients and mode collapse in GANs.
method Introduced a tunable GAN α-GAN using a supervised α-loss function. result Holistic understanding of α-GAN related to Arimoto divergence and convergence properties. New findings on shrinking Ricci solitons with vanishing Bach-like tensors.
problem Characterizing gradient shrinking Ricci solitons with vanishing Bach-like tensors.
method Defining and analyzing Bach-like tensors, proving rigidity results, and deriving variational formulas.
result Vanishing Bach-like tensors force solitons to be either Einstein or isometric to the Gaussian soliton.
Maxout networks study gradients and propose initialization strategies.
problem Complexity in input-output Jacobian distribution complicates stable parameter initialization.
method Obtained bounds on moments of gradients and formulated initialization strategies.
result Parameter initialization strategies improve training of deep maxout networks.
Batch-normalized RHN improves gradient control in recurrent networks.
problem Gradient vanishing or exploding in recurrent networks.
method Batch normalization applied at each recurrence loop in RHN.
result Batch-normalized RHN converges faster and performs better.
LSTMs are explained as a weighted sum of context-independent functions.
problem Vanishing gradients in simple RNNs.
method Decouple LSTM gates from simple RNNs, computing an element-wise weighted sum of context-independent functions.
result Gating mechanism alone performs as well as an LSTM in most settings, suggesting more representational power.
Spectral normalization stabilizes GANs by controlling gradient explosion and vanishing.
problem Stability and sample quality issues in GAN training.
method Spectral normalization controls gradient explosion and vanishing, improving GAN training stability and sample quality.
result Bidirectional Scaled Spectral Normalization (BSSN) outperforms standard spectral normalization in sample quality and training stability.
New neural networks use distance measures and RMSProp for better training.
problem Vanishing gradient problem in deep neural networks.
method Distance measure-based neural networks with RMSProp optimization.
result RMSProp enables efficient learning of multi-layer neural networks.
Infinitesimal gradient boosting is a new algorithm derived from gradient boosting.
problem Improving the efficiency and smoothness of gradient boosting.
method Introduced a new class of randomized regression trees and used a limit process in vanishing-learning-rate asymptotic.
result Convergence of the stochastic algorithm and characterization of the limiting procedure as a unique solution of a nonlinear ODE.
Quantized Decentralized Gradient Descent (QDGD) solves distributed optimization with quantized communications.
problem Minimizing the sum of smooth and strongly convex functions over a network of distributed agents with quantized communications.
method Proposes QDGD algorithm combining quantized and local information for decentralized gradient descent.
result Achieves vanishing mean solution error under strong convexity and smoothness assumptions.
Gradient penalty improves GAN performance by inducing a large-margin classifier.
problem Improving GAN performance and addressing vanishing gradients.
method A unifying framework of expected margin maximization, showing gradient penalties induce large-margin classifiers.
result Gradient penalties reduce vanishing gradients and produce better generated outputs.
Study the properties of SGD in non-vanishing learning rate regime.
problem Understanding the noise and fluctuation in SGD with finite learning rates.
method Derive exact solvable results for discrete-time SGD in quadratic loss functions.
result Fluctuation caused by discrete-time dynamics is larger than continuous-time theory predicts.
High-order RNNs reduce speech recognition errors.
problem Vanishing gradients in RNNs.
method High-order RNNs with multiple connections from previous time steps.
result HORNNs reduce WER by 4.2% and 6.3% over RNNs.
We show that the only complete shrinking gradient Ricci solitons with vanishing Weyl tensor are quotients of the standard ones. This gives a new proof of the Hamilton-Ivey-Perel'man classification of 3-dimensional shrinking gradient solitons. We also prove a classification for expanding gradient Ricci solitons with con…
Improved GNN handles long-range dependencies in multi-relational graphs.
problem Vanishing gradients in GNNs for multi-relational graphs.
method Proposes a Gated Graph Neural Network with improved long-range dependency handling.
result Outperforms popular GNN models in synthetic tasks.
Study fluid spacetimes, proving shear-free implies vanishing expansion or vorticity.
problem Understanding shear and vorticity in perfect-fluid spacetimes.
method Analyzing perfect-fluid spacetimes using Weyl tensor and divergence.
result Proves shear-free implies vanishing expansion or vorticity for perfect fluids.
ERNNs evolve hidden states on an ODE's equilibrium manifold to mitigate vanishing and exploding gradients.
problem Vanishing and exploding gradients in RNNs.
method Develop a novel family of RNNs (ERNNs) that evolve hidden states on the equilibrium manifold of an ODE.
result ERNNs achieve state-of-the-art accuracy with 3-10x speedups and 1.5-3x model size reduction.
The paper studies topological properties of Ricci shrinkers using weighted L2 cohomology.
problem Proving topological results for smooth gradient Ricci shrinkers.
method Weighted L2 cohomology and extensions to mean curvature flow self-shrinkers. result Establishes upper bounds for Betti numbers, vanishing theorem for cohomology, and dichotomy for ends.
Stagewise boosting improves gradient boosting for distributional regression.
problem Vanishing gradient in gradient boosting for distributional regression leads to suboptimal models.
method Proposes a stagewise boosting-type algorithm for distributional regression, combining stagewise regression ideas with gradient boosting and incorporating a novel regularization method, correlation filtering.
result The proposed algorithm provides better results, especially for complex distributions, by reducing the risk of being trapped in a local optimum.
In this paper, we introduce the concept of quasi Yamabe gradient solitons, which generalizes the concept of Yamabe gradient solitons. By using some ideas in [7,8], we prove that n-dimensional (n≥3) complete quasi Yamabe gradient solitons with vanishing Weyl curvature tensor and positive sectional curvature must …
ADMMiRNN solves RNN training issues with stable convergence.
problem Training RNN with stable convergence and avoiding gradient issues.
method Built ADMMiRNN framework on unfolded RNN, providing novel update rules and theoretical analysis.
result ADMMiRNN achieves convergent results and outperforms baselines.