LayerNorm improves training and generalization by re-centering and re-scaling backward gradients.
problem Understanding the effectiveness of LayerNorm and addressing over-fitting.
method Re-centering and re-scaling backward gradients; proposing AdaNorm.
result LayerNorm without bias and gain performs better than LayerNorm on most datasets.