arXiv research

This paper analyzes how normalization layers improve neural network training.

We improve neural network explainability by bypassing batch normalization.

Layer normalization improves federated learning with skewed labels.

Fine-tuning normalization layers can reconstruct smaller networks.

Layer normalization with activations prevents Gram matrix rank collapse at initialization.

Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute…

Normalization layers control deep neural network capacity, improving stability and generalization.

PLN-Nets with two linear layers and parallel LN achieve universal approximation.

A new method normalizes activations to match batch normalization without batch dependence.

A method predicts GNS of transformer layers using normalization layer norms.

We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output balanced. We validate our method on a set o…

Normalization layers improve the accuracy of Differentially Private training of deep neural networks.

Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural n…

Normalization methods play an important role in enhancing the performance of deep learning while their theoretical understandings have been limited. To theoretically elucidate the effectiveness of normalization, we quantify the geometry of the parameter space determined by the Fisher information matrix (FIM), which als…

ButterflyFlow uses butterfly matrices for efficient invertible layers in normalizing flows.

EDAIN layer normalizes time series data for neural networks, improving model performance.

Layer normalization placement affects training stability and warm-up stage necessity.

CW normalizes and decorrelates neural network layers for better concept understanding.

Automatically designs normalization and activation layers together.

FRN layer eliminates batch dependence in deep learning, improving performance across various tasks.

Batch normalization has become ubiquitous in many state-of-the-art nets. It accelerates training and yields good performance results. However, there are various other alternatives to normalization, e.g. orthonormalization. The objective of this paper is to explore the possible alternatives to channel normalization with…

Batch normalization biases linear models towards uniform margins, improving performance in binary classification.

Proposes BN layers for neural networks on complex domains, improving training stability and accuracy.

In this work, we propose a novel technique to boost training efficiency of a neural network. Our work is based on an excellent idea that whitening the inputs of neural networks can achieve a fast convergence speed. Given the well-known fact that independent components must be whitened, we introduce a novel Independent-…

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change o…

Stochastic gradient descent (SGD) has been the dominant optimization method for training deep neural networks due to its many desirable properties. One of the more remarkable and least understood quality of SGD is that it generalizes relatively well on unseen data even when the neural network has millions of parameters…

We propose a simple but effective multi-source domain generalization technique based on deep neural networks by incorporating optimized normalization layers that are specific to individual domains. Our approach employs multiple normalization methods while learning separate affine parameters per domain. For each domain,…

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …

Improved sample efficiency with normalized RBF kernels in neural networks.

A new layer, funnel, reduces dimensionality in flows for better performance.

This paper uses a geometric approach to understand how normalization layers affect neural network optimization.

Normalization effects on deep neural networks impact output variance and test accuracy.

We propose a novel unsupervised domain adaptation framework based on domain-specific batch normalization in deep neural networks. We aim to adapt to both domains by specializing batch normalization layers in convolutional neural networks while allowing them to share all other model parameters, which is realized by a tw…

Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimiz…

Proposes a new normalization method using convolutional neural networks.

FF algorithm uses goodness as a likelihood-ratio test for scalar normalization.

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- \textit{Internal Covariate Shift}-- the current solution has certain drawbacks. For instance, BN depends on batch statistics for layerwise input normalization during training which makes the esti…

As an indispensable component, Batch Normalization (BN) has successfully improved the training of deep neural networks (DNNs) with mini-batches, by normalizing the distribution of the internal representation for each hidden layer. However, the effectiveness of BN would diminish with scenario of micro-batch (e.g., less …

LayerNorm improves training and generalization by re-centering and re-scaling backward gradients.

A new method speeds up spectral normalization for neural nets.

Normalizing flows are shown to be equivalent to Bayesian networks, revealing new insights.

Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge…

In this paper, we propose a new pooling method called spatial pyramid encoding (SPE) to generate speaker embeddings for text-independent speaker verification. We first partition the output feature maps from a deep residual network (ResNet) into increasingly fine sub-regions and extract speaker embeddings from each sub-…

Bayesian neural networks improve uncertainty calibration without sacrificing accuracy.

Neural collapse occurs in normalized features over a Riemannian manifold.

EMFs combine deep learning and probabilistic models for better density estimation.

We characterize convolutional neural networks with respect to the relative amount of features per layer. Using a skew normal distribution as a parametrized framework, we investigate the common assumption of monotonously increasing feature-counts with higher layers of architecture designs. Our evaluation on models with …

This work presents a novel approach to train invertible linear layers by adding rank-one perturbations.