Batch normalization in the last layer reduces sharpness in wide neural networks.
problem Pathological sharpness in wide neural networks.
method Quantifying the geometry of the parameter space using Fisher information matrix and analyzing deep neural networks with random initialization.
result Batch normalization in the last layer significantly decreases pathological sharpness under specific conditions.
Batch Normalization (BN) is a common technique used to speed-up and stabilize training. On the other hand, the learnable parameters of BN are commonly used in conditional Generative Adversarial Networks (cGANs) for representing class-specific information using conditional Batch Normalization (cBN). In this paper we pro…
Differentiable sorting and rank normalization are incompatible, with specific conditions for admissibility.
problem Incompatibility between differentiable sorting and rank normalization.
method Formalized admissibility through monotone invariance, batch independence, and rank-space stability conditions.
result Different gap-sensitive and batchwise relaxations of rank normalization violate the conditions for admissibility.
The study explains delayed spikes in batch-normalized models.
problem Delayed spikes in batch-normalized models.
method Analyzing batch-normalized linear models, deriving conditions for delayed onset and waiting time.
result Explicit conditions for delayed-onset and waiting time in whitened square-loss linear regression.
BN refines local partition geometry in piecewise-affine networks during training.
problem Understanding the effect of BN on the function realized during training in piecewise-affine networks.
method Analyzing the geometry of switching hyperplanes and affine-region partition conditioned on a mini-batch.
result BN increases expected local partition refinement in ReLU and piecewise-affine networks.
Study shows how network width affects SGD hyper-parameters and generalization.
problem Understanding how network width impacts SGD hyper-parameters and generalization.
method Generated model families by increasing network width, performed hyper-parameter search.
result Wider networks achieve higher test accuracy and optimal normalized noise scale.
A new method normalizes activations to match batch normalization without batch dependence.
problem Performance degradation with batch-independent normalization techniques.
method Proxy-Normalizing Activations
result Proxy-Normalization technique emulates batch normalization's behavior and performance.
Four improvements to Batch Normalization improve deep learning performance.
problem Improving Batch Normalization for better deep learning performance.
method Proposed improvements include reasoning about current examples, Ghost Batch Normalization, weight decay regularization, and a new normalization algorithm for small batch sizes.
result Performance gains across all batch sizes with no additional computation during training.
Online Normalization normalizes neural network activations without batching for better accuracy.
problem Theoretical limitations of Batch Normalization and its inapplicability to certain network types.
method Introduces an unbiased gradient computation technique for normalized activations without using batches.
result Equivalent accuracy to Batch Normalization without batch usage.
Batch normalization prevents rank collapse in deep networks, improving training stability.
problem Rank collapse in randomly initialized deep networks with increasing depth.
method Investigates spectral instabilities in random matrices and uses batch normalization to avoid rank collapse.
result Batch normalization prevents rank collapse in both linear and ReLU networks, improving training stability.
This paper reveals periodic behavior in neural network training with BN and weight decay.
problem Understanding the dynamics of neural network training with BN and weight decay.
method Rigorous investigation of empirical and theoretical mechanisms.
result Periodic behavior in training is a generalization of previously opposing perspectives.
CBN improves batch normalization for small mini-batch sizes.
problem Reduced effectiveness of Batch Normalization in small mini-batch sizes.
method CBN uses statistics from multiple recent iterations, compensating for network weight changes via Taylor polynomials.
result CBN outperforms original batch normalization and direct iteration statistics in object detection and image classification.
FRN layer eliminates batch dependence in deep learning, improving performance across various tasks.
problem Batch Normalization's dependency on mini-batch elements can degrade performance for small batches.
method Filter Response Normalization (FRN) operates independently on each activation channel of each batch element.
result FRN layer outperforms BN and other alternatives in various settings for all batch sizes.
Batch normalization biases linear models towards uniform margins, improving performance in binary classification.
problem Understanding the implicit bias of batch normalization in linear models and neural networks.
method Analyzing gradient descent convergence on linear models and two-layer CNNs with batch normalization.
result Gradient descent with batch normalization in linear models converges to a uniform margin classifier with an exponential convergence rate.
TaskNorm improves meta-learning performance by rethinking batch normalization.
problem Challenges in batch normalization for meta-learning with deep networks.
method Developed TaskNorm, a novel approach to batch normalization for meta-learning.
result TaskNorm consistently improves meta-learning performance across various datasets and meta-learning approaches.
Paper proposes MABN to stabilize BN in small batch sizes.
problem Weakness of BN in small batch sizes limits its use in tasks with small batch sizes.
method Identifies and addresses two extra batch statistics in BN's backward propagation.
result MABN completely restores BN's performance in small batch sizes without additional nonlinear operations.
Batch normalization makes deep residual networks train faster.
problem Training deep residual networks with large depths.
method Downscaling the residual branch by a normalizing factor early in training.
result Normalized residual blocks compute functions close to the identity function early in training.
This paper explores orthonormalization layers as alternatives to batch normalization.
problem The need for alternatives to batch normalization.
method Investigates orthonormalization layers as an alternative to batch normalization.
result Compared the performance of orthonormalization layers with batch normalization.
Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute…
Batch normalization makes neural networks more vulnerable to small adversarial perturbations.
problem Adversarial vulnerability of neural networks trained with batch normalization.
method Investigated the impact of batch normalization on adversarial robustness and compared it to weight decay.
result Substituting weight decay for batch norm nullifies adversarial vulnerability.
Improves deep networks' performance by applying Bayesian learning to batch normalization.
problem Improving generalization performance of deep networks.
method Bayesian learning applied to deterministic normalization techniques.
result Significant improvement in generalization performance compared to traditional methods.
In this work, we investigate Batch Normalization technique and propose its probabilistic interpretation. We propose a probabilistic model and show that Batch Normalization maximazes the lower bound of its marginalized log-likelihood. Then, according to the new probabilistic model, we design an algorithm which acts cons…
Proposes a novel framework for unsupervised domain adaptation using specialized batch normalization.
problem Improves unsupervised domain adaptation in deep neural networks.
method Integrates domain-specific batch normalization layers in convolutional neural networks, estimating pseudo-labels for target domain examples and learning final models with multi-task classification loss.
result Achieves state-of-the-art accuracy in standard and multi-source domain adaptation scenarios.
Batch uniformization improves anomaly detection in sound data.
problem Anomaly scores for rare and frequent normal sounds are not uniform.
method Propose batch uniformization to minimize anomaly scores by weighting samples based on their density.
result Improves performance of unsupervised anomaly detection in sound data.
Zero-shot anomaly detection method using batch normalization.
problem Adapting anomaly detectors to new normal data distributions without training data.
method Adaptive Centered Representations (ACR) with batch normalization.
result First zero-shot AD results for tabular data and image data.
New method uses batch normalization to improve OoD detection.
problem Out-of-distribution samples are not reliably detected by generative models.
method Proposes exploiting in-batch dependencies for OoD detection.
result Empirical results show improved robustness for high-dimensional images.
Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies. However, they are computationally expensive to train and difficult to parallelize. Recent work has shown that normalizing intermediate representations of neural networks can significantly im…
A new loss function speeds up deep neural network training.
problem Training deep neural networks is slow and dependent on batch size.
method Introducing a standardization loss as a secondary objective.
result The standardization loss accelerates training on various architectures and batch sizes.
Normalization techniques such as Batch Normalization have been applied successfully for training deep neural networks. Yet, despite its apparent empirical benefits, the reasons behind the success of Batch Normalization are mostly hypothetical. We here aim to provide a more thorough theoretical understanding from a clas…
This research investigates if deep neural networks can be trained without batch normalization.
problem Training deep neural networks efficiently without batch normalization.
method Detailed study of batch normalization, comparison with other methods, and adaptation of training process.
result It is possible to train deep neural networks effectively without batch normalization.
As an indispensable component, Batch Normalization (BN) has successfully improved the training of deep neural networks (DNNs) with mini-batches, by normalizing the distribution of the internal representation for each hidden layer. However, the effectiveness of BN would diminish with scenario of micro-batch (e.g., less …
ConvS2S-VC converts voice characteristics and pitch contour using a fully convolutional seq2seq model.
problem Voice conversion with preservation of pitch contour and duration.
method Fully convolutional seq2seq architecture with conditional batch normalization.
result ConvS2S-VC outperforms baseline methods in sound quality and speaker similarity.
IEBN normalizes noise by enhancing instance-specific information, improving deep learning performance.
problem Improving deep learning performance by regulating noise in batch normalization.
method Integrates self-attention mechanism to recalibrate channel information in BN.
result IEBN outperforms BN with improved generalization and stability.
Develops methods for reliable inference on batched bandit data.
problem Need for reliable inference methods based on adaptively-collected data from bandit algorithms.
method Introduces Batched OLS (BOLS) estimator for reliable inference on bandit data.
result BOLS is asymptotically normal and robust to non-stationarity in the baseline reward.
New method trains deep ResNets without normalization, achieving state-of-the-art performance.
problem Training deep ResNets without normalization layers leads to instability and lower accuracy.
method Adaptive gradient clipping and Normalizer-Free ResNets design.
result Normalizer-Free ResNets achieve 86.5% top-1 accuracy on ImageNet, matching EfficientNet-B7.
Finet uses FBN for efficient, lightweight neural networks.
problem Building efficient neural networks with limited computational resources.
method Introduces Fine-grained Batch Normalization (FBN) and a novel light-weight network (Finet) that combines FBN with standard convolution.
result Finet achieves state-of-the-art performance on ImageNet classification with reduced computational complexity.
Proposes group whitening to enhance deep learning models' performance.
problem Improving learning efficiency and representational capacity in deep learning models.
method Group Whitening (GW) combines whitening and group normalization to balance these aspects.
result Group Whitening consistently improves model performance across different architectures and benchmarks.
BYOL learns useful representations without batch statistics.
problem BYOL's reliance on batch statistics for representation learning.
method Training BYOL without batch normalization and using alternative normalization schemes.
result BYOL performance comparable to vanilla BYOL without batch normalization.
Gradient control plays an important role in feed-forward networks applied to various computer vision tasks. Previous work has shown that Recurrent Highway Networks minimize the problem of vanishing or exploding gradients. They achieve this by setting the eigenvalues of the temporal Jacobian to 1 across the time steps. …
New Convolutional Unit improves Batch Whitening performance.
problem Improving the efficiency and effectiveness of Batch Whitening.
method Proposes a new Convolutional Unit that aligns with Batch Whitening theory and empirically analyzes the original Convolutional Unit.
result Significantly improved performance on multiple image classification datasets.
Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural n…
Deep Neural Networks (DNNs) thrive in recent years in which Batch Normalization (BN) plays an indispensable role. However, it has been observed that BN is costly due to the reduction operations. In this paper, we propose alleviating this problem through sampling only a small fraction of data for normalization at each i…
This paper explains how batch normalization auto-tunes the regularization parameter based on data statistics.
problem Batch normalization accelerates deep learning training but the exact relationship to regularization is unclear.
method Theoretical analysis and empirical validation of batch normalization's role in auto-tuning the regularization parameter.
result Batch normalization auto-tunes the regularization parameter based on data statistics.
Analysis shows BN prevents gradient vanishing/explosion in residual networks.
problem Gradient vanishing/explosion problem in residual networks.
method Mathematical analysis of BN and residual network training.
result BN confounds gradient variance, preventing vanishing/explosion.
New normalization technique balances positive and negative weights for faster convergence.
problem Balancing positive and negative weights for faster convergence.
method Transformation of layer weights instead of outputs, balancing positive and negative contributions.
result Balanced normalization leads to faster convergence on standard benchmarks.
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …
Novel approach characterizes deep neural networks at initialization.
problem Characterizing the behavior of deep neural networks at initialization.
method A novel approach considering the evolution of statistical moments of signal and noise.
result Established that skip-connections in residual networks lead to well-behaved moments and no pathology.
Batch normalization makes deep neural networks' representations increasingly orthogonal.
problem Orthogonality of deep neural network representations.
method Random linear transformations in successive batch-normalizations.
result Orthogonality of representations improves SGD performance.