This paper investigates the impact of normalization on deep neural networks for click-through rate prediction.
problem The effect of normalization on deep neural network models for CTR estimation.
method Systematic study of various normalization approaches applied to feature embedding and MLP part of DNN models.
result Correct normalization significantly enhances model performance, as demonstrated by extensive experiments on real-world datasets.
The paper proposes deep normalization to improve speaker recognition performance.
problem Non-Gaussian and non-homogeneous distributions of deep speaker vectors negatively impact speaker recognition.
method Proposes a deep normalization approach based on a novel discriminative normalization flow (DNF) model.
result DNF-based normalization delivers substantial performance gains and strong generalization capability.
Batch normalization makes deep residual networks train faster.
problem Training deep residual networks with large depths.
method Downscaling the residual branch by a normalizing factor early in training.
result Normalized residual blocks compute functions close to the identity function early in training.
The paper analyzes various normalization methods in deep learning.
problem Lack of mathematical tools to analyze normalization methods.
method Proposed a lemma to define tools, analyzed BN, LN, WN, GN.
result Normalization methods can be unified on a sphere, improving training stability and weight norm.
Normalization layers are widely used in deep neural networks to stabilize training. In this paper, we consider the training of convolutional neural networks with gradient descent on a single training example. This optimization problem arises in recent approaches for solving inverse problems such as the deep image prior…
Normalization layers improve the accuracy of Differentially Private training of deep neural networks.
problem Reduced accuracy in deep neural networks with Differentially Private training.
method Proposed a novel method for integrating batch normalization with Differentially Private Stochastic Gradient Descent (DPSGD) without additional privacy loss.
result Training deeper networks with better utility-privacy trade-off is possible.
Very deep CNNs achieve state-of-the-art results in both computer vision and speech recognition, but are difficult to train. The most popular way to train very deep CNNs is to use shortcut connections (SC) together with batch normalization (BN). Inspired by Self- Normalizing Neural Networks, we propose the self-normaliz…
Normalization layers control deep neural network capacity, improving stability and generalization.
problem Excessive capacity in deep neural networks leads to overfitting and poor generalization.
method Developed a theoretical framework to explain normalization's role in capacity control.
result Normalization layers reduce the Lipschitz constant exponentially, smoothing the loss landscape and enhancing generalization.
Proposes a new normalization method for deep neural networks in financial forecasting.
problem Deep neural networks are sensitive to input variable range and prone to numerical issues, especially with financial time-series.
method Bilinear input normalization method that handles high-frequency financial time-series without expert knowledge.
result Significant improvements in forecasting future stock price dynamics over other normalization techniques.
Batch normalization prevents rank collapse in deep networks, improving training stability.
problem Rank collapse in randomly initialized deep networks with increasing depth.
method Investigates spectral instabilities in random matrices and uses batch normalization to avoid rank collapse.
result Batch normalization prevents rank collapse in both linear and ReLU networks, improving training stability.
This paper analyzes how normalization layers improve neural network training.
problem Improving generalization performance and training speed of neural networks.
method Global convergence analysis of two-layer neural networks with ReLU activations and Weight Normalization.
result Introduction of normalization layers changes the optimization landscape, enabling faster convergence.
This paper presents a general framework for norm-based capacity control for Lp,q weight normalized deep neural networks. We establish the upper bound on the Rademacher complexities of this family. With an Lp,q normalization where q≤p∗, and 1/p+1/p∗=1, we discuss properties of a width-independent ca…
This review compares various deep generative models.
problem Training deep neural networks to model data distributions.
method Comprehensive comparison of VAEs, GANs, flows, energy models, and autoregressives.
result Trade-offs and interrelationships among different models.
GAS-Norm improves deep learning time series forecasting in non-stationary settings.
problem Deep learning models struggle with non-stationary time series data.
method Combines GAS model for adaptive normalization with deep neural networks.
result Improves deep learning performance in 21 out of 25 settings.
This paper explores normalization in neural ODEs, achieving high accuracy in CIFAR-10.
problem Understanding the role of normalization in neural ODEs.
method Investigated different normalization techniques and their impact on neural ODEs performance.
result Achieved 93% accuracy in CIFAR-10 classification task.
This paper reviews normalization techniques for DNNs.
problem Improving training speed and generalization of DNNs.
method Taxonomy of normalization methods, decomposition of activation methods.
result Insight for designing new normalization techniques.
EDAIN layer normalizes time series data for neural networks, improving model performance.
problem Irregularities in time series data degrade model performance in neural networks.
method EDAIN layer learns adaptive normalization parameters during end-to-end training.
result EDAIN layer outperforms conventional normalization methods and adaptive layers.
Enhances DNN robustness and accuracy with L2,∞ normalization.
problem Improving the robustness and accuracy of deep neural networks.
method Introducing L2,∞ normalization of weight matrices in DNNs with Relu activation. result Lower bound for robustness measure in terms of L2,∞ norm and upper bound for Rademacher complexity. We develop a general method for estimating a finite mixture of non-normalized models. Here, a non-normalized model is defined to be a parametric distribution with an intractable normalization constant. Existing methods for estimating non-normalized models without computing the normalization constant are not applicable …
Batch normalization makes deep neural networks' representations increasingly orthogonal.
problem Orthogonality of deep neural network representations.
method Random linear transformations in successive batch-normalizations.
result Orthogonality of representations improves SGD performance.
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …
Proposes a new normalization method using convolutional neural networks.
problem Slow and inefficient training of deep neural networks.
method Uses depth-wise convolutional neural networks to approximate statistics.
result Learned coefficients improve the approximation of statistics.
Zero-shot anomaly detection method using batch normalization.
problem Adapting anomaly detectors to new normal data distributions without training data.
method Adaptive Centered Representations (ACR) with batch normalization.
result First zero-shot AD results for tabular data and image data.
This paper explains how batch normalization auto-tunes the regularization parameter based on data statistics.
problem Batch normalization accelerates deep learning training but the exact relationship to regularization is unclear.
method Theoretical analysis and empirical validation of batch normalization's role in auto-tuning the regularization parameter.
result Batch normalization auto-tunes the regularization parameter based on data statistics.
Lipschitz normalization boosts deep attention models, especially for graph neural networks.
problem Gradient explosion in deep graph attention networks leads to poor performance.
method Enforcing Lipschitz continuity by normalizing attention scores.
result Deep GAT models with LipschitzNorm achieve state-of-the-art results for tasks with long-range dependencies.
We improve neural network explainability by bypassing batch normalization.
problem Lack of transparency in neural networks.
method Layer-wise Relevance Propagation with a method to include normalization layers.
result Heatmaps are more accurate for convolutional layers with our method.
Adaptive feature normalization improves model robustness to extraneous variables.
problem Degrading model performance due to extraneous variables in deep learning.
method Adaptive feature normalization using instance normalization instead of batch normalization.
result Adaptive normalization leads to significant performance gains across different datasets and architectures.
Proposes a method to detect anomalies in multi-subgroup normal data.
problem Anomaly detection with limited labeled anomalies and multi-subgroup normal data.
method Learn multi-normal prototypes with deep embedding clustering and contrastive learning. Estimate the likelihood of unlabeled samples being normal during training.
result Superior performance compared to state-of-the-art methods on various datasets.
In this paper we introduce a novel method of gradient normalization and decay with respect to depth. Our method leverages the simple concept of normalizing all gradients in a deep neural network, and then decaying said gradients with respect to their depth in the network. Our proposed normalization and decay techniques…
Proposes group whitening to enhance deep learning models' performance.
problem Improving learning efficiency and representational capacity in deep learning models.
method Group Whitening (GW) combines whitening and group normalization to balance these aspects.
result Group Whitening consistently improves model performance across different architectures and benchmarks.
Normalization methods play an important role in enhancing the performance of deep learning while their theoretical understandings have been limited. To theoretically elucidate the effectiveness of normalization, we quantify the geometry of the parameter space determined by the Fisher information matrix (FIM), which als…
SAM optimizer benefits from normalization, stabilizing and guiding optimization.
problem Improving deep neural network performance with SAM optimizer.
method Theoretical and empirical study of normalization in SAM for convex and non-convex functions.
result Normalization helps SAM in stabilizing and guiding optimization along a continuum of minima.
Improves anomaly detection with contaminated unlabeled data.
problem Weakness in existing semi-supervised anomaly detection methods when unlabeled data contain anomalies.
method Integrates positive-unlabeled learning with deep anomaly detection models.
result Achieves better detection performance on various datasets.
Layer normalization with activations prevents Gram matrix rank collapse at initialization.
problem Rank collapse in Gram matrices at initialization slows training in deep networks.
method Proved that layer normalization, with activation layers, biases Gram matrix towards identity matrix at exponential rate.
result Layer normalization with activations biases Gram matrix towards identity matrix at exponential rate with depth at initialization.
Proposes a context-aware approach to deep autoencoder novelty detection.
problem Challenges of static distribution in novelty detection.
method Semi-supervised network architecture with auxiliary labels for contextual information.
result Single model achieves performance of individually trained models on various contexts.
Deep convolutional neural networks are known to be unstable during training at high learning rate unless normalization techniques are employed. Normalizing weights or activations allows the use of higher learning rates, resulting in faster convergence and higher test accuracy. Batch normalization requires minibatch sta…
New method trains deep ResNets without normalization, achieving state-of-the-art performance.
problem Training deep ResNets without normalization layers leads to instability and lower accuracy.
method Adaptive gradient clipping and Normalizer-Free ResNets design.
result Normalizer-Free ResNets achieve 86.5% top-1 accuracy on ImageNet, matching EfficientNet-B7.
Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present …
Class Normalization improves zero-shot learning models.
problem Improving zero-shot learning models in a continual setting.
method Class Normalization (CN) technique to address irregular loss surfaces and improve training efficiency.
result CN significantly outperforms state-of-the-art models on 4 standard ZSL datasets.
Normalization methods are a central building block in the deep learning toolbox. They accelerate and stabilize training, while decreasing the dependence on manually tuned learning rate schedules. When learning from multi-modal distributions, the effectiveness of batch normalization (BN), arguably the most prominent nor…
New method trains deep vanilla networks as fast as ResNets without shortcut connections.
problem Training very deep neural networks is challenging.
method Developed a new type of transformation compatible with Leaky ReLUs.
result Validation accuracies with deep vanilla networks are competitive with ResNets and significantly higher.
In this paper, we propose a new pooling method called spatial pyramid encoding (SPE) to generate speaker embeddings for text-independent speaker verification. We first partition the output feature maps from a deep residual network (ResNet) into increasingly fine sub-regions and extract speaker embeddings from each sub-…
NVAE improves VAE performance on large image datasets.
problem Improving variational autoencoder performance for large image datasets.
method Deep hierarchical VAE with depth-wise separable convolutions and batch normalization, residual parameterization of Normal distributions, and spectral regularization.
result NVAE achieves state-of-the-art results on MNIST, CIFAR-10, CelebA 64, and CelebA HQ datasets.
Normalization effects on deep neural networks impact output variance and test accuracy.
problem The impact of normalization on deep neural networks' statistical behavior and test accuracy.
method Asymptotic expansion analysis of neural network's output for different γi values. result Equal γi values (one) provide the best statistical behavior and test accuracy. A significant advance in accelerating neural network training has been the development of normalization methods, permitting the training of deep models both faster and with better accuracy. These advances come with practical challenges: for instance, batch normalization ties the prediction of individual examples with o…
Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets. Typically anomaly detection is treated as an unsupervised learning problem. In practice however, one may have---in addition to a large set of unlabeled samples---access to a small pool of label…
This research investigates if deep neural networks can be trained without batch normalization.
problem Training deep neural networks efficiently without batch normalization.
method Detailed study of batch normalization, comparison with other methods, and adaptation of training process.
result It is possible to train deep neural networks effectively without batch normalization.
Deep learning improves GW signal detection efficiency and robustness.
problem Traditional matched-filtering techniques are limited in detecting new GW signals.
method Optimized CNN models with techniques like batch normalization and dropout.
result CNN models are robust to the variation of GW waveform parameters.