arXiv research

This paper investigates the impact of normalization on deep neural networks for click-through rate prediction.

We propose a simple but effective multi-source domain generalization technique based on deep neural networks by incorporating optimized normalization layers that are specific to individual domains. Our approach employs multiple normalization methods while learning separate affine parameters per domain. For each domain,…

Paper extends LME models to allow sign constraints on coefficients with SDTN random effects.

The choice of normalization affects the coefficients in regularized regression models.

Study on neural networks' performance under different normalizations as N grows.

Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute…

Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regulariza…

A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically improve upon Batch Normalization and to understand the circumstances that lend them…

CBN improves batch normalization for small mini-batch sizes.

Normalization methods are a central building block in the deep learning toolbox. They accelerate and stabilize training, while decreasing the dependence on manually tuned learning rate schedules. When learning from multi-modal distributions, the effectiveness of batch normalization (BN), arguably the most prominent nor…

Globally normalized neural sequence models are considered superior to their locally normalized equivalents because they may ameliorate the effects of label bias. However, when considering high-capacity neural parametrizations that condition on the whole input sequence, both model classes are theoretically equivalent in…

Braids can be represented geometrically as laminations of punctured disks. The geometric complexity of a braid is the minimal complexity of a lamination that represents it, and tight laminations are representatives of minimal complexity. These laminations give rise to a normal form of braids, via a relaxation algorithm…

Normalization layers improve the accuracy of Differentially Private training of deep neural networks.

This research investigates if deep neural networks can be trained without batch normalization.

Deep Learning (DL) models can be used to tackle time series analysis tasks with great success. However, the performance of DL models can degenerate rapidly if the data are not appropriately normalized. This issue is even more apparent when DL is used for financial time series forecasting tasks, where the non-stationary…

Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural n…

Zero-shot anomaly detection method using batch normalization.

Normalization effects on deep neural networks impact output variance and test accuracy.

A new method to improve deep neural networks using weight rescaling.

This paper reviews normalization techniques for DNNs.

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.

Let $M^n$, $n \in \{4,5,6\}$, be a compact, simply connected $n$-manifold which admits some Riemannian metric with non-negative curvature and an isometry group of maximal possible rank. Then any smooth, effective action on $M^n$ by a torus $T^{n-2}$ is equivariantly diffeomorphic to an isometric action on a normal biqu…

GraphNorm accelerates GNN training by adapting InstanceNorm, improving convergence and generalization.

Normalization layers are widely used in deep neural networks to stabilize training. In this paper, we consider the training of convolutional neural networks with gradient descent on a single training example. This optimization problem arises in recent approaches for solving inverse problems such as the deep image prior…

Paper classifies rational 3-tangles using normal forms and minimal coordinates.

Paper introduces new estimator for continuous treatment effects.

Contrast uses normalizing flows to create precise prediction regions for multi-dimensional outputs.

In this paper, we introduce a new sparsity-promoting prior, namely, the "normal product" prior, and develop an efficient algorithm for sparse signal recovery under the Bayesian framework. The normal product distribution is the distribution of a product of two normally distributed variables with zero means and possibly …

The Normalized Mutual Information (NMI) has been widely used to evaluate the accuracy of community detection algorithms. However in this article we show that the NMI is seriously affected by systematic errors due to finite size of networks, and may give a wrong estimate of performance of algorithms in some cases. We gi…

TaskNorm improves meta-learning performance by rethinking batch normalization.

Calculation of the log-normalizer is a major computational obstacle in applications of log-linear models with large output spaces. The problem of fast normalizer computation has therefore attracted significant attention in the theoretical and applied machine learning literature. In this paper, we analyze a recently pro…

Paper proves robust M-estimators' coordinates' normality in high dimensions.

Bayesian Gaussian Process ODEs enhanced with normalizing flows for improved flexibility and accuracy.

Estimates RL data for dynamic treatment effects using GMM.

Batch normalization makes deep residual networks train faster.

We introduce online learning algorithms which are independent of feature scales, proving regret bounds dependent on the ratio of scales existent in the data rather than the absolute scale. This has several useful effects: there is no need to pre-normalize data, the test-time and test-space complexity are reduced, and t…

We introduce online learning algorithms which are independent of feature scales, proving regret bounds dependent on the ratio of scales existent in the data rather than the absolute scale. This has several useful effects: there is no need to pre-normalize data, the test-time and test-space complexity are reduced, and t…

In this paper, we prove that the two well-known natural normalizations of Hamiltonian functions on the symplectic manifold $(M,ω)$ canonically relates the action spectra of different normalized Hamiltonians on {\it arbitrary} symplectic manifolds $(M,ω)$. The natural class of normalized Hamiltonians consists of those w…

The paper proposes deep normalization to improve speaker recognition performance.

Entropy corrections improve GBM's predictive accuracy for non-log-normal distributions.

Copula-based normalizing flows improve flexibility and stability for heavy-tailed data.

Combines variational autoencoders with normalizing flows for faster training.

TCNF models SDEs using time deformation of Brownian motion.

A new method speeds up spectral normalization for neural nets.

This paper uses a geometric approach to understand how normalization layers affect neural network optimization.

Layer normalization improves federated learning with skewed labels.

Normalization methods play an important role in enhancing the performance of deep learning while their theoretical understandings have been limited. To theoretically elucidate the effectiveness of normalization, we quantify the geometry of the parameter space determined by the Fisher information matrix (FIM), which als…

Paper explores whether gradient normalization can replace clipping for SGD in heavy-tailed noise.