arXiv research

The paper studies matrix normalization and graph balancing using a new functional and gradient descent.

This paper analyzes how normalization layers improve neural network training.

Improved RCPs for MABs using normalized weight functions.

Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present …

We prove that square integrable holomorphic functions (with respect to a plurisubharmonic weight) can be extended in a square integrable manner from certain singular hypersurfaces (which include uniformly flat, normal crossing divisors) to entire functions in affine space. This provides evidence for a conjecture regard…

Normalization methods such as batch [Ioffe and Szegedy, 2015], weight [Salimansand Kingma, 2016], instance [Ulyanov et al., 2016], and layer normalization [Baet al., 2016] have been widely used in modern machine learning. Here, we study the weight normalization (WN) method [Salimans and Kingma, 2016] and a variant call…

The paper develops approximations for Pearson's chi-square statistic and applies them to confidence intervals.

Method measures weight similarity in neural networks using normalization and statistical inference.

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change o…

Recent seminal work at the intersection of deep neural networks practice and random matrix theory has linked the convergence speed and robustness of these networks with the combination of random weight initialization and nonlinear activation function in use. Building on those principles, we introduce a process to trans…

Mitigates anomaly score imbalance in long-tailed distributions.

OPAA estimates probability densities using functional analysis.

The paper analyzes various normalization methods in deep learning.

Finet uses FBN for efficient, lightweight neural networks.

Given a triangulation of a closed, oriented, irreducible, atoroidal 3-manifold every oriented, incompressible surface may be isotoped into normal position relative to the triangulation. Such a normal oriented surface is then encoded by non-negative integer weights, 14 for each 3-simplex, that describe how many copies o…

Constructs weight 1/2 multiplier systems for a specific group and relates to geometric edge paths.

Gradient descent on normalized networks reveals sparsity preferences.

Protein function prediction is the important problem in modern biology. In this paper, the un-normalized, symmetric normalized, and random walk graph Laplacian based semi-supervised learning methods will be applied to the integrated network combined from multiple networks to predict the functions of all yeast proteins …

Weight normalization speeds up matrix sensing problems.

We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output balanced. We validate our method on a set o…

Nonuniform tubular neighborhoods of curves in Euclidean n-space are studied by using weighted distance functions and generalizing the normal exponential map. Different notions of injectivity radii are introduced to investigate singular but injective exponential maps. A generalization of the thickness formula is obtaine…

FTIP uses normalizing flows to improve posterior inference in function space.

We reduce boundary determination of an unknown function and its normal derivatives from the (possibly weighted and attenuated) broken ray data to the injectivity of certain geodesic ray transforms on the boundary. For determination of the values of the function itself we obtain the usual geodesic ray transform, but for…

This paper presents a general framework for norm-based capacity control for $L_{p,q}$ weight normalized deep neural networks. We establish the upper bound on the Rademacher complexities of this family. With an $L_{p,q}$ normalization where $q\le p^*$, and $1/p+1/p^{*}=1$, we discuss properties of a width-independent ca…

Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.

Enhances DNN robustness and accuracy with $L_{2,\infty}$ normalization.

In this study, we consider classification problems based on neural networks in data-imbalanced environment. Learning from an imbalanced data set is one of the most important and practical problems in the field of machine learning. A weighted loss function based on cost-sensitive approach is a well-known effective metho…

New ODE-Block handles stateful layers with continuous-in-depth functions using basis functions.

Truncated densities are probability density functions defined on truncated domains. They share the same parametric form with their non-truncated counterparts up to a normalizing constant. Since the computation of their normalizing constants is usually infeasible, Maximum Likelihood Estimation cannot be easily applied t…

Use of an autoencoder (AE) as a normal model is a state-of-the-art technique for unsupervised-anomaly detection in sounds (ADS). The AE is trained to minimize the sample mean of the anomaly score of normal sounds in a mini-batch. One problem with this approach is that the anomaly score of rare-normal sounds becomes hig…

Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regulariza…

Residual networks (ResNet) and weight normalization play an important role in various deep learning applications. However, parameter initialization strategies have not been studied previously for weight normalized networks and, in practice, initialization methods designed for un-normalized networks are used as a proxy.…

This paper reveals periodic behavior in neural network training with BN and weight decay.

Study on volumes of random inscribed polytopes in projective geometries.

A new method to improve deep neural networks using weight rescaling.

With the development of deep neural networks, the size of network models becomes larger and larger. Model compression has become an urgent need for deploying these network models to mobile or embedded devices. Model quantization is a representative model compression technique. Although a lot of quantization methods hav…

In this study, a numerical quadrature for the generalized inverse Gaussian distribution is derived from the Gauss-Hermite quadrature by exploiting its relationship with the normal distribution. The proposed quadrature is not Gaussian, but it exactly integrates the polynomials of both positive and negative orders. Using…

A new clustering method for functional data using skewed distributions.

The paper integrates behavioral distortions into portfolio optimization using implied probability weighting functions.

LAWN normalizes logits to improve deep network adaptability and generalization.

In this paper we describe a procedure for refining the given triangulation of a 3-manifold that scales the PL-metric according to a given weight function while creating no new normal surfaces. It is known that an incompressible surface $F$ in a triangulated 3-manifold $M$ is isotopic to a normal surface that is of mini…

In this study, classification problems based on feedforward neural networks in a data-imbalanced environment are considered. Learning from an imbalanced dataset is one of the most important practical problems in the field of machine learning. A weighted loss function (WLF) based on a cost-sensitive approach is a well-k…

Recently, self-normalizing neural networks (SNNs) have been proposed with the intention to avoid batch or weight normalization. The key step in SNNs is to properly scale the exponential linear unit (referred to as SELU) to inherently incorporate normalization based on central limit theory. SELU is a monotonically incre…

In recent studies, several asymptotic upper bounds on generalization errors on deep neural networks (DNNs) are theoretically derived. These bounds are functions of several norms of weights of the DNNs, such as the Frobenius and spectral norms, and they are computed for weights grouped according to either input and outp…

Batch normalization improves deep networks by aligning their decision boundaries with data.

This paper examines weight initialization for 1-Lipschitz networks to improve robustness against adversarial attacks.

New method improves ResNet performance without batch normalization.

Theoretical justification for deep networks' performance with regularization techniques.