Method measures weight similarity in neural networks using normalization and statistical inference.
problem Quantifying weight similarity in non-convex neural networks.
method Chain normalization rule and hypothesis-training-testing statistical inference.
result Weights of identical neural networks converge to similar local solutions.
The paper analyzes various normalization methods in deep learning.
problem Lack of mathematical tools to analyze normalization methods.
method Proposed a lemma to define tools, analyzed BN, LN, WN, GN.
result Normalization methods can be unified on a sphere, improving training stability and weight norm.
Finet uses FBN for efficient, lightweight neural networks.
problem Building efficient neural networks with limited computational resources.
method Introduces Fine-grained Batch Normalization (FBN) and a novel light-weight network (Finet) that combines FBN with standard convolution.
result Finet achieves state-of-the-art performance on ImageNet classification with reduced computational complexity.
Gradient descent on normalized networks reveals sparsity preferences.
problem Understanding the inductive bias of gradient descent on normalized neural nets.
method Analysis of gradient descent on weight-normalized smooth homogeneous neural nets, focusing on SWN and EWN.
result EWN causes weights to be updated in a way that prefers asymptotic relative sparsity.
This paper analyzes how normalization layers improve neural network training.
problem Improving generalization performance and training speed of neural networks.
method Global convergence analysis of two-layer neural networks with ReLU activations and Weight Normalization.
result Introduction of normalization layers changes the optimization landscape, enabling faster convergence.
Weight normalization speeds up matrix sensing problems.
problem Matrix sensing with overparameterization.
method Generalized weight normalization with Riemannian optimization.
result WN achieves linear convergence, improving speed and complexity.
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output balanced. We validate our method on a set o…
This paper presents a general framework for norm-based capacity control for Lp,q weight normalized deep neural networks. We establish the upper bound on the Rademacher complexities of this family. With an Lp,q normalization where q≤p∗, and 1/p+1/p∗=1, we discuss properties of a width-independent ca…
Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.
problem Improving convergence of SGD in various settings.
method Develops a general weighted averaging scheme for SGD and establishes asymptotic normality.
result Establishes asymptotic normality and optimality of weighted averaged SGD solutions.
Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present …
Use of an autoencoder (AE) as a normal model is a state-of-the-art technique for unsupervised-anomaly detection in sounds (ADS). The AE is trained to minimize the sample mean of the anomaly score of normal sounds in a mini-batch. One problem with this approach is that the anomaly score of rare-normal sounds becomes hig…
Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regulariza…
Residual networks (ResNet) and weight normalization play an important role in various deep learning applications. However, parameter initialization strategies have not been studied previously for weight normalized networks and, in practice, initialization methods designed for un-normalized networks are used as a proxy.…
This paper reveals periodic behavior in neural network training with BN and weight decay.
problem Understanding the dynamics of neural network training with BN and weight decay.
method Rigorous investigation of empirical and theoretical mechanisms.
result Periodic behavior in training is a generalization of previously opposing perspectives.
A new method to improve deep neural networks using weight rescaling.
problem Overfitting and sensitivity to hyperparameters in weight decay.
method Weight rescaling (WRS) to control weight norm and prevent overfitting.
result WRS outperforms weight decay and other methods in various applications.
With the development of deep neural networks, the size of network models becomes larger and larger. Model compression has become an urgent need for deploying these network models to mobile or embedded devices. Model quantization is a representative model compression technique. Although a lot of quantization methods hav…
LAWN normalizes logits to improve deep network adaptability and generalization.
problem Large logits and weights lead to overfitting in deep networks.
method Logit Attenuating Weight Normalization (LAWN) constrains weight norms in the final sub-network.
result LAWN improves generalization and adaptability of deep networks.
The paper studies matrix normalization and graph balancing using a new functional and gradient descent.
problem Matrix normalization and graph balancing.
method A new functional called the non-normal energy, and gradient descent.
result Gradient descent of the non-normal energy converges to balanced graphs and preserves spectra and realness of weights.
Normalization methods such as batch [Ioffe and Szegedy, 2015], weight [Salimansand Kingma, 2016], instance [Ulyanov et al., 2016], and layer normalization [Baet al., 2016] have been widely used in modern machine learning. Here, we study the weight normalization (WN) method [Salimans and Kingma, 2016] and a variant call…
This paper examines weight initialization for 1-Lipschitz networks to improve robustness against adversarial attacks.
problem Improving the robustness of deep neural networks against adversarial attacks.
method Examined weight parametrization of AOL and SLL networks, calculated weight variance bounds, and demonstrated weight decay.
result Weight initialization causes deep 1-Lipschitz networks to decay to zero, and weight variance does not affect output variance distribution.
Theoretical justification for deep networks' performance with regularization techniques.
problem Understanding the performance of deep networks trained with the square loss.
method Analysis of gradient flow and theoretical justification of regularization techniques.
result Convergence to solutions with smaller Frobenius norms leads to better classification error bounds.
Flat semigroups can represent normal weighted homogeneous surface singularities.
problem Representability of flat semigroups in normal weighted homogeneous surface singularities.
method Study of numerical semigroups associated with surface singularities and prove representability conditions.
result A numerical semigroup is representable if and only if it can be written as a quotient of a flat semigroup.
PSiLON Net uses L1 weight normalization and 1-path-norm regularization for efficient learning and sparsity.
problem Efficient learning and sparsity in neural networks with limited data.
method PSiLON Net employs L1 weight normalization and 1-path-norm regularization to simplify the 1-path-norm and achieve efficient learning and near-sparse parameters. result PSiLON Net achieves reliable optimization and strong performance in the small data regime.
Unified framework for measuring concentration in weighted networks considering both weight distributions and network structure.
problem Traditional indices neglect the topology of relationships among network elements.
method Develops a family of topology-aware concentration indices that jointly account for weight distributions and network structure.
result The proposed indices preserve key properties and allow concentration to be evaluated across different dimensions of dependence.
Mitigates anomaly score imbalance in long-tailed distributions.
problem Class imbalance in normal data leads to skewed anomaly detection performance.
method Proposes an importance-weighted loss function to balance anomaly scores.
result Improves anomaly detection performance by 0.043 on real-world datasets.
Improved RCPs for MABs using normalized weight functions.
problem Slow convergence and inferior rewards in RCPs for MABs.
method Generalized marginalization of rewards using normalized weight functions.
result Improved RCPs become competitive with classic methods.
Study reveals dynamics of neural networks with normalization, weight decay, and SGD.
problem Understanding the equilibrium condition in Spherical Motion Dynamics (SMD).
method Investigates SMD by exploring the cause of equilibrium condition, introducing assumptions, proposing angular update, and verifying theoretical results.
result Proves weight norm and angular update can converge at linear rate under given assumptions.
Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change o…
The study explores weightings on submanifolds and their geometric properties.
problem Understanding weightings on submanifolds and their geometric implications.
method Detailed exploration of weighted normal bundles, weighted deformation spaces, and weighted blow-ups.
result A description of weightings in terms of subbundles of higher tangent bundles, leading to new concepts for Lie algebroids and groupoids.
Paper develops a weighted linearization approach for vector fields.
problem Linearizability of vector fields under weighted conditions.
method Formal Moser trick applied to power series, addressing weighted non-resonance condition.
result Formal Moser trick works over any field of characteristic zero.
Estimates RL data for dynamic treatment effects using GMM.
problem Estimating dynamic treatment effects from RL data with nonstationary behavior policies.
method Weighted GMM approach to stabilize variance in adaptive RL settings.
result Valid hypothesis testing and confidence regions for dynamic treatment effects.
Given a triangulation of a closed, oriented, irreducible, atoroidal 3-manifold every oriented, incompressible surface may be isotoped into normal position relative to the triangulation. Such a normal oriented surface is then encoded by non-negative integer weights, 14 for each 3-simplex, that describe how many copies o…
Deep convolutional neural networks are known to be unstable during training at high learning rate unless normalization techniques are employed. Normalizing weights or activations allows the use of higher learning rates, resulting in faster convergence and higher test accuracy. Batch normalization requires minibatch sta…
This paper explains how batch normalization auto-tunes the regularization parameter based on data statistics.
problem Batch normalization accelerates deep learning training but the exact relationship to regularization is unclear.
method Theoretical analysis and empirical validation of batch normalization's role in auto-tuning the regularization parameter.
result Batch normalization auto-tunes the regularization parameter based on data statistics.
The paper addresses statistical inference for online decision-making in a contextual bandit setting.
problem Understanding the performance of reward models in online decision-making with contextual information.
method The paper uses the contextual bandit framework with a linear reward model and the ε-greedy policy to address the exploration-exploitation dilemma. It employs the martingale central limit theorem and inverse propensity score weighting to establish asymptotic normality of parameter estimators. result The online ordinary least squares estimator and the online weighted least squares estimator are asymptotically normal, providing insights into the performance of the reward model.
The study of quotient structures in multi-graded bundles, including double vector bundles.
problem Understanding quotients of multi-graded bundles, especially double vector bundles.
method Analyzing quotients as towers of affine bundles and constructing normal bundles.
result Any quotient of multi-graded bundles fits into a tower of affine bundles.
A well-known issue of Batch Normalization is its significantly reduced effectiveness in the case of small mini-batch sizes. When a mini-batch contains few examples, the statistics upon which the normalization is defined cannot be reliably estimated from it during a training iteration. To address this problem, we presen…
Normal forms and moduli stacks for flat connections on complex manifolds.
problem Understanding singular flat connections on complex manifolds.
method Introducing homogeneous Lie groupoids and studying their representation theory to prove normal form theorems and moduli space structures.
result Moduli spaces of singular flat connections admit the structure of algebraic quotient stacks.
New algorithm detects communities in weighted networks, improving on binary ones.
problem Few methods exist for detecting communities in weighted networks.
method Pseudo-likelihood approach for weighted stochastic block model.
result The method is consistent and works well for both homogeneous and heterogeneous networks.
One of the challenges in the study of generative adversarial networks is the instability of its training. In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator. Our new normalization technique is computationally light and easy to in…
We prove that square integrable holomorphic functions (with respect to a plurisubharmonic weight) can be extended in a square integrable manner from certain singular hypersurfaces (which include uniformly flat, normal crossing divisors) to entire functions in affine space. This provides evidence for a conjecture regard…
We show that the coefficients of the re-normalized link invariants of the paper "Multivariable link invariants arising from Lie superalgebras of type I" are Vassiliev invariants which give rise to a canonical family of weight systems.
Intriguing empirical evidence exists that deep learning can work well with exoticschedules for varying the learning rate. This paper suggests that the phenomenon may be due to Batch Normalization or BN, which is ubiquitous and provides benefits in optimization and generalization across all standard architectures. The f…
BR-SNIS reduces bias in self-normalized IS without increasing variance.
problem Bias in self-normalized IS.
method Iterated sampling-importance resampling (ISIR) to form a bias-reduced estimator.
result Significant reduction in bias without increasing variance.
Utilizing a weight matrix we study surfaces of prescribed weighted mean curvature which yield a natural generalisation to critical points of anisotropic surface energies. We first derive a differential equation for the normal of immersions with prescribed weighted mean curvature, generalising a result of Clarenz and vo…
Defines linear weightings for vector bundles and explores their applications.
problem Understanding and extending the concept of weightings in vector bundles.
method Constructs weighted normal bundles and deformation spaces; explains the relationship between weightings and differential operators.
result Captures the rescaled spinor bundle and related constructions.
New method calculates Ricci curvature from distances between weighted volumes.
problem Calculating Ricci curvature for weighted Riemannian manifolds.
method Asymptotic retrieval of generalized Ricci tensor from scaled metric derivatives of Wasserstein 1-distances.
result Limiting coarse curvature of random graphs converges to generalized Ricci tensor.
FredNormer improves time series forecasting by adapting to frequency domain patterns.
problem Current normalization methods struggle with non-stationary time series due to their time-domain approach.
method FredNormer analyzes frequency components, adapts weights, and improves robustness.
result FredNormer boosts forecasting accuracy by 33.3% on ETTm2 dataset.