Proposes a novel framework for unsupervised domain adaptation using specialized batch normalization.
problem Improves unsupervised domain adaptation in deep neural networks.
method Integrates domain-specific batch normalization layers in convolutional neural networks, estimating pseudo-labels for target domain examples and learning final models with multi-task classification loss.
result Achieves state-of-the-art accuracy in standard and multi-source domain adaptation scenarios.
Zero-shot anomaly detection method using batch normalization.
problem Adapting anomaly detectors to new normal data distributions without training data.
method Adaptive Centered Representations (ACR) with batch normalization.
result First zero-shot AD results for tabular data and image data.
This research investigates if deep neural networks can be trained without batch normalization.
problem Training deep neural networks efficiently without batch normalization.
method Detailed study of batch normalization, comparison with other methods, and adaptation of training process.
result It is possible to train deep neural networks effectively without batch normalization.
Study improves BN TTA under distribution shift using higher-order asymptotics.
problem Improving BN TTA for changing data distributions.
method Integrates Edgeworth expansion and saddlepoint approximation with one-step M-estimation.
result Derives optimal weighting parameter for minimized mean-squared error.
Adaptive feature normalization improves model robustness to extraneous variables.
problem Degrading model performance due to extraneous variables in deep learning.
method Adaptive feature normalization using instance normalization instead of batch normalization.
result Adaptive normalization leads to significant performance gains across different datasets and architectures.
Study on sparsity in CNNs trained with adaptive methods.
problem Understanding and optimizing sparsity in CNNs trained with adaptive methods.
method Experimental study of filter level sparsity in CNNs with BN and ReLU, using adaptive gradient descent and L2 regularization.
result Implicit sparsity can improve CNN performance and speedup without modifications.
Batch normalization improves deep networks by aligning their decision boundaries with data.
problem Improving the performance and generalization of deep networks.
method Theoretical analysis of batch normalization as a function approximation technique for continuous piecewise affine splines.
result Batch normalization adapts the geometry of a deep network's partition to match the data, improving learning and generalization.
IEBN normalizes noise by enhancing instance-specific information, improving deep learning performance.
problem Improving deep learning performance by regulating noise in batch normalization.
method Integrates self-attention mechanism to recalibrate channel information in BN.
result IEBN outperforms BN with improved generalization and stability.
Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute…
New method trains deep ResNets without normalization, achieving state-of-the-art performance.
problem Training deep ResNets without normalization layers leads to instability and lower accuracy.
method Adaptive gradient clipping and Normalizer-Free ResNets design.
result Normalizer-Free ResNets achieve 86.5% top-1 accuracy on ImageNet, matching EfficientNet-B7.
AdaFlow adapts DNN density estimators to new distributions for anomaly detection and cross-domain translation.
problem Unsupervised anomaly detection and cross-domain translation with limited paired data.
method AdaFlow combines Normalizing Flows and Adaptive Batch-Normalizations for domain adaptation.
result AdaFlow efficiently adapts to new distributions with minimal computational resources.
Study reveals unique spectral features in neural network optimization.
problem Understanding dynamics of optimization in deep neural networks.
method Developed a tool to study Hessian spectrum evolution; analyzed structural features.
result Large isolated eigenvalues and gradient concentration observed in non-batch normalized networks.
Batch normalization biases linear models towards uniform margins, improving performance in binary classification.
problem Understanding the implicit bias of batch normalization in linear models and neural networks.
method Analyzing gradient descent convergence on linear models and two-layer CNNs with batch normalization.
result Gradient descent with batch normalization in linear models converges to a uniform margin classifier with an exponential convergence rate.
TaskNorm improves meta-learning performance by rethinking batch normalization.
problem Challenges in batch normalization for meta-learning with deep networks.
method Developed TaskNorm, a novel approach to batch normalization for meta-learning.
result TaskNorm consistently improves meta-learning performance across various datasets and meta-learning approaches.
Four improvements to Batch Normalization improve deep learning performance.
problem Improving Batch Normalization for better deep learning performance.
method Proposed improvements include reasoning about current examples, Ghost Batch Normalization, weight decay regularization, and a new normalization algorithm for small batch sizes.
result Performance gains across all batch sizes with no additional computation during training.
In this work, we investigate Batch Normalization technique and propose its probabilistic interpretation. We propose a probabilistic model and show that Batch Normalization maximazes the lower bound of its marginalized log-likelihood. Then, according to the new probabilistic model, we design an algorithm which acts cons…
Batch-normalized RHN improves gradient control in recurrent networks.
problem Gradient vanishing or exploding in recurrent networks.
method Batch normalization applied at each recurrence loop in RHN.
result Batch-normalized RHN converges faster and performs better.
CBN improves batch normalization for small mini-batch sizes.
problem Reduced effectiveness of Batch Normalization in small mini-batch sizes.
method CBN uses statistics from multiple recent iterations, compensating for network weight changes via Taylor polynomials.
result CBN outperforms original batch normalization and direct iteration statistics in object detection and image classification.
A new method to improve deep neural networks using weight rescaling.
problem Overfitting and sensitivity to hyperparameters in weight decay.
method Weight rescaling (WRS) to control weight norm and prevent overfitting.
result WRS outperforms weight decay and other methods in various applications.
We revisit fuzzy neural network with a cornerstone notion of generalized hamming distance, which provides a novel and theoretically justified framework to re-interpret many useful neural network techniques in terms of fuzzy logic. In particular, we conjecture and empirically illustrate that, the celebrated batch normal…
Study shows skewed data labels significantly impact decentralized ML accuracy.
problem Skewed data labels across devices/locations cause significant accuracy loss in decentralized ML.
method Detailed experimental study on skewed data labels, presenting SkewScout system-level approach.
result Skewed data labels are a fundamental challenge for decentralized learning, affecting many applications and models.
We find that CNNs with BN and ReLU exhibit feature sparsity through selective feature penalization.
problem Feature sparsity in CNNs trained with specific techniques.
method Empirical study and hypothesis testing on sparsification mechanisms.
result Selective feature penalization leads to feature sparsity in CNNs, comparable to explicit pruning.
Batch normalization makes deep residual networks train faster.
problem Training deep residual networks with large depths.
method Downscaling the residual branch by a normalizing factor early in training.
result Normalized residual blocks compute functions close to the identity function early in training.
Normalization techniques such as Batch Normalization have been applied successfully for training deep neural networks. Yet, despite its apparent empirical benefits, the reasons behind the success of Batch Normalization are mostly hypothetical. We here aim to provide a more thorough theoretical understanding from a clas…
New method uses batch normalization to improve OoD detection.
problem Out-of-distribution samples are not reliably detected by generative models.
method Proposes exploiting in-batch dependencies for OoD detection.
result Empirical results show improved robustness for high-dimensional images.
Real-time personalization for HAR models learns from new users without prior data.
problem Poor performance of HAR models on new users without labeled data.
method Incremental online domain adaptation using batch normalization.
result Personalized HAR models adapt to new users in real-time.
Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies. However, they are computationally expensive to train and difficult to parallelize. Recent work has shown that normalizing intermediate representations of neural networks can significantly im…
Bias-free CNNs outperform biased ones in blind image denoising.
problem Deep learning methods often require bias terms, leading to overfitting and poor performance outside training noise levels.
method Developed a bias-free convolutional neural network architecture by removing additive constant terms in every layer, including batch normalization.
result Bias-free CNNs generalize robustly across noise levels, preserving state-of-the-art performance within the training range.
Batch normalization makes neural networks more vulnerable to small adversarial perturbations.
problem Adversarial vulnerability of neural networks trained with batch normalization.
method Investigated the impact of batch normalization on adversarial robustness and compared it to weight decay.
result Substituting weight decay for batch norm nullifies adversarial vulnerability.
This paper explores orthonormalization layers as alternatives to batch normalization.
problem The need for alternatives to batch normalization.
method Investigates orthonormalization layers as an alternative to batch normalization.
result Compared the performance of orthonormalization layers with batch normalization.
New method improves ResNet performance without batch normalization.
problem Improving ResNet performance without batch normalization.
method Adapted Weight Standardization to maintain signal propagation.
result Highly performant ResNets achieve state-of-the-art performance on ImageNet.
Analysis shows BN prevents gradient vanishing/explosion in residual networks.
problem Gradient vanishing/explosion problem in residual networks.
method Mathematical analysis of BN and residual network training.
result BN confounds gradient variance, preventing vanishing/explosion.
Batch normalization prevents rank collapse in deep networks, improving training stability.
problem Rank collapse in randomly initialized deep networks with increasing depth.
method Investigates spectral instabilities in random matrices and uses batch normalization to avoid rank collapse.
result Batch normalization prevents rank collapse in both linear and ReLU networks, improving training stability.
Batch normalization in the last layer reduces sharpness in wide neural networks.
problem Pathological sharpness in wide neural networks.
method Quantifying the geometry of the parameter space using Fisher information matrix and analyzing deep neural networks with random initialization.
result Batch normalization in the last layer significantly decreases pathological sharpness under specific conditions.
A new method normalizes activations to match batch normalization without batch dependence.
problem Performance degradation with batch-independent normalization techniques.
method Proxy-Normalizing Activations
result Proxy-Normalization technique emulates batch normalization's behavior and performance.
Novel approach characterizes deep neural networks at initialization.
problem Characterizing the behavior of deep neural networks at initialization.
method A novel approach considering the evolution of statistical moments of signal and noise.
result Established that skip-connections in residual networks lead to well-behaved moments and no pathology.
Online Normalization normalizes neural network activations without batching for better accuracy.
problem Theoretical limitations of Batch Normalization and its inapplicability to certain network types.
method Introduces an unbiased gradient computation technique for normalized activations without using batches.
result Equivalent accuracy to Batch Normalization without batch usage.
Batch normalization makes deep neural networks' representations increasingly orthogonal.
problem Orthogonality of deep neural network representations.
method Random linear transformations in successive batch-normalizations.
result Orthogonality of representations improves SGD performance.
This research explains how Batch Normalization improves neural networks through theoretical analysis.
problem Understanding the impact of Batch Normalization on neural network training and generalization.
method Using a basic neural network block, the study analyzes Batch Normalization by decomposing it into population normalization and gamma decay.
result Batch Normalization acts as an implicit regularizer, decomposable into explicit components that affect learning dynamics and generalization.
This paper reveals periodic behavior in neural network training with BN and weight decay.
problem Understanding the dynamics of neural network training with BN and weight decay.
method Rigorous investigation of empirical and theoretical mechanisms.
result Periodic behavior in training is a generalization of previously opposing perspectives.
Finet uses FBN for efficient, lightweight neural networks.
problem Building efficient neural networks with limited computational resources.
method Introduces Fine-grained Batch Normalization (FBN) and a novel light-weight network (Finet) that combines FBN with standard convolution.
result Finet achieves state-of-the-art performance on ImageNet classification with reduced computational complexity.
This paper explains how batch normalization auto-tunes the regularization parameter based on data statistics.
problem Batch normalization accelerates deep learning training but the exact relationship to regularization is unclear.
method Theoretical analysis and empirical validation of batch normalization's role in auto-tuning the regularization parameter.
result Batch normalization auto-tunes the regularization parameter based on data statistics.
BatchNorm helps BNNs avoid exploding gradients during training.
problem Difficulty in training Binary Neural Networks (BNNs).
method Theoretical study and numerical experiments on the role of BatchNorm in BNNs.
result BatchNorm is crucial for avoiding exploding gradients in BNNs.
The study explains delayed spikes in batch-normalized models.
problem Delayed spikes in batch-normalized models.
method Analyzing batch-normalized linear models, deriving conditions for delayed onset and waiting time.
result Explicit conditions for delayed-onset and waiting time in whitened square-loss linear regression.
Unified framework for Riemannian deep learning across manifold-valued representations.
problem Deep learning on manifold-valued representations often relies on Euclidean approximations or costly geometric operations.
method Develops reusable neural modules, manifold-specific network architectures, and geometric designs.
result Generalizes batch normalization and multinomial logistic regression to broader classes of manifolds.
Unified framework for Riemannian deep learning across manifold-valued representations.
problem Deep learning on manifold-valued data lacks reusable modules, specific network architectures, and efficient geometric operations.
method Develops reusable neural modules, manifold-specific network architectures, and geometric designs for broad classes of Lie groups and gyrogroups.
result Generalizes batch normalization and multinomial logistic regression to Riemannian manifolds, including SPD and hyperbolic spaces.
Split-BN improves SSL by separating BN stats for unlabeled data.
problem SSL benefits from unlabeled data are often lost or even harmful when class mismatch exists.
method Split-BN uses separate BN statistics for unlabeled data.
result Split-BN improves SSL performance in shifted domain settings.
Paper proposes MABN to stabilize BN in small batch sizes.
problem Weakness of BN in small batch sizes limits its use in tasks with small batch sizes.
method Identifies and addresses two extra batch statistics in BN's backward propagation.
result MABN completely restores BN's performance in small batch sizes without additional nonlinear operations.