BatchNorm helps train quantized networks by avoiding gradient explosion.
problem Training quantized neural networks is difficult due to gradient issues.
method Investigated the impact of BatchNorm on both full-precision and quantized networks.
result BatchNorm avoids gradient explosion in quantized networks, contrary to expectations.
New normalization method makes neural networks more robust to adversarial attacks.
problem Adversarial vulnerability of BatchNorm in deep neural networks.
method Identified distribution shift caused by adversarial images, proposed RobustNorm to use inference-time statistics.
result RobustNorm makes models more robust to adversarial attacks without sacrificing BatchNorm benefits.
BatchNorm helps BNNs avoid exploding gradients during training.
problem Difficulty in training Binary Neural Networks (BNNs).
method Theoretical study and numerical experiments on the role of BatchNorm in BNNs.
result BatchNorm is crucial for avoiding exploding gradients in BNNs.
Training only BatchNorm parameters achieves surprisingly high performance in deep networks.
problem Understanding the role and expressive power of affine parameters in BatchNorm.
method Investigating performance when training only BatchNorm parameters and freezing all other weights.
result BatchNorm achieves high performance by learning to disable around a third of random features.
Batch Normalization (BatchNorm) is a widely adopted technique that enables faster and more stable training of deep neural networks (DNNs). Despite its pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly understood. The popular belief is that this effectiveness stems from controlling the chan…
Batch Normalization (BatchNorm) is an extremely useful component of modern neural network architectures, enabling optimization using higher learning rates and achieving faster convergence. In this paper, we use mean-field theory to analytically quantify the impact of BatchNorm on the geometry of the loss landscape for …
Riemannian batchnorm improves SPD matrix networks.
problem Learning SPD matrices in deep networks.
method Introduces a novel Riemannian batchnorm algorithm.
result Riemannian batchnorm outperforms existing methods.
Derandomizing PAC-Bayes bounds for smooth loss functions
problem Derandomizing PAC-Bayes bounds for smooth loss functions
method Exploiting smoothness properties of both the loss and the predictor class
result Bounds for deterministic predictors that involve flatness quantities
Derives bounds for deterministic predictors using smooth loss functions.
problem Generalizing probabilistic predictors to deterministic ones.
method Exploits smoothness properties of loss and predictor classes, controlling the Jensen gap class through Rademacher complexity.
result Derives bounds for deterministic predictors involving flatness quantities from Jacobians and Hessians.
Normalization techniques play an important role in supporting efficient and often more effective training of deep neural networks. While conventional methods explicitly normalize the activations, we suggest to add a loss term instead. This new loss term encourages the variance of the activations to be stable and not va…
DEEP-BO optimizes hyperparameters of deep networks, outperforming existing methods.
problem Hyperparameter optimization of deep networks is challenging due to the complexity and sensitivity of DNN performance.
method Enhanced Bayesian Optimization (DEEP-BO) specifically designed for deep networks, incorporating diversification, early termination, and parallelism.
result DEEP-BO outperforms or matches other state-of-the-art methods on six DNN benchmarks.
GraphNorm accelerates GNN training by adapting InstanceNorm, improving convergence and generalization.
problem Improving convergence and generalization of Graph Neural Networks (GNNs).
method Adapting InstanceNorm to GNNs, proposing GraphNorm with a learnable shift.
result GNNs with GraphNorm converge faster and achieve better performance on benchmarks.
Study interprets neural network generalization and memorization on corrupted data.
problem Understanding when a neural network has memorized corrupted data versus learned the underlying rule.
method Analyzes multi-layer perceptrons and Transformers on modular arithmetic tasks with corrupted labels.
result Regularization methods can force networks to ignore corrupted data, improving accuracy on uncorrupted data.
Adaptive algorithms improve performance in non-convex optimization across various scenarios.
problem Improper handling of noise scales, gradient magnitudes, and smoothness in non-convex optimization.
method Design and analysis of noise-adaptive, scale-free, and generalized algorithms.
result Adaptive algorithms achieve optimal rates and performance in diverse optimization settings.
AutoInit automatically finds good neural network initialization.
problem Finding optimal neural network initialization is crucial but time-consuming.
method Uses Jacobian tuning to automatically adjust network hyperparameters.
result The method finds good initialization for various network architectures.
Existing Rademacher complexity bounds for neural networks rely only on norm control of the weight matrices and depend exponentially on depth via a product of the matrix norms. Lower bounds show that this exponential dependence on depth is unavoidable when no additional properties of the training data are considered. We…
Study reveals how initial weights influence convergence in deep ReLU networks.
problem Understanding the dynamics and generalization of deep ReLU networks.
method Teacher-student setting, gradient analysis, and activation assumptions.
result Initial weights close to teacher nodes lead to faster convergence, and fan-out weights of other nodes converge to zero in over-parameterized cases.
The study analyzes optimization trajectories in neural networks to reveal redundancy and redundancy-reducing strategies.
problem Understanding the directional structure and redundancy in neural network optimization.
method Introducing natural notions of complexity for optimization trajectories and analyzing their directional nature.
result Training only scalar batchnorm parameters can match the performance of training the entire network, indicating potential for hybrid optimization schemes.
Geometric theory of projection heads in self-supervised learning.
problem Dimensional collapse and information invariance trade-off in projection heads.
method Geometric modeling of projection heads as Riemannian metrics, analyzing Hessian eigenvalues, and tracking optimization geometry.
result Smooth nonlinear heads induce negative curvature, preventing collapse; linear and ReLU heads cannot.
AriaNN enables private deep learning with minimal interaction and reduced key sizes.
problem Private deep learning with minimal interaction and reduced key sizes.
method Semi-honest 2-party computation protocol with function secret sharing, optimized primitives for neural network operations.
result Efficient private comparison for ReLU operations with reduced key size and improved performance.
Automatically designs normalization and activation layers together.
problem Designing normalization and activation layers separately.
method Unified tensor-to-tensor computation graph, low-level mathematical functions, rejection protocols, multi-objective evolution.
result Discovery of EvoNorms with novel structures.