arXiv research

Weight decay stabilizes training dynamics by slowing progressive sharpening.

Cautious Weight Decay modifies weight decay for better optimization.

Study on optimal ReLU networks with weight decay for interpolation.

Novel Adam-family method with decoupled weight decay for training neural networks.

SGD and weight decay encourage neural networks to learn low-rank weight matrices.

Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization. Literal weight decay has been shown to outperform $L_2$ regularization for…

A new method to improve deep neural networks using weight rescaling.

This paper reveals periodic behavior in neural network training with BN and weight decay.

Analyzes minima of deep linear networks with weight decay.

Adam's generalization performance is improved by batch size and weight decay in neural networks.

Gradient flow with weight decay shows grokking effect in deep learning.

Regularization in the optimization of deep neural networks is often critical to avoid undesirable over-fitting leading to better generalization of model. One of the most popular regularization algorithms is to impose L-2 penalty on the model parameters resulting in the decay of parameters, called weight-decay, and the …

Proposes volumization for neural networks to control bias-variance tradeoff.

Weight Decay induces low-rank weight matrices in neural networks, improving generalization.

We introduce a new weight-decay scaling rule to maintain sublayer gains across different widths in modern scale-invariant architectures.

This paper analyzes convergence of large-scale Transformers with weight decay.

The paper connects neural collapse and low-rank bias in networks with L2 regularization.

Study reveals dynamics of neural networks with normalization, weight decay, and SGD.

Performance-critical machine learning models should be robust to input perturbations not seen during training. Adversarial training is a method for improving a model's robustness to some perturbations by including them in the training process, but this tends to exacerbate other vulnerabilities of the model. The adversa…

OUI tool detects optimal Weight Decay for DNNs without validation data.

Wide neural networks with weight decay exhibit neural collapse.

Regularization and data augmentation can be class-dependent, leading to poor performance on some classes.

Regularizers change the geometric properties of loss functions in neural networks.

This paper investigates the effectiveness of decoupled weight decay at the start of training.

Lion optimizer performs well in training AI models with memory efficiency.

Adversarial training effectiveness varies widely due to inconsistent training settings.

Theoretical justification for deep networks' performance with regularization techniques.

AdamW optimizes a constrained loss with $\ell_\infty$ norm constraint.

New regularizer improves neural network robustness and generalization.

Develops exact convex optimization formulations for neural networks.

A recent breakthrough in deep learning theory shows that the training of over-parameterized deep neural networks can be characterized by a kernel function called \textit{neural tangent kernel} (NTK). However, it is known that this type of results does not perfectly match the practice, as NTK-based analysis requires the…

Intriguing empirical evidence exists that deep learning can work well with exoticschedules for varying the learning rate. This paper suggests that the phenomenon may be due to Batch Normalization or BN, which is ubiquitous and provides benefits in optimization and generalization across all standard architectures. The f…

Convex dual network improves neural network reconstruction for medical imaging.

Framework models supervised learning in non-stationary data.

Hybrid regularization avoids double descent in random feature models.

Variational Laplace improves Bayesian neural networks performance.

We analyze dropout in deep networks with rectified linear units and the quadratic loss. Our results expose surprising differences between the behavior of dropout and more traditional regularizers like weight decay. For example, on some simple data sets dropout training produces negative weights even though the output i…

Variational Laplace improves Bayesian neural network performance without sampling.

Analyzes why neural networks generalize beyond training data.

New method for training deep neural networks with regularization, converging to better generalization.

Paper introduces vector-valued variation spaces for multi-output neural networks.

Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present …

Although deep learning has produced dazzling successes for applications of image, speech, and video processing in the past few years, most trainings are with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyper-parameters remains a black art that requires years of experience to ac…

New method trains neural networks with threshold activation functions efficiently.

New framework explains deep learning using signal processing techniques.

This study reveals the critical role of scale vectors in large language models, improving optimization and expressivity.

New method proves non-contrastive self-supervised learning learns useful features.

The paper reveals low-rank structure in neural network gradients, influenced by data and model parameters.