Diagonal linear networks converge to lasso regularization path during training.
problem Understanding the regularization behavior of diagonal linear networks.
method Analyzing the training trajectory of diagonal linear networks and comparing it to the lasso regularization path.
result The training trajectory of diagonal linear networks is closely related to the lasso regularization path.
Enhances neural network regularization with no extra cost.
problem Improving neural network robustness and generalization.
method Train an ensemble of weight matrices with stochastic regularization and explicitly average outputs.
result Consistent improvement on various image classification tasks.
A simple regularization technique speeds up training of Neural ODEs.
problem Training Neural ODEs is computationally expensive.
method Randomly sampling the end time of the ODE during training.
result Significantly decreases training time and improves performance.
Cross-regularization adapts model complexity during training.
problem Manual tuning of model complexity for overfitting prevention.
method Directly adapts regularization parameters through validation gradients during training.
result Organic emergence of architecture-specific regularization during training.
Unified framework for understanding and optimizing training acceleration.
problem Challenges in optimizing training with regularization and acceleration techniques.
method Explains how AdaGrad, RMSProp, and Adam accelerate training, and derives a generalization for L1-regularization. result Derives a unified mathematical framework for understanding and optimizing training acceleration.
New method for training deep neural networks with regularization, converging to better generalization.
problem Improving generalization of deep neural networks through explicit regularization.
method Regularizer Mirror Descent (RMD) method, inspired by convergence properties of stochastic mirror descent (SMD).
result RMD converges to a point close to the minimizer of the cost function, leading to better generalization performance.
Study L2 regularization in deep networks, uncovering performance relations and proposing a training schedule.
problem Understanding and optimizing L2 regularization in deep learning models. method Empirical observations and theoretical analysis of gradient flow dynamics in infinitely wide networks.
result Empirical relations between model performance, L2 coefficient, learning rate, and training steps; optimal regularization parameter prediction; improved training schedule. Simple regularization methods mimic adversarial training's robustness.
problem Expensive adversarial training for robustness.
method Label smoothing and logit squeezing.
result Achieves strong adversarial robustness without adversarial examples.
Improves deep transfer learning by preventing performance degradation.
problem Deep transfer learning can degrade performance when using inappropriate pre-trained weights.
method Proposes a novel strategy to compute new descent directions that preserve regularization effects.
result DTNH strategy improves performance of deep transfer learning tasks by 0.1%--7%.
Efficient regularization mitigates catastrophic overfitting in single-step adversarial training.
problem Catastrophic overfitting in single-step adversarial training.
method ELLE regularization term to enforce local linearity of the loss function.
result Our regularization term effectively mitigates catastrophic overfitting without the drawbacks of previous methods.
The paper examines how adversarial training and noise affect neural network performance.
problem Overfitting in adversarial training and data augmentation.
method Adversarial training and data augmentation with noise in the context of regularized regression in RKHS.
result Appropriate regularization can prevent overfitting and improve performance.
Adversarial training linked to operator norm regularization, proving network sensitivity to attacks.
problem Robustifying neural networks against adversarial attacks.
method Theoretical link established between adversarial training and operator norm regularization.
result Adversarial training is equivalent to data-dependent operator norm regularization.
RMT reveals self-regularization in neural networks, including traditional and heavy-tailed forms.
problem Understanding and quantifying self-regularization in neural networks.
method Application of Random Matrix Theory to analyze weight matrices of various neural network models.
result Identification of 5+1 phases of training in neural networks, including traditional and heavy-tailed self-regularization.
New implicit regularization drives deep networks towards simple models.
problem Training deep neural networks with noise.
method Stochastic gradient descent with perturbed labels, analyzing dynamics near zero-error parameters.
result The training dynamics are governed by an implicit regularization term, leading to simpler models.
Random Matrix Theory (RMT) is applied to analyze weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly indicate that t…
Adversarial Lipschitz Regularization improves Wasserstein GANs without gradient norm penalties.
problem Training stability and sample quality issues in Wasserstein GANs.
method Explicit Lipschitz penalty using adversarial training.
result Explicit Lipschitz penalty leads to competitive performance in Wasserstein GANs.
The paper introduces a novel method for training neural network Stein critics with staged L2-regularization.
problem Learning to differentiate model distributions from observed data in high-dimensional settings.
method Developed a novel staging procedure for L2 regularization over training time, leveraging the advantages of highly-regularized training at early times. result Theoretical guarantees and empirical validation show that the method improves the approximation of the training dynamic by the kernel optimization, leading to faster convergence and better performance.
This work analyzes how to choose regularization norms for adversarial training in high dimensions.
problem Choosing the right regularization norm for adversarial training in high-dimensional settings.
method Derives asymptotic descriptions and uniform convergence bounds for robust, regularized empirical risk minimizers.
result Characterizes the relationship between perturbation size and optimal regularization choice.
Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling t…
Shakeout improves deep neural network training by adaptively combining regularization terms.
problem Overfitting in deep neural networks.
method Randomly enhances or reverses each unit's contribution to the next layer.
result Shakeout outperforms Dropout in classification tasks and reduces training instability.
We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-deman…
The paper shows how data augmentation and regularization can enforce group equivariance in machine learning models.
problem Improving model performance by leveraging known symmetries in machine learning tasks.
method Training with data augmentation and regularization to enforce group equivariance.
result Equivariance of the trained model can be achieved through training on augmented data in tandem with regularization.
Study optimal ridge regularization for out-of-distribution prediction.
problem Optimal ridge regularization for predicting out-of-distribution data.
method Established conditions for optimal regularization under covariate and regression shifts, proving monotonic risk in data aspect ratio.
result Negative regularization can be optimal under shifts, even with isotropic or underparameterized training features.
New neural network training method uses bi-fidelity data to reduce errors.
problem Training neural networks with limited high-fidelity data.
method Bi-fidelity ℓ1-regularization strategies. result Bi-fidelity ℓ1-regularization reduces errors by one order of magnitude. Despite the growing interest in generative adversarial networks (GANs), training GANs remains a challenging problem, both from a theoretical and a practical standpoint. To address this challenge, in this paper, we propose a novel way to exploit the unique geometry of the real data, especially the manifold information. …
Noise regularization improves CDE models without overfitting.
problem Overfitting in neural network-based conditional density estimation.
method Noise regularization method that adds random perturbations to data.
result Noise regularization significantly outperforms other methods across various datasets and models.
Consistency regularization improves robustness to noisy labels.
problem Improving model robustness to noisy labels in machine learning.
method Empirical study of consistency regularization on noisy datasets.
result Consistency regularization improves model robustness to label noise.
New input gradient regularization improves adversarial robustness efficiently.
problem Improving adversarial robustness in machine learning models.
method Derive robustness bounds, implement scaleable input gradient regularization, avoid double backpropagation.
result Input gradient regularization is competitive with adversarial training and avoids gradient obfuscation.
Improves generalization with few samples using a new regularization method.
problem Training deep neural networks with limited data leads to overfitting.
method Sample-based regularization (SBR) to improve generalization without relying on source model knowledge.
result SBR outperformed existing methods in various configurations.
New geometric insights reveal properties of adversarial training problems.
problem Adversarial training in binary classification.
method Equivalence with regularized risk minimization and convex relaxations.
result Existence of minimal and maximal solutions, and regular solutions.
RMDA trains structured neural networks with regularization and variance reduction.
problem Training structured neural networks with desired properties.
method RMDA algorithm for structured NNs with regularization and variance reduction.
result RMDA achieves desired structures identical to regularizer's at stationary points.
Two regularization methods improve deep neural network performance on noisy data.
problem Improving deep neural network performance on noisy labeled data.
method Two simple regularization methods: distance regularization and auxiliary variable.
result Gradient descent with either method achieves generalization guarantee on clean data despite noisy labels.
New method trains normalizing flows using entropy-regularized transport.
problem Training continuous normalizing flows efficiently.
method Formulates flows as gradients of scalar potentials, training only these potentials.
result Trains normalizing flows without explicit flow computation during training.
New method encodes function preferences into neural nets for better generalization.
problem Challenges in encoding explicit function preferences in neural network training.
method Function-space empirical Bayes (FSEB) regularization.
result FSEB leads to near-perfect semantic shift detection and improved generalization.
A new training method for GANs improves image generation quality.
problem Training GANs is challenging and unstable.
method Consistency regularization to stabilize GAN training.
result CR-GAN achieves best FID scores for unconditional image generation.
A new method reduces adversarial training time without overfitting.
problem Catastrophic overfitting in single-step adversarial training.
method FGSMPR: FGSM with PGD Regularization.
result Reduces the gap to multi-step adversarial training.
Artificial datasets can serve as a form of regularization for deep learning.
problem Real data shortage in deep learning.
method Injecting noise to high-level features in artificial data generation.
result Artificial data generation can be treated as a form of 'deep' regularization.
We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations. Our regularizer can be derived as a controlled approximation from first principles, leveraging the fundamental link between training with noise and regularization. It adds…
New measure shows various training techniques control model complexity.
problem Understanding how to control model complexity in deep learning.
method Developed geometric complexity measure and demonstrated its effectiveness.
result Many training techniques control geometric complexity, providing a unified framework.
Regularization improves policy optimization in RL, especially on harder tasks.
problem Lack of conventional regularization in RL methods.
method Comprehensive study of regularization techniques on policy networks with multiple RL algorithms.
result Conventional regularization techniques significantly improve policy optimization, especially on harder tasks.
SGD can converge to bad models if not explicitly regularized.
problem Overparameterized neural networks can converge to poorly-generalizing models.
method Train with random labels first, then correct labels; observe SGD behavior.
result SGD can reach bad global minima if not explicitly regularized.
ALPS improves neural network robustness and generalization.
problem Challenges in designing effective regularization schemes for adversarial robustness.
method Adversarial Labelling of Perturbed Samples (ALPS) using synthetic samples and min-max formulation.
result ALPS achieves state-of-the-art regularization performance and adversarial robustness.
This study connects Jacobian regularization to adversarial robustness and improves generalization.
problem Adversarial attacks make deep neural networks vulnerable.
method Developed a connection between Jacobian regularization and adversarial training, and established robust generalization gaps.
result Jacobian norms are related to both standard and robust generalization.
The paper proposes effective margin regularization to improve adversarial robustness in deep neural networks.
problem Adversarial vulnerability of deep neural networks (DNNs).
method Regularization of effective weight norm during training to maximize effective margins.
result Effective margin regularization (EMR) boosts adversarial robustness in both standard and adversarial training.
We propose an algorithm for exploring the entire regularization path of asymmetric-cost linear support vector machines. Empirical evidence suggests the predictive power of support vector machines depends on the regularization parameters of the training algorithms. The algorithms exploring the entire regularization path…
New regularizers tighten convex relaxation bounds for neural networks.
problem Large gap between certifiable and empirical robustness in neural networks.
method Two regularizers to train neural networks yielding tighter convex relaxation bounds.
result Higher certified accuracy with proposed regularizers.
Regularization improves stability and consistency of sparse autoencoders.
problem Varying features across random seeds and training choices in SAEs.
method Added L1 or L2 penalties on encoder and decoder weights.
result L2 regularization increases cross-seed feature consistency.
Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant represent…