Noise Injection probes deep learning dynamics during training phases.
problem Understanding the learning mechanism of deep neural networks.
method Noise Injection Nodes (NINs) are used to perturb DNN architectures without altering the optimization algorithm.
result Distinct training phases are observed based on the scale of injected noise.
S-SGD adds symmetrical noise to weights to avoid sharp minima in deep learning.
problem SGD does not always converge to a flat minimum, leading to poor generalization.
method Symmetrical weight noise injection in SGD.
result S-SGD outperforms conventional SGD and weight-noise injection methods in large batch training.
Noise Injection Node Regularization improves robustness in neural networks.
problem Improving robustness of neural networks against various perturbations.
method Injecting structured noise into neural networks during training.
result Significant improvement in robustness against data perturbations.
This paper connects noise injection to Bayesian inference for neural networks, improving model uncertainty.
problem Improving the reliability and confidence of neural network predictions through uncertainty quantification.
method Introducing noise into neural network parameters during training and inference to estimate prediction uncertainty.
result The MCNI method outperforms baseline models in regression and classification tasks.
Proposes a new noise injection method for neural networks that improves accuracy and representation clarity.
problem Improving neural network performance and representation clarity.
method Adaptive Structured Noise Injection (ASNI) for shallow and deep neural networks.
result Boosts the accuracy of neural networks and disentangles hidden layer representations.
Noise injection improves inference privacy in DNN models.
problem Malicious servers can infer sensitive attributes from input data.
method Adaptive Noise Injection (ANI) using a lightweight DNN on the client.
result Significant improvement in privacy (up to 48.5% degradation in sensitive-task accuracy with <1% degradation in primary accuracy).
Improves RNN performance under noisy computations.
problem Power and speed limitations in deep learning with noisy analog circuits.
method Deep Noise Injection training to robustify RNN weights/biases.
result Trained RNNs show more consistent performance under noisy inference.
In Deep Learning, Stochastic Gradient Descent (SGD) is usually selected as a training method because of its efficiency; however, recently, a problem in SGD gains research interest: sharp minima in Deep Neural Networks (DNNs) have poor generalization; especially, large-batch SGD tends to converge to sharp minima. It bec…
Artificial datasets can serve as a form of regularization for deep learning.
problem Real data shortage in deep learning.
method Injecting noise to high-level features in artificial data generation.
result Artificial data generation can be treated as a form of 'deep' regularization.
Enhances DIM to match learned representations to a specific distribution.
problem Learning representations conforming to a specific distribution.
method Injecting noise into normalized outputs of the encoder while keeping the InfoMax training objective.
result Learning uniformly and normally distributed representations, as well as representations of other absolutely continuous distributions.
Geometric analysis improves noise injection in GANs.
problem Unclear mechanism of noise injection in GANs.
method Geometric framework based on Riemannian geometry.
result A new strategy for noise injection is devised.
Enhances deep learning robustness to noise without sacrificing clean data accuracy.
problem Robustness of deep neural networks to input noise.
method Discriminative loss at penultimate layer and class-wise feature alignment with Gaussian noise.
result Improves robustness to various perturbations without degrading clean data accuracy.
New method for certified unlearning reduces noise injection.
problem Achieving formal unlearning guarantees with adaptive noise calibration.
method Adaptive per-instance noise calibration based on individual data point sensitivities.
result Derivation of high-probability per-instance sensitivity bounds for ridge regression.
Noise improves model quality in non-linear neural networks during decentralized training.
problem Improving generalization of locally trained neural networks.
method Injecting noise into the weights of neural networks during decentralized training.
result Noise injection improves model quality for non-linear neural networks, but not for linear models.
Colored noise improves neural network robustness against adversarial attacks.
problem Vulnerability of neural networks to adversarial perturbations.
method Injection of colored noise into network weights and activations during adversarial training.
result Our approach outperforms previous methods in terms of adversarial accuracy on CIFAR-10 and CIFAR-100 datasets.
Theoretical framework for data augmentation in finance improves portfolio construction.
problem Improving portfolio construction in speculative markets.
method Developed a theoretical framework for data augmentation and regularization in deep learning for finance.
result A simple noise injection algorithm improves portfolio construction over no noise.
We present a noise-injected version of the Expectation-Maximization (EM) algorithm: the Noisy Expectation Maximization (NEM) algorithm. The NEM algorithm uses noise to speed up the convergence of the EM algorithm. The NEM theorem shows that injected noise speeds up the average convergence of the EM algorithm to a local…
Noise injection before gradient steps helps in regularization for neural networks.
problem Improving generalization in overparametrized neural networks.
method Injecting small noise perturbations before computing gradient steps, especially in layer-wise fashion.
result Small noise perturbations can explicitly regularize neural networks without variance explosion.
PDA improves deep neural networks' robustness against adversarial and common corruptions.
problem Deep neural networks' lack of robustness against common corruptions and adversarial attacks.
method Progressive Data Augmentation (PDA) that injects diverse adversarial noises during training.
result PDA-trained networks are more robust against both adversarial and common corruptions.
GNIs induce a regulariser that penalizes high-frequency components in neural network activations.
problem Understanding the regularizing effect of Gaussian noise injections on neural network activations.
method Deriving the explicit regularizer by marginalizing out injected noise and analyzing its effect in the Fourier domain.
result GNIs induce a regularizer that produces calibrated classifiers with large margins.
Adversarial examples have been shown to exist for a variety of deep learning architectures. Deep reinforcement learning has shown promising results on training agent policies directly on raw inputs such as image pixels. In this paper we present a novel study into adversarial attacks on deep reinforcement learning polic…
GNIs induce asymmetric heavy-tailed noise in SGD, affecting network performance.
problem The effect of Gaussian noise injections on SGD dynamics and network performance.
method Developed a Langevin-like SDE driven by asymmetric heavy-tailed noise to model the modified SGD dynamics.
result GNIs induce an implicit bias that varies with noise heaviness and asymmetry, affecting network performance.
Study noise in inference to improve accuracy and security.
problem Noise in inference affects deep learning systems' accuracy and security.
method Noise-injected training and voting method for improving accuracy; defensive architecture for adversarial attacks.
result Significant improvement in accuracy and robustness against attacks.
Noise in RNNs promotes flatter minima and more stable dynamics.
problem Understanding and optimizing the training of RNNs with noise.
method Formalizing RNNs as stochastic differential equations and analyzing the effect of noise in the hidden states.
result Noise injection in RNNs leads to flatter minima, more stable dynamics, and improved robustness.
Noise improves deep neural network performance, especially in knowledge distillation.
problem Improving deep neural network performance and reducing performance gap.
method Injecting constructive noise at different levels in the collaborative learning framework.
result Constructive noise enables effective training and distillation of desirable characteristics.
Label smoothing improves model performance even with noisy labels.
problem Mitigating label noise in deep learning models.
method Examined label smoothing as a technique to cope with label noise and compared it to loss-correction methods.
result Label smoothing is competitive with loss-correction techniques under label noise and beneficial for distillation from noisy data.
Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatica…
Anti-correlated noise improves machine learning model generalization.
problem Improving machine learning model generalization.
method Injecting anticorrelated noise into gradient descent steps.
result Anti-correlated noise leads to better model generalization than uncorrelated noise.
Paper develops robust neural network sensors for fuel injection quantities.
problem Adversarial noise increases error in standard neural network models for fuel injection measurements.
method Apply provable robust network learning and verification methods to fuel injection measurements.
result Provable robust model reduces mean relative error to 16.5% under sensor noise.
Noise injection regularizes Hessian, improving neural network training and generalization.
problem Regularizing over-parameterized neural networks with nonconvex and nonlinear geometry.
method Injecting isotropic Gaussian noise into weight matrices and designing a two-point estimate of the Hessian penalty.
result Effective regularization of Hessian improves generalization, achieving up to 2.4% test accuracy increase.
STAG injects noise into graph neural networks to improve performance.
problem Graph neural networks suffer from over-smoothing and limited discrimination.
method Introduces a stochastic aggregation framework (STAG) with adaptive noise injection.
result STAG models correct both over-smoothing and discrimination issues.
Adapts RL regularization techniques to prevent overfitting.
problem Preventing reinforcement learning agents from overfitting to limited training environments.
method Selective Noise Injection (SNI) and Information Bottleneck (IB) techniques.
result Significantly improved generalization performance on Coinrun benchmark.
Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise directly to the agent's parameters, which can lead to more consistent exploration and a richer set of behaviors. Methods such as evolutionary strategies use param…
Advancements in parallel processing have lead to a surge in multilayer perceptrons' (MLP) applications and deep learning in the past decades. Recurrent Neural Networks (RNNs) give additional representational power to feedforward MLPs by providing a way to treat sequential data. However, RNNs are hard to train using con…
In this paper, we focus on developing a novel mechanism to preserve differential privacy in deep neural networks, such that: (1) The privacy budget consumption is totally independent of the number of training steps; (2) It has the ability to adaptively inject noise into features based on the contribution of each to the…
Noise injection (NI) is an efficient technique to mitigate over-fitting in neural networks (NNs). The Bernoulli NI procedure as implemented in dropout and shakeout has connections with l1 and l2 regularization for the NN model parameters. We propose whiteout, a family NI regularization techniques (NIRT) through i…
Paper presents faster, robust adversarial training methods.
problem Increasing neural network robustness against adversarial attacks.
method Integrates FGSM with Pixelwise Noise Injection Layer (PNIL) and uniform noise.
result Achieves comparable results to PGD-based adversarial training but faster.
Proposes a differentially private bandit algorithm reducing noise over time.
problem Privacy concerns in interactive recommendation systems.
method Tree-based mechanism to add Laplace or Gaussian noise to model parameters, focusing on dynamic global sensitivity.
result Demonstrates (ε,δ)-differential privacy with reduced noise and improved regret. Paper tackles noisy neural networks and proposes a method to enhance their robustness.
problem Noisy neural networks struggle with random continuous noise in weights.
method Knowledge distillation combined with noise injection during training.
result Models achieve up to twice greater noise tolerance.
New method reduces uncertainty in deep neural networks with minimal computation.
problem Uncertainty in over-parameterized neural networks hinders reliability and statistical guarantees.
method Procedural-noise-correcting (PNC) predictor and resampling methods.
result Asymptotically exact-coverage confidence intervals constructed with minimal computation.
Proposes NCMN to remove feature correlation in multiplicative noise.
problem High feature correlation in multiplicative noise reduces network performance.
method Exploits batch normalization to remove correlation effect.
result Significantly improves performance on image classification tasks.
Paper finds dropout noise approximation invalid for logistic regression and neural networks.
problem Invalidity of dropout noise approximation for logistic regression and neural networks.
method Derived equivalence between dropout noise injection and L2 regularisation using divergent Taylor expansion. result Approximation approach is invalid for robust constraints and general neural network topologies.
FCNv2 robustness tested under noise and random initial conditions.
problem Assessing AI weather forecasting model robustness to input noise.
method Two experiments with varying noise levels and random initial conditions.
result FCNv2 preserves hurricane features under low to moderate noise, but underestimates intensity and persistence.
Geometry-aware noise improves model generalization on complex manifolds.
problem Improving model generalization on highly curved data manifolds.
method Add geometry-aware noise to input space, projecting Gaussian noise onto tangent space of manifold and mapping it via geodesic curve.
result Geometry-aware noise leads to improved generalization and robustness on highly curved manifolds.
Deep learning improves stochastic downscaling of climate variables.
problem Accurately capturing climatic variability at local scales.
method Proposed improvements to GANs for stochastic downscaling of climate variables.
result Improved stochastic calibration of GANs for high-resolution climate predictions.
Paper proposes NeuroAttack to undermine SNNs security through bit-flips.
problem Security and reliability issues in SNNs.
method Cross-layer attack exploiting low-level reliability issues via adversarial input noise.
result Serious integrity threat to SNNs and DNNs.
Deep ReLU networks show that 4 layers suffice for unique input recovery.
problem Injectivity capacity of deep ReLU networks.
method Developed a program connecting deep ReLU injectivity to an l-extension of the ℓ0 spherical perceptrons, using random duality theory. result Only 4 layers are needed for unique input recovery, showing expansion saturation effect.
Proposes Neural SDE for better model robustness and generalization.
problem Missing regularization mechanisms in Neural ODE networks.
method Integrates various regularization mechanisms via stochastic noise injection.
result Improves robustness and generalization compared to Neural ODE.