S-SGD adds symmetrical noise to weights to avoid sharp minima in deep learning.
problem SGD does not always converge to a flat minimum, leading to poor generalization.
method Symmetrical weight noise injection in SGD.
result S-SGD outperforms conventional SGD and weight-noise injection methods in large batch training.
Paper shows attackers can steal model weights with just noise inputs.
problem Model weight theft with minimal inputs.
method Used i.i.d. Bernoulli noise inputs to achieve high accuracy.
result Achieved high accuracy (96% for MNIST, 82% for KMNIST) with minimal inputs.
A new method improves graph-based semi-supervised classification by removing noise and mixed signs.
problem Inaccurate soft labels and noise in graph-based semi-supervised classification.
method Triple-matrix-recovery-based robust auto-weighted label propagation framework (ALP-TMR).
result Improved robustness to noise and outliers in label estimation.
Diffusion models accurately recover mixture weights from generated samples despite score function insensitivity.
problem Score-based generative models often fail to learn correct relative mode amplitudes (mixture weights) from generated samples.
method Relate diffusion score matching (DSM) loss to mixture weight estimation error, define diffusion score sensitivity index (DSSI), and prove its governing role in mixture weight recovery.
result Generated samples can accurately recover mixture weights from the DSM loss, even when the target score is insensitive to mixture weights.
New algorithm trains binary-activation, multi-level RNNs for noise-resilient, ADC-/DAC-free PIM inference.
problem Training noise-resilient, ADC-/DAC-free neural networks.
method Binary activations and multi-level weights for eNVM-based processing-in-memory circuits.
result Higher accuracy and noise resilience for recurrent networks compared to existing methods.
Gradient descent with random weights in linear regression analyzed for various noise types.
problem Analyzing the impact of random noise on gradient descent in linear regression.
method Gradient descent with randomly weighted data points, various weighting distributions, geometric moment contraction.
result Characterization of implicit regularization and non-asymptotic convergence bounds.
Study evaluates thresholds for removing noise from DNN weights using random matrix theory.
problem Removing noise from deep neural network weights for better approximation.
method Model weights as signal + noise, use random matrix theory to estimate thresholds, evaluate using cosine similarity.
result Proposed threshold estimation method improves approximation quality.
Noise injection regularizes Hessian, improving neural network training and generalization.
problem Regularizing over-parameterized neural networks with nonconvex and nonlinear geometry.
method Injecting isotropic Gaussian noise into weight matrices and designing a two-point estimate of the Hessian penalty.
result Effective regularization of Hessian improves generalization, achieving up to 2.4% test accuracy increase.
In this work, we study robust deep learning against abnormal training data from the perspective of example weighting built in empirical loss functions, i.e., gradient magnitude with respect to logits, an angle that is not thoroughly studied so far. Consequently, we have two key findings: (1) Mean Absolute Error (MAE) D…
Corrupting the input and hidden layers of deep neural networks (DNNs) with multiplicative noise, often drawn from the Bernoulli distribution (or 'dropout'), provides regularization that has significantly contributed to deep learning's success. However, understanding how multiplicative corruptions prevent overfitting ha…
CWGD measures gradient diversity weighted by curvature, improving SGD convergence.
problem Gradient noise in high-curvature directions is underestimated by standard methods.
method CWGD weights gradient diversity by the inverse square root of the Hessian.
result CWGD-Cosine reduces optimization error by up to 20% compared to standard cosine annealing.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
Analytic expressions for deep neural network output under stochastic training.
problem Understanding the impact of noise and hyperparameters on deep neural network performance.
method Taylor expansion of network output to derive analytical expressions for weights and output.
result Noise in training affects generalization by preventing the output from fully converging on train data, but does not provide explicit regularization.
Method reweights instances and classes to improve robustness in noisy data.
problem Improving deep learning performance in the presence of label noise.
method Formulates constrained optimization problems to assign importance weights to instances and class labels.
result Significant performance gains observed in benchmark datasets with label noise.
New method reduces neural network memorization of noisy labels.
problem Neural networks memorize label noise, leading to poor generalization.
method Proposes an auxiliary network that predicts gradients without labels.
result Reduces memorization of label-noise, improving generalization.
Gradient-based methods introduce noise that penalizes models sensitive to weight perturbations.
problem Noise in gradient-based optimization methods.
method Analysis of Gradient Descent (GD) and Stochastic Gradient Descent (SGD) updating all parameters simultaneously.
result Noise introduced by simultaneous parameter updates penalizes models sensitive to weight perturbations.
Diffusion models optimize objectives similar to ELBO with Gaussian noise augmentation.
problem Optimizing diffusion models for high perceptual quality.
method Showed diffusion objectives are weighted ELBOs over noise levels, with Gaussian noise augmentation.
result Diffusion objectives equate to ELBO with Gaussian noise augmentation under monotonic weighting.
TrustNet robustly learns noise patterns from trusted data to improve weakly-supervised classification.
problem Robustness to label noise in weakly-supervised learning.
method TrustNet learns noise patterns from trusted data, then trains a robust classifier using these patterns.
result TrustNet outperforms state-of-the-art methods in robustness to various noise patterns.
We consider a problem of manifold estimation from noisy observations. Many manifold learning procedures locally approximate a manifold by a weighted average over a small neighborhood. However, in the presence of large noise, the assigned weights become so corrupted that the averaged estimate shows very poor performance…
Novel model detects communities in noisy multilayer networks.
problem Understanding communities in noisy multilayer networks.
method Hierarchical variational inference for joint detection and typologizing.
result Discover communities of subjects with co-occurrent psychopathologies.
Study examines noise sensitivity of DNNs for binary classification.
problem Understanding non-robustness of DNN classifiers under noise.
method Defined and extended noise sensitivity and stability concepts for Boolean functions, applied to DNN models.
result Sorted out the relation between definitions and properties of DNN architectures under noise.
This letter presents an improved version of diffusion least mean ppower (LMP) algorithm for distributed estimation. Instead of sum of mean square errors, a weighted sum of mean square error is defined as the cost function for global and local cost functions of a network of sensors. The weight coefficients are updated b…
Study efficient learning of halfspaces with constant noise tolerance.
problem Learning halfspaces in the presence of both instance and label corruption.
method Develops an algorithm to minimize reweighted hinge loss for robustness.
result Achieves constant noise tolerance for halfspace learning.
RID-Noise improves robust design under noisy conditions using neural networks.
problem Design robustness under noisy environments.
method Robust Inverse Design under Noise (RID-Noise) using conditional invertible neural networks (cINNs).
result RID-Noise achieves more effective robust design compared to state-of-the-art methods.
Latent FxLMS accelerates ANC by adapting along low-dimensional filter weights.
problem Improving active noise control with neural adaptive filters.
method Training an auto-encoder on filter coefficients, constraining weights to latent variables, and updating in latent space.
result Latent FxLMS converges in fewer steps with comparable error to standard FxLMS.
Paper presents a robust Kalman filter for state estimation.
problem Robust state estimation under process and measurement noise.
method Generalized Bayesian approach to a Weighted Observation Likelihood Filter (WoLF) framework.
result Achieved robust state estimation against both process and measurement noise.
Nonnegative matrix factorization (NMF) is a linear dimensionality reduction technique for analyzing nonnegative data. A key aspect of NMF is the choice of the objective function that depends on the noise model (or statistics of the noise) assumed on the data. In many applications, the noise model is unknown and difficu…
Low-rank matrix factorization (LRMF) has received much popularity owing to its successful applications in both computer vision and data mining. By assuming noise to come from a Gaussian, Laplace or mixture of Gaussian distributions, significant efforts have been made on optimizing the (weighted) L1 or L2-norm los…
Heuristic weighting improves denoising score matching without requiring noise distribution assumptions.
problem Improving denoising score matching without assuming noise distribution.
method Demonstrated heteroskedasticity, derived optimal weighting functions, and provided theoretical and empirical comparisons.
result Heuristical weighting function can achieve lower variance than optimal weighting, facilitating more stable and efficient training.
Proposes a new k-NN algorithm to improve classification accuracy by removing noise and pseudo-neighbours.
problem Noise and pseudo-neighbours in large-scale databases affect k-NN performance.
method Introduces a weighted mutual k-Nearest Neighbour algorithm to detect and remove noise, and minimize distant neighbours' influence.
result The proposed algorithm provides comparative better results compared to standard k-NN.
Improves consistency of weighted averaging estimators in active learning.
problem Consistency issues in weighted averaging estimators under active learning.
method Enforces random sampling through an augmented active learning algorithm.
result Generalizes Stone's Theorem for consistency of estimators under active learning.
Differentiable model compression adds noise to parameters during training.
problem Model compression for efficient deployment.
method Adding pseudo quantization noise to model parameters during training.
result Compresses model by more than 8x on ImageNet with 0.3% accuracy loss.
NROWAN-DQN improves stability and exploration in noisy networks.
problem Noisy networks struggle with stable exploration in complex tasks.
method Noise reduction and online weight adjustment for stable actions.
result NROWAN-DQN outperforms prior algorithms in stability and exploration.
We consider the problem of learning the weighted edges of a graph by observing the noisy times of infection for multiple epidemic cascades on this graph. Past work has considered this problem when the cascade information, i.e., infection times, are known exactly. Though the noisy setting is well motivated by many epide…
Analog BNNs perform similarly regardless of noise distribution shape.
problem Difficulty in precisely controlling noise distribution shape in analog devices.
method Used real device noise as the variational distribution in MFVI training.
result Predictive distributions converge to the same distribution regardless of noise shape.
The paper proposes a novel Kernelized image segmentation scheme for noisy images that utilizes the concept of Smallest Univalue Segment Assimilating Nucleus (SUSAN) and incorporates spatial constraints by computing circular colour map induced weights. Fuzzy damping coefficients are obtained for each nucleus or center p…
Theoretical analysis of the error landscape of deep neural networks has garnered significant interest in recent years. In this work, we theoretically study the importance of noise in the trajectories of gradient descent towards optimal solutions in multi-layer neural networks. We show that adding noise (in different wa…
Convolutional neural networks (CNNs) are known for their good performance and generalization in vision-related tasks and have become state-of-the-art in both application and research-based domains. However, just like other neural network models, they suffer from a susceptibility to noise and adversarial attacks. An adv…
Convolutional sparse coding (CSC) can learn representative shift-invariant patterns from multiple kinds of data. However, existing CSC methods can only model noises from Gaussian distribution, which is restrictive and unrealistic. In this paper, we propose a general CSC model capable of dealing with complicated unknown…
Adapts example weights to optimize black-box metrics.
problem Optimizing metrics defined by black-box functions.
method Adaptive example weighting and iterative post-shifting.
result Improves classification performance compared to baselines.
Given full or partial information about a collection of points that lie close to a union of several subspaces, subspace clustering refers to the process of clustering the points according to their subspace and identifying the subspaces. One popular approach, sparse subspace clustering (SSC), represents each sample as a…
New mechanism protects neural network weights from privacy attacks during self-supervised learning.
problem Privacy risks during fine-tuning stage of self-supervised learning.
method Proposes a novel differential privacy mechanism using additive logistic noise.
result Reduces membership inference attack accuracy to 50% while maintaining below 5% performance loss.
Many state-of-the-art machine learning models such as deep neural networks have recently shown to be vulnerable to adversarial perturbations, especially in classification tasks. Motivated by adversarial machine learning, in this paper we investigate the robustness of sparse regression models with strongly correlated co…
It has been shown that injecting noise into the neural network weights during the training process leads to a better generalization of the resulting model. Noise injection in the distributed setup is a straightforward technique and it represents a promising approach to improve the locally trained models. We investigate…
New findings show that minimal feature diversity at neural network initialization is harmful but can be mitigated with noise.
problem The importance of feature diversity at neural network initialization.
method A series of experiments comparing different initialization schemes, including adding noise.
result Minimal feature diversity is harmful but can be mitigated with noise, even standard GPU noise is sufficient.
Improves deep learning performance on noisy datasets using inverse-variance weighting.
problem Heteroscedastic regression with varying noise levels.
method Batch Inverse-Variance (BIV) loss function for neural networks.
result Significantly improves network performance on noisy datasets compared to other methods.
A new method for robust training under label noise using weighted gradient descent.
problem Overfitting to noisy examples in machine learning.
method Exponentiated gradient reweighting for flexible handling of noisy data.
result Improved generalization in noisy classification and PCA problems.
We introduce NoisyNet, a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent's policy can be used to aid efficient exploration. The parameters of the noise are learned with gradient descent along with the remaining network weights. NoisyNet …