Noise in SGD affects overparameterized models, favoring sparse solutions.
problem Understanding and mitigating implicit bias in SGD with parameter-dependent noise.
method Theoretical analysis of a quadratically-parameterized model with label noise and Gaussian noise.
result SGD with label noise recovers sparse ground-truth solutions, while SGD with Gaussian noise overfits dense solutions.
Study on noise models for noisy labels in NLP.
problem Quality of noise models from noisy labels.
method Theoretical analysis and synthetic dataset creation.
result Expected error of noise models derived.
Using a Bayesian approach, we consider the problem of recovering sparse signals under additive sparse and dense noise. Typically, sparse noise models outliers, impulse bursts or data loss. To handle sparse noise, existing methods simultaneously estimate the sparse signal of interest and the sparse noise of no interest.…
Because large, human-annotated datasets suffer from labeling errors, it is crucial to be able to train deep neural networks in the presence of label noise. While training image classification models with label noise have received much attention, training text classification models have not. In this paper, we propose an…
Enhanced consistency bounds derived for classification under a new noise condition.
problem Enhanced consistency bounds for classification under a new noise condition.
method Model Margin Noise (MM noise) assumption, derived enhanced H-consistency bounds.
result Enhanced H-consistency bounds under MM noise condition, interpolates between linear and square-root regimes.
Paper proposes a self-supervised method to denoise autoregressive signals with heavy-tailed noise.
problem Denoising autoregressive signals corrupted by heavy-tailed noise.
method Self-supervised learning approach without requiring full noise distribution knowledge.
result Strong denoising performance compared to baseline methods, especially for impulsive noise.
New method designs joint initial noises for diffusion models to improve diversity and alignment.
problem Independent initial noises limit diversity in generated images.
method Coupling of initial noises, maintaining Gaussian distribution while allowing dependence.
result Repulsive Gaussian coupling improves diversity without increasing sampling cost.
Two models incorporate market microstructure noise into asset pricing and option valuation.
problem Effect of market microstructure noise on asset pricing and option valuation.
method Developed two models: a continuous-time Black-Scholes-Merton model and a discrete binomial tree model.
result Extracted coefficients to quantify noise impact on volatility and drift.
New insights on robust learning under strong noise models.
problem Challenging label-noise models in robust learning.
method Extending statistical query framework to more general noise models and using evolutionary algorithms.
result First polynomial time algorithm for learning linear threshold functions with arbitrarily small excess error in presence of Tsybakov noise.
Deep-neural-network (DNN) based noise suppression systems yield significant improvements over conventional approaches such as spectral subtraction and non-negative matrix factorization, but do not generalize well to noise conditions they were not trained for. In comparison to DNNs, humans show remarkable noise suppress…
Alternative to likelihood-based LSNM model selection, residual independence testing is more robust to noise misspecification.
problem Cause-effect inference in location-scale noise models with misspecified noise distributions.
method Residual independence testing as an alternative to likelihood-based model selection.
result Residual independence testing is more robust to noise misspecification.
We present a noise-injected version of the Expectation-Maximization (EM) algorithm: the Noisy Expectation Maximization (NEM) algorithm. The NEM algorithm uses noise to speed up the convergence of the EM algorithm. The NEM theorem shows that injected noise speeds up the average convergence of the EM algorithm to a local…
Develops efficient inference for noise heterogeneity in machine learning models.
problem Downstream procedures based on residuals can be biased in additive noise models.
method Semiparametrically efficient inference using a novel Hilbert-valued one-step estimator.
result Constructs tests and confidence intervals for residual independence and goodness of fit.
Existence of incomplete Radner equilibrium with endogenous noise tracker.
problem Existence of incomplete Radner equilibrium in a model with endogenous noise tracker.
method Proved existence through a coupled system of ODEs, reduced to two coupled ODEs.
result Endogenous noise tracker leads to higher aggregate welfare for large stock supply.
Proposes a new model for noisy labels considering multiple labelers and adversarial attacks.
problem Real-world noisy label models with multiple labelers and adversarial attacks.
method Labeler-dependent noise model with adversarial attack vectors.
result State-of-the-art approaches for learning from noisy labels are defeated by adversarial label attacks.
We introduce a new approach for designing computationally efficient learning algorithms that are tolerant to noise, and demonstrate its effectiveness by designing algorithms with improved noise tolerance guarantees for learning linear separators. We consider both the malicious noise model and the adversarial label nois…
Additive asynchronous and cyclostationary impulsive noise limits communication performance in OFDM powerline communication (PLC) systems. Conventional OFDM receivers assume additive white Gaussian noise and hence experience degradation in communication performance in impulsive noise. Alternate designs assume a parametr…
It has been shown that injecting noise into the neural network weights during the training process leads to a better generalization of the resulting model. Noise injection in the distributed setup is a straightforward technique and it represents a promising approach to improve the locally trained models. We investigate…
We propose a robust method to estimate heteroscedastic noise models using Student's t-distribution.
problem Identifying cause and effect from bivariate observational data with non-Gaussian noise.
method We propose a novel approach using Student's t-distribution to estimate heteroscedastic noise models, which is more robust and achieves better performance.
result Our estimators are more robust and achieve better overall performance across synthetic and real benchmarks.
Paper improves image classification accuracy with a new Noise Modeling Network.
problem Improving performance of multi-label image classifiers with noisy or missing labels.
method Integrates a Noise Modeling Network (NMN) with a CNN to jointly learn noise distribution and CNN parameters.
result Consistently improves classification performance on MSR-COCO and MSR-VTT datasets.
Geometry-aware noise improves model generalization on complex manifolds.
problem Improving model generalization on highly curved data manifolds.
method Add geometry-aware noise to input space, projecting Gaussian noise onto tangent space of manifold and mapping it via geodesic curve.
result Geometry-aware noise leads to improved generalization and robustness on highly curved manifolds.
Antithetic noise improves diffusion models' uncertainty quantification.
problem Improving uncertainty quantification in diffusion models.
method Pairing each noise sample with its negation, leading to strong negative correlation.
result Substantially more reliable uncertainty quantification with up to 90% narrower confidence intervals.
NoiseRank reduces label noise without supervision, improving classification accuracy.
problem Label noise in datasets from noisy channels.
method NoiseRank uses Markov Random Fields to estimate and rank instances based on their noise probability.
result NoiseRank improves classification accuracy on noisy datasets.
This paper connects noise injection to Bayesian inference for neural networks, improving model uncertainty.
problem Improving the reliability and confidence of neural network predictions through uncertainty quantification.
method Introducing noise into neural network parameters during training and inference to estimate prediction uncertainty.
result The MCNI method outperforms baseline models in regression and classification tasks.
Diffusion models generate new samples by adding and removing noise.
problem Generating new samples from data.
method Apply noise to data, reverse the process to generate new samples.
result Diffusion models can improve classifier performance on imbalanced data.
DA-GNN improves robustness of GNNs by modeling noise dependencies.
problem Real-world graph node features often contain noise, leading to performance degradation in GNNs.
method DA-GNN captures noise dependencies using variational inference and new benchmark datasets.
result DA-GNN consistently outperforms existing baselines across various noise scenarios.
New method tackles label noise on imbalanced datasets by considering class-specific uncertainty.
problem Label noise and class imbalance in imbalanced datasets.
method Epistemic and aleatoric uncertainty-aware class-specific noise modeling.
result Proposed ULC framework improves performance on imbalanced datasets.
Anti-correlated noise improves machine learning model generalization.
problem Improving machine learning model generalization.
method Injecting anticorrelated noise into gradient descent steps.
result Anti-correlated noise leads to better model generalization than uncorrelated noise.
In learning with noisy labels, for every instance, its label can randomly walk to other classes following a transition distribution which is named a noise model. Well-studied noise models are all instance-independent, namely, the transition depends only on the original label but not the instance itself, and thus they a…
We present and study models of adversarial online learning where the feedback observed by the learner is noisy, and the feedback is either full information feedback or bandit feedback. Specifically, we consider binary losses xored with the noise, which is a Bernoulli random variable. We consider both a constant noise r…
Study evaluates how noise affects ANMs' ability to identify causal directions.
problem Challenges in identifying causal relationships in bivariate cases with noise.
method Empirical study using Regression with Subsequent Independence Test (RESIT) on various ANM models.
result ANMs can fail to identify true causal directions for certain noise levels.
This paper studies how label noise affects Federated Learning.
problem The impact of label noise on Federated Learning.
method The paper derives an upper bound for the generalization error and conducts experiments on MNIST and CIFAR-10 datasets.
result The global model accuracy decreases linearly with increasing label noise, consistent with theoretical analysis.
The study analyzes how label noise affects deep learning feature learning.
problem The impact of label noise on deep learning feature learning.
method Theoretical analysis of a two-layer convolutional neural network under noisy label conditions.
result Two key stages identified: signal learning in Stage I and noise memorization in Stage II.
Unified framework for word embedding models using noise examples.
problem Improving word embedding models with negative sampling.
method Formulated a Word-Context Classification (WCC) framework that generalizes SkipGram word embedding models.
result The best noise distribution is the data distribution, improving both performance and training speed.
A new method handles label noise by leveraging causal information.
problem Label noise degrades deep learning performance.
method Proposes a novel generative approach using a structural causal model.
result Improves classifier performance on label-noise datasets.
In the geophysical field, seismic noise attenuation has been considered as a critical and long-standing problem, especially for the pre-stack data processing. Here, we propose a model to leverage the deep-learning model for this task. Rather than directly applying an existing de-noising model from ordinary images to th…
Measures three types of noise in LLM evaluations.
problem Separating signal from noise in LLM experiments.
method Defined and measured three types of noise: prediction, data, and total noise. Proposed the all-pairs paired method for statistical power.
result Total noise level is characteristic and predictable across all model pairs.
While noise is commonly considered a nuisance in computing systems, a number of studies in neuroscience have shown several benefits of noise in the nervous system from enabling the brain to carry out computations such as probabilistic inference as well as carrying additional information about the stimuli. Similarly, no…
Noise stability improves understanding of Transformer models.
problem Lack of robustness metrics for real-valued domains and junta-like input dependence in modern LLMs.
method Proposed noise stability as a new metric and developed a practical regularization method.
result Noise stability regularization method accelerates training by 35-75%.
New auto-encoder handles varying noise levels without retraining.
problem Auto-encoders degrade in noisy conditions.
method Formalized auto-encoders as transform learning, derived new architecture.
result Models generalize well to different noise levels.
Multi-view subspace learning (MSL) aims to find a low-dimensional subspace of the data obtained from multiple views. Different from single view case, MSL should take both common and specific knowledge among different views into consideration. To enhance the robustness of model, the complexity, non-consistency and simil…
Boosting algorithm reduces error in noisy data.
problem Improving weak learners in the presence of Massart noise.
method First computationally efficient boosting algorithm for Massart noise.
result Achieves misclassification error arbitrarily close to Massart noise threshold.
Label smoothing improves model performance even with noisy labels.
problem Mitigating label noise in deep learning models.
method Examined label smoothing as a technique to cope with label noise and compared it to loss-correction methods.
result Label smoothing is competitive with loss-correction techniques under label noise and beneficial for distillation from noisy data.
SAP corrects model for label noise by identifying and removing noisy samples.
problem Label corruption degrades model performance; acquiring perfect labels is costly.
method SAP uses SVD to identify and project model weights onto a clean activation space.
result SAP improves model generalization by up to 6% on CIFAR dataset with 25% synthetic corruption.
Inserting label noise can improve model accuracy and fairness.
problem Improving model accuracy and fairness with noisy labels.
method Increasing label noise rates to balance and detect noisy instances.
result Inserting label noise can lead to more accurate and fair models.
In this paper, we build tests for the presence of residual noise in a model where the market microstructure noise is a known parametric function of some variables from the limit order book. The tests compare two distinct quasi-maximum likelihood estimators of volatility, where the related model includes a residual nois…
We study a generalization of the Heston model, which consists of two coupled stochastic differential equations, one for the stock price and the other one for the volatility. We consider a cubic nonlinearity in the first equation and a correlation between the two Wiener processes, which model the two white noise sources…
Real-world measurement noise in applications like robotics is often correlated in time, but we typically assume i.i.d. Gaussian noise for filtering. We propose general Gaussian Processes as a non-parametric model for correlated measurement noise that is flexible enough to accurately reflect correlation in time, yet sim…