Noise in linear networks minimizes sharpness and leads to shrinkage-thresholding.
problem Minimizing sharpness in diagonal linear networks.
method Stochastic sharpness-aware minimization (SAM) with isotropic noise.
result Noise forces shrinkage-thresholding of true parameters.
Noise-ignorant empirical risk minimization achieves state-of-the-art performance on noisy data.
problem Learning with noisy labels in multi-class classification problems.
method Introducing relative signal strength (RSS) to quantify transferability and applying Noise Ignorant Empirical Risk Minimization (NI-ERM).
result NI-ERM achieves state-of-the-art performance on CIFAR-N data challenge.
Using a recently developed method of noise level estimation that makes use of properties of the coarse grained-entropy we have analyzed the noise level for the Dow Jones index and a few stocks from the New York Stock Exchange. We have found that the noise level ranges from 40 to 80 percent of the signal variance. The c…
Noise makes data unlearnable by tricking models.
problem Unauthorized exploitation of personal data by deep learning models.
method Error-minimizing noise to reduce training examples' learnability.
result Error-minimizing noise can make training examples unlearnable by deep learning models.
New research shows that binary classification can be done with noisy data, but only if there are clean samples available.
problem Learning binary classification with instance and label dependent label noise.
method Theoretical analysis and empirical risk minimization.
result Empirical risk minimization achieves the optimal excess risk bound without additional assumptions.
We consider the problem of minimizing a convex objective function F when one can only evaluate its noisy approximation F^. Unless one assumes some structure on the noise, F^ may be an arbitrary nonconvex function, making the task of minimizing F intractable. To overcome this, prior work has often focu…
New privacy mechanism reduces error in query results.
problem Achieving privacy while minimizing noise in query results.
method Extended sufficient and necessary condition for (ε,δ)-differential privacy for symmetric and log-concave noise densities. result Significantly lower mean squared errors than Laplace and Gaussian mechanisms.
RAD improves robustness to domain annotation noise without explicit domain annotations.
problem Robustness to domain annotation noise in training data.
method Regularized Annotation of Domains (RAD) for last layer retraining.
result RAD outperforms state-of-the-art methods even with 5% noise in training data.
In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and…
Unhinged loss minimization fails to improve classifier accuracy for simple data.
problem Accuracy of classifiers minimizing the unhinged loss.
method Minimizing the unhinged loss function.
result Minimizing the unhinged loss yields classifiers with accuracy no better than random guessing for simple data.
Self-adaptive training improves deep learning robustness.
problem Improving deep learning performance on corrupted data.
method Dynamic correction of problematic labels using model predictions.
result Self-adaptive training significantly improves generalization over ERM under various levels of noise.
Label noise in SGD helps converge to flatter minima.
problem Improving generalization in overparametrized models.
method Analyzes SGD with label noise, showing convergence to regularized minima.
result SGD with label noise converges to flatter minima, improving generalization.
Improved privacy and utility in machine learning with adaptive differential privacy.
problem Enhancing privacy in machine learning models while maintaining utility.
method Adaptive differentially private (ADP) learning method that optimally adapts noise to stepsize.
result ADP method significantly improves utility compared to standard differentially private methods.
Time changes of noise level at Warsaw Stock Market are analyzed using a recently developed method basing on properties of the coarse grained entropy. The condition of the minimal noise level is used to build an efficient portfolio. Our noise level approach seems to be a much better tool for risk estimations than standa…
AGNES accelerates gradient descent with noisy gradients.
problem Minimizing smooth convex and strongly convex functions with noisy gradients.
method Generalization of Nesterov's accelerated gradient descent algorithm for noisy conditions.
result AGNES achieves acceleration for noisy gradients with a constant of proportionality up to 1.
The ability to detect sparse signals from noisy high-dimensional data is a top priority in modern science and engineering. A sparse solution of the linear system Aρ=b0 can be found efficiently with an l1-norm minimization approach if the data is noiseless. Detection of the signal's support from data corrupted b…
This work improves structured prediction by learning the balance between signal and random noise.
problem Structured prediction with random perturbations.
method Learning the variance of randomized structured predictors to balance signal and noise.
result Learning the balance improves structured prediction effectiveness.
CMRM improves robustness in noisy label settings without requiring privileged knowledge.
problem Learning with noisy labels without privileged knowledge.
method Conformal Margin Risk Minimization (CMRM) framework.
result CMRM consistently improves accuracy and reduces mislabeling under various noise conditions.
Scaled sparse linear regression jointly estimates the regression coefficients and noise level in a linear model. It chooses an equilibrium with a sparse regression method by iteratively estimating the noise level via the mean residual square and scaling the penalty in proportion to the estimated noise level. The iterat…
Efficiently learns halfspaces with malicious noise, near-optimal label complexity.
problem Learning s-sparse halfspaces under malicious label noise. method Active learning algorithm with instance reweighting and empirical risk minimization.
result Near-optimal label complexity of O(slog4d/ε) and noise tolerance Ω(ε). New error bounds for noisy phase retrieval problems using empirical risk minimization.
problem Estimating signals in noisy phase retrieval problems.
method Empirical ℓ2 risk minimization (ERM) with new error bounds for different noise patterns. result Established new error bounds for NPR and NGPR, showing improved performance under various noise conditions.
Unbiased methods for alpha-divergence minimization struggle in high dimensions.
problem The difficulty of unbiased alpha-divergence minimization in high dimensions.
method Signal-to-Noise Ratio (SNR) analysis of gradient estimators.
result The SNR of the gradient estimator worsens exponentially with dimensionality.
Nonnegative matrix factorization (NMF) is a linear dimensionality reduction technique for analyzing nonnegative data. A key aspect of NMF is the choice of the objective function that depends on the noise model (or statistics of the noise) assumed on the data. In many applications, the noise model is unknown and difficu…
Paper presents robust boosting methods for label noise.
problem Boosting methods degrade in noisy environments.
method Robust Minimax Boosting (RMBoost) with theoretical guarantees.
result RMBoost provides strong classification accuracy and robustness.
Enhanced consistency bounds derived for classification under a new noise condition.
problem Enhanced consistency bounds for classification under a new noise condition.
method Model Margin Noise (MM noise) assumption, derived enhanced H-consistency bounds.
result Enhanced H-consistency bounds under MM noise condition, interpolates between linear and square-root regimes.
Volatility dynamics of wavelet - filtered stock price time series is studied. Using the universal thresholding method of wavelet filtering and a principle of minimal linear autocorrelation of noise component we find that the quantitative characteristics of volatility dynamics of denoised series are noticeably different…
We study the robustness properties of ℓ1 norm minimization for the classical linear regression problem with a given design matrix and contamination restricted to the dependent variable. We perform a fine error analysis of the ℓ1 estimator for measurements errors consisting of outliers coupled with noise. We…
New method learns vector fields from noisy time series data.
problem Learning vector fields from noisy time series data.
method Neural network architecture with tensor products of one-dimensional neural shape functions for vector field approximation, alternating minimization for noise handling.
result Neural shape function architecture robust to noise, learning accurate vector fields from data with up to 10% Gaussian noise.
Continuous-time analysis shows SGD with noise prefers flat minima.
problem Optimizing neural networks using SGD with noise.
method Continuous-time model for SGD with noise analysis.
result Optimization prefers flat minima in certain noise regimes.
Optimal test for detecting signal in noisy matrix model.
problem Signal detection in noisy matrix models with unknown rank.
method Hypothesis test based on linear spectral statistics, optimal under Gaussian noise.
result Optimal test under Gaussian noise, improved with non-Gaussian noise.
Using integration by parts on Gaussian space we construct a Stein Unbiased Risk Estimator (SURE) for the drift of Gaussian processes using their local and occupation times. By almost-sure minimization of the SURE risk of shrinkage estimators we derive an estimation and de-noising procedure for an input signal perturbed…
Label noise SGD converges to a simple model with a single linear feature.
problem Understanding the simplicity bias in neural network training.
method Analyzing the convergence of label noise SGD on two-layer neural networks.
result Label noise SGD converges to a model with a single linear feature.
PANDA augments data to regularize GLM estimation and inference.
problem Regularizing estimation and inference in GLMs with noisy data.
method Iteratively optimizes augmented noise data to converge to regularized model estimates.
result Established convergence and asymptotic distributions for regularized parameters.
Study efficient learning of halfspaces with constant noise tolerance.
problem Learning halfspaces in the presence of both instance and label corruption.
method Develops an algorithm to minimize reweighted hinge loss for robustness.
result Achieves constant noise tolerance for halfspace learning.
S-SGD adds symmetrical noise to weights to avoid sharp minima in deep learning.
problem SGD does not always converge to a flat minimum, leading to poor generalization.
method Symmetrical weight noise injection in SGD.
result S-SGD outperforms conventional SGD and weight-noise injection methods in large batch training.
Shot-Noise processes constitute a useful tool in various areas, in particular in finance. They allow to model abrupt changes in a more flexible way than processes with jumps and hence are an ideal tool for modelling stock prices, credit portfolio risk, systemic risk, or electricity markets. Here we consider a general f…
SAM improves neural network generalization better than SGD, especially in noisy data.
problem Overfitting in large neural networks with label noise.
method Sharpness-Aware Minimization (SAM) compared to Stochastic Gradient Descent (SGD).
result SAM prevents noise learning and facilitates feature learning better than SGD.
Noise in imputed values corrects biases in machine learning models.
problem Systematic biases in imputed values affect downstream analyses.
method Introducing noise to imputed values to correct biases.
result Noise-corrected imputation methods produce unbiased estimates.
Proposes a method to improve classification robustness against label noise.
problem Improving classification robustness against label noise.
method Extends a simple regression approach to classification, modeling labels as samples from a logistic-normal distribution.
result The method improves robustness against label noise in classification.
Improves signal detection in non-Gaussian noise using transformed data.
problem Signal detection in rank-one signal-plus-noise data matrices.
method Pre-transforming matrix entries and using linear spectral statistics for hypothesis testing.
result Sharp phase transition of largest eigenvalues in spiked rectangular matrices.
Unified approach for first-order methods with Markovian noise in stochastic optimization and variational inequalities.
problem Stochastic optimization problems with Markovian noise.
method Unified theoretical analysis of first-order gradient methods using randomized batching and multilevel Monte Carlo.
result Optimal (linear) dependence on the mixing time of the noise sequence, eliminating previous limiting assumptions.
We propose an AdaPtive Noise Augmentation (PANDA) technique to regularize the estimation and construction of undirected graphical models. PANDA iteratively optimizes the objective function given the noise augmented data until convergence to achieve regularization on model parameters. The augmented noises can be designe…
While noise is commonly considered a nuisance in computing systems, a number of studies in neuroscience have shown several benefits of noise in the nervous system from enabling the brain to carry out computations such as probabilistic inference as well as carrying additional information about the stimuli. Similarly, no…
The process of data mining with differential privacy produces results that are affected by two types of noise: sampling noise due to data collection and privacy noise that is designed to prevent the reconstruction of sensitive information. In this paper, we consider the problem of designing confidence intervals for the…
Paper introduces a new cost function to improve deep learning model generalization.
problem Overfitting and poor extrapolation of deep learning models.
method Introduces a 'whitening' cost function based on the Ljung-Box statistic.
result Significant improvement in generalization for RNNs and image autoencoders.
NAPP-ERM improves ERM with differential privacy guarantees by iteratively achieving target regularization and delivering strong convexity.
problem Over-regularization in privacy-preserving ERM approaches.
method Noise-Augmented Privacy-Preserving Empirical Risk Minimization (NAPP-ERM) with a dual-purpose l2 regularizer and privacy budget retrieval strategy.
result Mitigates over-regularization and achieves strong convexity through a single regularizer.
A method for noise reduction in functional time series using FPCA.
problem Noise contamination in functional time series.
method Extending FPCA to separate signal and noise components.
result Optimal projection minimizes mean integrated squared error.
Noise injection (NI) is an efficient technique to mitigate over-fitting in neural networks (NNs). The Bernoulli NI procedure as implemented in dropout and shakeout has connections with l1 and l2 regularization for the NN model parameters. We propose whiteout, a family NI regularization techniques (NIRT) through i…