Optimizes SGLD noise structure for better generalization bounds.
problem Improving generalization bounds for large models trained with SGLD.
method Manipulates the noise structure in SGLD to optimize information-theoretical bounds.
result Optimal noise covariance is the square root of the expected gradient covariance under certain constraints.
Develops efficient inference for noise heterogeneity in machine learning models.
problem Downstream procedures based on residuals can be biased in additive noise models.
method Semiparametrically efficient inference using a novel Hilbert-valued one-step estimator.
result Constructs tests and confidence intervals for residual independence and goodness of fit.
Paper estimates noise covariance in correlated multi-task linear models.
problem Estimating noise covariance in multi-task high-dimensional linear models with correlated noise.
method Uses multi-task elastic-net and lasso estimators to estimate noise covariance, correcting bias in squared residual matrix.
result Develops a novel estimator of noise covariance that converges at rate n−1/2, matching oracle estimator under suitable conditions. New method clusters high-dimensional data with anisotropic noise.
problem Clustering high-dimensional anisotropic mixtures with varying noise structures.
method Covariance Projected Spectral Clustering (COPO) method that projects data onto a low-dimensional space and reassigns clusters based on estimated covariances.
result COPO achieves minimax-optimal misclustering rates in Gaussian settings.
The gradient noise of SGD is considered to play a central role in the observed strong generalization abilities of deep learning. While past studies confirm that the magnitude and the covariance structure of gradient noise are critical for regularization, it remains unclear whether or not the class of noise distribution…
Efficiently estimates sparse linear regression with heavy-tailed data and outliers.
problem Sparse estimation of linear regression coefficients with heavy-tailed covariates and noises, including outliers.
method Efficient computation of robust estimator with nearly optimal error bound.
result Nearly optimal error bound for robust sparse estimation.
Efficiently estimates sparse linear regression with heavy-tailed and outlier-contaminated data.
problem Estimating sparse linear regression coefficients with heavy-tailed and outlier-contaminated data.
method Efficient computation of estimators with sharp error bounds.
result Sharp error bounds for efficient estimators.
Improved Kalman filtering with hierarchical variational approach.
problem Inconsistent process covariance estimation and slow convergence speed in traditional variational Kalman filtering.
method Introducing a surrogate variable for process-noise-free state, reformulating CAVI, and sliding-window hyperparameter estimation.
result Enhanced convergence speed and superior estimation accuracy compared to existing methods.
Enhances UPSA to reduce noise in financial data.
problem Noise in financial data affects UPSA's performance.
method Time-averaging optimal penalty weights and using Average Oracle correlation eigenvalues.
result Combining time-averaging and Average Oracle correlation eigenvalues improves UPSA's performance.
Robustly estimates linear regression coefficients with adversarial and noisy data.
problem Estimating robust linear regression coefficients with adversarial and noisy data.
method Adversarial robust weighted Huber regression with polynomial computational complexity.
result Derives an estimation error bound that depends on the stable rank and condition number of the covariance matrix.
A new method learns noise characteristics for better state estimation in real-time systems.
problem Challenges in accurately estimating states due to uncertainty in process and measurement models.
method Proposes a learning-based approach with different loss functions to identify noise characteristics.
result Demonstrates improved performance in real-time vehicle state estimation.
AniDS improves molecular force field modeling by learning anisotropic noise.
problem Molecular force field modeling suffers from oversimplified assumptions about atomic motions.
method AniDS introduces anisotropic noise generation for better modeling of directional and structural variability.
result AniDS outperforms existing methods on benchmarks, achieving significant improvements in force prediction accuracy.
Paper proposes a new covariance estimator ensuring positive semi-definite matrices.
problem Estimating spot covariance matrices while maintaining positive semi-definiteness.
method Modification of the Fourier covariance estimator with a symmetric positive semi-definite constraint.
result The estimator is consistent and produces accurate positive semi-definite matrices.
Noise in SGD affects overparameterized models, favoring sparse solutions.
problem Understanding and mitigating implicit bias in SGD with parameter-dependent noise.
method Theoretical analysis of a quadratically-parameterized model with label noise and Gaussian noise.
result SGD with label noise recovers sparse ground-truth solutions, while SGD with Gaussian noise overfits dense solutions.
Safety filter for unknown discrete-time systems with learned models and noise covariance.
problem Ensuring safety for unknown discrete-time linear systems with Gaussian noise.
method Develops a learning-based safety filter using empirical model and noise covariance, optimizing control actions to stay within safety constraints.
result Minimally modifies nominal control actions to ensure safety with high probability, tightening constraints as more data is collected.
The exact meaning of the noise spectrum of eigenvalues of the covariance matrix is discussed. In order to better understand the possible phenomena behind the observed noise, the spectrum of eigenvalues of the covariance matrix is studied under a model where most of the true eigenvalues are zero and the parameters are n…
Study mini-batch SGD noise and its limits, proving complexity guarantees.
problem Analyzing the noise in mini-batch SGD and its impact on optimization.
method Examined the conditional covariance and diffusion limits of SGD under different sampling designs.
result Proved mean-square upper bounds and Fisher van Trees lower bounds for SGD, linking them to effective dimension and condition number.
The paper proposes using network science to improve portfolio optimization by reducing noise in covariance estimation.
problem Noise in covariance estimation leads to suboptimal portfolio performance.
method The paper introduces SR-IFN, a network-based method to filter out noise from empirical covariance, enhancing portfolio optimization.
result The SR-IFN network improves portfolio performance by selecting peripheral, diversified assets and inversely weighting them based on centrality.
SVDD and Deep SVDD improve radar target detection in clutter.
problem Clutter and thermal noise degrade classical radar detection methods.
method Support Vector Data Description (SVDD) and Deep SVDD for one-class learning.
result SVDD and Deep SVDD outperform traditional methods on simulated radar data.
Improved image classification accuracy with a probabilistic model of label noise.
problem Noisy labels in large-scale image classification datasets.
method A probabilistic model using a multivariate Normal distribution on the final hidden layer of a neural network, capturing input-dependent label noise.
result Significantly improved accuracy on various datasets compared to standard methods.
Study mutual info for community detection with covariate and correlated networks.
problem Community detection with covariate and correlated networks.
method Asymptotic upper bound and MMSE matrix heuristic analysis.
result Explicit characterization of combined information effects.
The choice of batch-size in a stochastic optimization algorithm plays a substantial role for both optimization and generalization. Increasing the batch-size used typically improves optimization but degrades generalization. To address the problem of improving generalization while maintaining optimal convergence in large…
We interpret the variational inference of the Stochastic Gradient Descent (SGD) as minimizing a new potential function named the \textit{quasi-potential}. We analytically construct the quasi-potential function in the case when the loss function is convex and admits only one global minimum point. We show in this case th…
Method estimates noise variance in Gaussian process regression.
problem Estimating noise variance in Gaussian process regression models.
method Reduces hyperparameter space, uses marginal likelihood function, derives bounds and asymptotes.
result Computational advantages and robustness compared to traditional methods.
Study predictive performance of linear regression with random functional covariates.
problem Theoretical predictive performance of linear regression with random functional covariates.
method Theoretical analysis of ridge and ridge-less least-squares regression with random functional covariates.
result Probabilistic bounds on predictive excess risk for random functional covariates.
Recent studies inspired by results from random matrix theory [1,2,3] found that covariance matrices determined from empirical financial time series appear to contain such a high amount of noise that their structure can essentially be regarded as random. This seems, however, to be in contradiction with the fundamental r…
Unified framework for isotropic SG noise in posterior sampling.
problem Bayesian posterior sampling with practical and robust methods.
method Designing a novel, isotropic SG noise approach with fixed learning rate.
result Competitive and practical method compared to state-of-the-art.
GATs improve node regression on noisy graphs with provable advantage.
problem Improving node regression on graphs with noisy covariates and edges.
method Proposes a GAT designed for denoising proxy features in node regression.
result GAT achieves lower error in estimating regression coefficient and predicting responses.
Method selects significant spatial covariates in noisy data.
problem Identifying true spatial covariates in noisy data.
method Combines sparsity-promoting estimation with noise-robust model selection.
result Method reliably recovers true covariates under diverse noise scenarios.
Extends neural network training framework to handle noise and uncertainty.
problem Handling noise and uncertainty in neural network training.
method Integrates non-zero aleatoric noise and derives posterior covariance for epistemic uncertainty.
result Derives an estimator for posterior covariance, providing a handle on epistemic uncertainty.
Noise addition prevents overfitting in adaptive data analysis.
problem Overfitting in repeated use of a data sample via adaptively chosen queries.
method Simple noise addition algorithms and differential privacy-based analysis.
result Noise-addition algorithms provide variance-dependent guarantees for unbounded queries.
Study non-asymptotic estimation bounds for LTI models with Gaussian noise.
problem Estimating parameters of LTI models with non-asymptotic error bounds.
method Sharp non-asymptotic lower bounds using Cramér-Rao and van Trees inequalities, concentration results, and differential geometric constructions.
result Sharp and rate-optimal lower bounds for mean square estimation risk.
We propose a method to clean covariance matrices of nonstationary systems by using time-independent eigenvalues.
problem Noise in covariance matrices of nonstationary systems with time-independent eigenvalues.
method Data-driven approach to use independent eigenvalues encoding long-term influence of future on present.
result Our method outperforms optimal stationary methods for filtering covariance matrix and its inverse.
We consider the problem of estimating the mean and covariance of a distribution from iid samples in Rn, in the presence of an η fraction of malicious noise; this is in contrast to much recent work where the noise itself is assumed to be from a distribution of known type. The agnostic problem includes many…
Improved covariate shift handling with node-based Bayesian neural networks.
problem Improving generalization under covariate shift in neural networks.
method Introduced node-based Bayesian neural networks that learn latent noise variables to represent input corruptions.
result Node-based BNNs perform well under covariate shift due to input perturbations, improving uncertainty estimation and robustness.
The problem of image restoration in cryo-EM entails correcting for the effects of the Contrast Transfer Function (CTF) and noise. Popular methods for image restoration include `phase flipping', which corrects only for the Fourier phases but not amplitudes, and Wiener filtering, which requires the spectral signal to noi…
WS diffusion models handle anisotropic Gaussian noise better than conventional methods.
problem Handling anisotropic Gaussian noise in imaging inverse problems.
method Whitened Score (WS) diffusion models based on stochastic differential equations.
result WS DMs outperform conventional DMs on anisotropic Gaussian noise.
Safe learning in uncertain systems with state measurements and optimization.
problem Safe learning in nonlinear control-affine systems with unknown additive uncertainty.
method Model uncertainty as Gaussian noise, learn mean and covariance, use optimization to adjust control input.
result Guaranteed safety with arbitrarily large probability while learning and control proceed simultaneously.
Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks. RMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning. However, since the first order gradients include noise caused by stochastic optimization, the approxima…
Study online linear regression with paid noise reduction.
problem Online linear regression with noisy features and the ability to pay for reduced noise.
method Analyzes regret against optimal predictor, uses matrix martingale concentration.
result Optimal regret rates for known and unknown noise covariance.
This work studies applications and generalizations of a simple estimation technique that provides exponential concentration under heavy-tailed distributions, assuming only bounded low-order moments. We show that the technique can be used for approximate minimization of smooth and strongly convex losses, and specificall…
Exact recovery method for community detection in Gaussian mixtures with dependent noise.
problem Community detection in Gaussian mixtures with dependent and heterogeneous noise.
method Maximum likelihood estimator (MLE) for constrained quadratic optimization problem, using Σ-whitened separation and local inequalities. result Sharp exact-recovery threshold and no-gap mechanism in the unknown-size setting.
We formulate and study a general family of (continuous-time) stochastic dynamics for accelerated first-order minimization of smooth convex functions. Building on an averaging formulation of accelerated mirror descent, we propose a stochastic variant in which the gradient is contaminated by noise, and study the resultin…
We study the robustness of classifiers to various kinds of random noise models. In particular, we consider noise drawn uniformly from the ℓ_p ball for p∈[1,∞] and Gaussian noise with an arbitrary covariance matrix. We characterize this robustness to random noise in terms of the distance to the decisio…
The paper proposes AIS for Bayesian inversion of multioutput signals with covariance estimation.
problem Performing uncertainty analysis of covariance matrices in Bayesian inversion problems for multioutput signals.
method Adaptive Importance Sampling (AIS) scheme, split variables, frequentist approach for noise covariance, prior density over covariance matrix.
result Estimation of model parameters and covariance matrix of noise.
Paper develops methods for PCA inference with missing data and heteroskedastic noise.
problem Constructing confidence regions for PCA in high dimensions with missing data and heteroskedastic noise.
method Proposes HeteroPCA and develops non-asymptotic distributional guarantees for valid inference.
result Valid inference on principal subspace and spiked covariance matrix with missing data.
In high dimension, it is customary to consider Lasso-type estimators to enforce sparsity. For standard Lasso theory to hold, the regularization parameter should be proportional to the noise level, yet the latter is generally unknown in practice. A possible remedy is to consider estimators, such as the Concomitant/Scale…
A general framework for principal component analysis (PCA) in the presence of heteroskedastic noise is introduced. We propose an algorithm called HeteroPCA, which involves iteratively imputing the diagonal entries of the sample covariance matrix to remove estimation bias due to heteroskedasticity. This procedure is com…