This paper analyzes bias-variance trade-off for clipped SFOMs, improving complexity guarantees for heavy-tailed noise.
problem Improving complexity guarantees for stochastic optimization methods with heavy-tailed noise.
method Novel analysis of bias-variance trade-off in gradient clipping for clipped SFOMs.
result Improved complexity guarantees for clipped SFOMs across various tail indices, including infinite mean noise.
Improves diffusion models by controlling total variance and signal-to-noise-ratio.
problem Long sampling time in diffusion models.
method Total-Variance/Signal-to-Noise-Ratio (TV/SNR) disentangled framework.
result Improves generation performance by controlling TV and SNR independently.
Paper proposes a self-supervised method to denoise autoregressive signals with heavy-tailed noise.
problem Denoising autoregressive signals corrupted by heavy-tailed noise.
method Self-supervised learning approach without requiring full noise distribution knowledge.
result Strong denoising performance compared to baseline methods, especially for impulsive noise.
Improved GP bandit algorithms for noiseless, varying noise, and RKHS norms.
problem Minimizing regret in Gaussian process bandits with unknown reward functions.
method New upper bound on maximum posterior variance, refined MVR and PE algorithms.
result Optimal regret bounds for noiseless, varying noise, and RKHS norms.
Increasing variance of losses improves learning with noisy labels.
problem Learning with noisy labels and the need to penalize variance of losses.
method Designing regularizers based on the label noise transition matrix to increase variance of losses.
result Increasing variance of losses significantly improves generalization ability.
This paper presents a novel approach for approximate integration over the uncertainty of noise and signal variances in Gaussian process (GP) regression. Our efficient and straightforward approach can also be applied to integration over input dependent noise variance (heteroscedasticity) and input dependent signal varia…
We study the variance of the REINFORCE policy gradient estimator in environments with continuous state and action spaces, linear dynamics, quadratic cost, and Gaussian noise. These simple environments allow us to derive bounds on the estimator variance in terms of the environment and noise parameters. We compare the pr…
Regularization helps resolve ambiguity in mean-variance models, improving predictive uncertainty quantification.
problem Signal-to-noise ambiguity in overparameterized mean-variance models.
method Statistical field theory framework to explain phase transition.
result Regularization reduces variability and improves predictive uncertainty quantification.
New method reduces inference variance for faster optimization.
problem High variance in black-box variational inference.
method Joint control variate addressing both data subsampling and Monte Carlo noise.
result Significantly reduced gradient variance, leading to faster optimization.
Simultaneous orthogonal matching pursuit (SOMP) and block OMP (BOMP) are two widely used techniques for sparse support recovery in multiple measurement vector (MMV) and block sparse (BS) models respectively. For optimal performance, both SOMP and BOMP require \textit{a priori} knowledge of signal sparsity or noise vari…
Simplified analysis of diffusion models using discrete random variables.
problem Theoretical analysis of diffusion models is complex and requires rigorous proofs.
method Simplified framework for analyzing Euler--Maruyama discretization of VP-SDEs using Grönwall's inequality.
result Standard Gaussian noise can be replaced by discrete random variables without sacrificing convergence guarantee.
Noise injection before gradient steps helps in regularization for neural networks.
problem Improving generalization in overparametrized neural networks.
method Injecting small noise perturbations before computing gradient steps, especially in layer-wise fashion.
result Small noise perturbations can explicitly regularize neural networks without variance explosion.
Proposes ENVAR for causal discovery in structural VAR models with equal noise variance.
problem Challenges in causal discovery from multivariate time series with contemporaneous effects.
method Introduces observational equivalence and the observational alignment discrepancy for structural VAR models with equal noise variance.
result Shows that multiple structural VAR parameterizations can induce the same stationary observed process law.
Method estimates noise variance in Gaussian process regression.
problem Estimating noise variance in Gaussian process regression models.
method Reduces hyperparameter space, uses marginal likelihood function, derives bounds and asymptotes.
result Computational advantages and robustness compared to traditional methods.
New algorithm reduces regret for linear bandits with unknown noise variance.
problem Finding optimal actions in linear bandits with varying noise variance.
method Adaptive algorithm with Freedman-type concentration inequality and multi-layer structure.
result Achieves i l d e O ( d ∑ k = 1 K σ k 2 + d ) ilde{O}(d \sqrt{\sum_{k = 1}^K σ_k^2} + d) i l d e O ( d ∑ k = 1 K σ k 2 + d ) regret for linear bandits. New method reduces variance in stochastic optimization with high confidence.
problem Achieving high-probability guarantees in stochastic optimization with weaker noise assumptions.
method Stochastic proximal point method combining proximal subproblem solver and probability booster.
result Demonstrates convergence with low sample complexity under bounded variance assumptions.
New convergence rates for SGD under heavy-tailed noise with infinite variance.
problem Convergence analysis of SGD under heavy-tailed noise with infinite variance.
method Identifying a condition on the Hessian and providing a convergence rate for the distance to the global optimum.
result SGD can converge to the global optimum under heavy-tailed noise with infinite variance.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
problem Understanding why large batch sizes work in DP-SGD.
method Decomposed total gradient variance into subsampling and noise-induced variances, proving batch size independence in the limit.
result Large batch sizes reduce effective total gradient variance, improving privacy in DP-SGD.
CARV reduces compute cost for downstream pipelines using diffusion models.
problem High variance in Monte Carlo estimators from diffusion models limits compute efficiency.
method CARV uses hierarchical MC estimation with amortized upstream computation and stratified-inverse-CDF.
result CARV delivers 2-3x effective compute multipliers without changing the objective.
Proposes incorporating noise sources in machine learning evaluation for more reliable conclusions.
problem Inadequate handling of nondeterminism in machine learning research leads to unreliable results.
method Uses linear mixed effects models (LMEMs) and generalized likelihood ratio tests (GLRT) to analyze performance evaluation scores and assess performance differences.
result Demonstrates how to incorporate various sources of noise and data properties into statistical significance testing and reliability analysis.
This paper analyzes M-estimators under infinite-variance noise in high dimensions.
problem High-dimensional M-estimation with infinite-variance noise.
method Study of the Fenchel conjugate domain and its impact on risk.
result Exact risk of M-estimators under infinite-variance noise is derived.
Study shows Stochastic Mirror Descent optimizes convex problems with infinite noise variance.
problem Optimizing convex problems with infinite noise variance.
method Stochastic Mirror Descent algorithm with uniformly convex mirror maps.
result Demonstrates convergence rate quantified in terms of iterations, dimensionality, and geometric parameters.
PS-IG improves feature attribution by reducing noise and variance.
problem Improving feature attribution in machine learning models.
method Path-sampled integrated gradients (PS-IG) computes expected value over sampled baselines.
result PS-IG reduces attribution variance by a factor of 1/3 under uniform sampling.
Measures three types of noise in LLM evaluations.
problem Separating signal from noise in LLM experiments.
method Defined and measured three types of noise: prediction, data, and total noise. Proposed the all-pairs paired method for statistical power.
result Total noise level is characteristic and predictable across all model pairs.
PEGR improves deep learning models' robustness against noisy data.
problem Learning signals from noisy data in deep learning models.
method Per-example gradient regularization (PEGR) to suppress noise.
result PEGR enhances test error and robustness against noise perturbations.
ALPCAH improves PCA for noisy data by estimating sample-wise noise variances.
problem Noisy data with varying noise levels in different samples.
method Sample-wise heteroscedastic PCA with tail singular value regularization.
result Improves subspace basis estimation for low-rank data.
Paper improves sparse linear bandits by accounting for noise variance.
problem Sparse linear bandits with unknown noise variance.
method Develops a general framework to convert variance-aware algorithms to sparse linear bandits.
result Achieves $\widetilde{\mathcal O}\left(\sqrt{d\sum_{t=1}^T σ_t^2} + 1
ight)$ regret, interpolating between worst-case and benign settings.
Introduces TPV to analyze model robustness without labels.
problem Analyzing post-training robustness of machine learning models.
method Parameter perturbations and test prediction variance (TPV) as a unifying framework.
result TPV connects various perturbations under a single lens, providing insights into model stability.
Paper improves confidence intervals and variance estimation for deep learning models.
problem Improving confidence intervals and variance estimation in deep learning models.
method Residual-based framework for conditional variance estimation; robust bootstrap procedure for confidence intervals.
result First non-asymptotic bounds for variance estimation using ReLU networks.
A trade-off exists between reconstruction quality and the prior regularisation in the Evidence Lower Bound (ELBO) loss that Variational Autoencoder (VAE) models use for learning. There are few satisfactory approaches to deal with a balance between the prior and reconstruction objective, with most methods dealing with t…
The paper analyzes continuous optimization for DAG structure learning and its limitations.
problem The performance of continuous structure learning approaches is not consistent after data standardization.
method Analysis of continuous optimization for DAG structure learning, focusing on equal and non-equal noise variances.
result Continuous structure learning approaches may not perform well after data standardization, especially with non-equal noise variances.
In this work, we study robust deep learning against abnormal training data from the perspective of example weighting built in empirical loss functions, i.e., gradient magnitude with respect to logits, an angle that is not thoroughly studied so far. Consequently, we have two key findings: (1) Mean Absolute Error (MAE) D…
Recovering the support of sparse vectors in underdetermined linear regression models, \textit{aka}, compressive sensing is important in many signal processing applications. High SNR consistency (HSC), i.e., the ability of a support recovery technique to correctly identify the support with increasing signal to noise rat…
Improves deep learning performance on noisy datasets using inverse-variance weighting.
problem Heteroscedastic regression with varying noise levels.
method Batch Inverse-Variance (BIV) loss function for neural networks.
result Significantly improves network performance on noisy datasets compared to other methods.
New findings show optimal noise in contrastive learning is not the same as data distribution.
problem The optimal noise distribution in contrastive learning is not the same as the data distribution.
method Empirical and theoretical analysis of contrastive learning methods.
result Deviation from the assumption of equal noise and data distribution leads to better statistical estimators.
Alternative to likelihood-based LSNM model selection, residual independence testing is more robust to noise misspecification.
problem Cause-effect inference in location-scale noise models with misspecified noise distributions.
method Residual independence testing as an alternative to likelihood-based model selection.
result Residual independence testing is more robust to noise misspecification.
Recent reinforcement learning algorithms, though achieving impressive results in various fields, suffer from brittle training effects such as regression in results and high sensitivity to initialization and parameters. We claim that some of the brittleness stems from variance differences, i.e. when different environmen…
Reconstruction-based learning produces uninformative features for perception tasks.
problem Misalignment between reconstruction-based learning and perception tasks.
method Investigated the impact of input space reconstruction on feature learning for perception tasks.
result Reconstruction-based learning allocates model capacity to a subspace with uninformative features for perception tasks.
Improved analysis for fair federated learning reduces dependence on noise floor.
problem Asymptotic stationarity in group fair federated learning with reduced noise floor dependence.
method DS FedProxGrad framework with inexact local proximal solutions and fairness regularization.
result Algorithm converges asymptotically to stationarity without dependence on a noise floor.
ALPCAH improves PCA for noisy data samples.
problem Heteroscedastic data with varying noise levels.
method Subspace learning method estimating sample-wise noise variances.
result Improves subspace basis for low-rank data.
We introduce wavelet-based methodology for estimation of realized variance allowing its measurement in the time-frequency domain. Using smooth wavelets and Maximum Overlap Discrete Wavelet Transform, we allow for the decomposition of the realized variance into several investment horizons and jumps. Basing our estimator…
The Kalman filter is extensively used for state estimation for linear systems under Gaussian noise. When non-Gaussian Lévy noise is present, the conventional Kalman filter may fail to be effective due to the fact that the non-Gaussian Lévy noise may have infinite variance. A modified Kalman filter for linear systems wi…
We propose a Bayesian expectation-maximization (EM) algorithm for reconstructing Markov-tree sparse signals via belief propagation. The measurements follow an underdetermined linear model where the regression-coefficient vector is the sum of an unknown approximately sparse signal and a zero-mean white Gaussian noise wi…
VGE provides a practical approach to uncertainty estimation in ensemble models.
problem Uncertainty estimation in ensemble models using additive decomposition breaks down.
method Variance-Gated Ensembles (VGE) introduces a differentiable framework with a signal-to-noise gate.
result VGE provides a Variance-Gated Margin Uncertainty (VGMU) score and Variance-Gated Normalization (VGN) layer.
New sampling scheme improves privacy in DP-SGD without sacrificing utility.
problem Suboptimal privacy amplification due to participation variance in Poisson subsampling.
method Balanced Iteration Subsampling (BIS) with structured randomness.
result BIS achieves stronger privacy amplification than Poisson subsampling and is optimal at both extremes of noise spectrum.
Optimizes noisy IS with better proposal densities.
problem Improving IS estimators with noisy data.
method Derives optimal proposal densities considering noise variance.
result Optimal proposals enhance IS estimators by focusing on noisy regions.
Neural- σ 2 σ^2 σ 2 -LinearUCB improves regret in neural contextual bandits.
problem Balancing exploration and exploitation in neural contextual bandits.
method Proposes a variance-aware neural UCB algorithm using neural representations and an upper bound of reward noise variance.
result Oracle and practical versions of Neural- σ 2 σ^2 σ 2 -LinearUCB achieve better regret guarantees and performance. BNEM improves Boltzmann sampler efficiency.
problem Generating IID samples from Boltzmann distributions efficiently.
method Bootstrapped Noised Energy Matching (NEM) combined with diffusion-based learning and bootstrapping.
result BNEM achieves state-of-the-art performance with improved robustness.