Paper proposes a self-supervised method to denoise autoregressive signals with heavy-tailed noise.
problem Denoising autoregressive signals corrupted by heavy-tailed noise.
method Self-supervised learning approach without requiring full noise distribution knowledge.
result Strong denoising performance compared to baseline methods, especially for impulsive noise.
Study shows Stochastic Mirror Descent optimizes convex problems with infinite noise variance.
problem Optimizing convex problems with infinite noise variance.
method Stochastic Mirror Descent algorithm with uniformly convex mirror maps.
result Demonstrates convergence rate quantified in terms of iterations, dimensionality, and geometric parameters.
New convergence rates for SGD under heavy-tailed noise with infinite variance.
problem Convergence analysis of SGD under heavy-tailed noise with infinite variance.
method Identifying a condition on the Hessian and providing a convergence rate for the distance to the global optimum.
result SGD can converge to the global optimum under heavy-tailed noise with infinite variance.
This paper analyzes M-estimators under infinite-variance noise in high dimensions.
problem High-dimensional M-estimation with infinite-variance noise.
method Study of the Fenchel conjugate domain and its impact on risk.
result Exact risk of M-estimators under infinite-variance noise is derived.
This paper analyzes bias-variance trade-off for clipped SFOMs, improving complexity guarantees for heavy-tailed noise.
problem Improving complexity guarantees for stochastic optimization methods with heavy-tailed noise.
method Novel analysis of bias-variance trade-off in gradient clipping for clipped SFOMs.
result Improved complexity guarantees for clipped SFOMs across various tail indices, including infinite mean noise.
The Kalman filter is extensively used for state estimation for linear systems under Gaussian noise. When non-Gaussian Lévy noise is present, the conventional Kalman filter may fail to be effective due to the fact that the non-Gaussian Lévy noise may have infinite variance. A modified Kalman filter for linear systems wi…
Bayesian deep neural networks converge to processes with α-stable marginals under infinite variance weights.
problem Representation learning in deep kernel processes is hindered by deterministic covariance kernels.
method Showed convergence to α-stable processes with conditionally Gaussian representations in infinite-width networks.
result Conditional random covariance kernels can be recursively linked, even if the process is α-stable.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
problem Understanding why large batch sizes work in DP-SGD.
method Decomposed total gradient variance into subsampling and noise-induced variances, proving batch size independence in the limit.
result Large batch sizes reduce effective total gradient variance, improving privacy in DP-SGD.
Volatility estimation based on high-frequency data is key to accurately measure and control the risk of financial assets. A Lévy process with infinite jump activity and microstructure noise is considered one of the simplest, yet accurate enough, models for financial data at high-frequency. Utilizing this model, we prop…
We consider the prediction of weak effects in a multiple-output regression setup, when covariates are expected to explain a small amount, less than ≈ 1 \approx 1% ≈ 1 , of the variance of the target variables. To facilitate the prediction of the weak effects, we constrain our model structure by introducing a novel Bayesian ap…
Study minimax regret in bilateral trade with heavy-tailed valuations.
problem Minimizing regret in bilateral trade with infinite variance valuations.
method Extended self-bounding property, truncated-mean estimation, epoch-based algorithm.
result Achieves regret bound of O ( T 1 − 2 β ( p − 1 ) / ( β p + d ( p − 1 ) ) ) O(T^{1-2β(p-1)/(βp + d(p-1))}) O ( T 1 − 2 β ( p − 1 ) / ( β p + d ( p − 1 )) ) under specific conditions. Study on fluctuations in neural network kernels and predictions, focusing on finite width effects.
problem Characterizing fluctuations in finite width neural networks.
method Dynamical mean field theory analysis of wide but finite feature learning neural networks.
result Fluctuations in kernels and predictions are dynamically coupled, leading to reduced variance in feature learning regimes.
Improves diffusion models by controlling total variance and signal-to-noise-ratio.
problem Long sampling time in diffusion models.
method Total-Variance/Signal-to-Noise-Ratio (TV/SNR) disentangled framework.
result Improves generation performance by controlling TV and SNR independently.
Variance reduction has been commonly used in stochastic optimization. It relies crucially on the assumption that the data set is finite. However, when the data are imputed with random noise as in data augmentation, the perturbed data set be- comes essentially infinite. Recently, the stochastic MISO (S-MISO) algorithm i…
The paper extends confidence sequences for infinite variance data.
problem Addressing confidence sequences for distributions with infinite variance.
method Establishing lower bounds and deriving tight confidence sequences for relaxed bounded p t h p^{th} p t h -moment distributions. result Derived confidence sequences are tighter than those using Dubins-Savage inequality.
The paper studies stochastic gradient descent with infinite variance gradients.
problem Theoretical properties of SGD with infinite variance gradients.
method Establish asymptotic behavior of SGD with infinite variance gradients.
result Asymptotic distribution of SGD is characterized as a stationary distribution of an Ornstein-Uhlenbeck process driven by a stable Lévy process.
Improved GP bandit algorithms for noiseless, varying noise, and RKHS norms.
problem Minimizing regret in Gaussian process bandits with unknown reward functions.
method New upper bound on maximum posterior variance, refined MVR and PE algorithms.
result Optimal regret bounds for noiseless, varying noise, and RKHS norms.
Increasing variance of losses improves learning with noisy labels.
problem Learning with noisy labels and the need to penalize variance of losses.
method Designing regularizers based on the label noise transition matrix to increase variance of losses.
result Increasing variance of losses significantly improves generalization ability.
Study on sample complexity of policy gradient for stabilizing linear systems under multiplicative noise.
problem Learning optimal feedback gain for stabilizing linear systems with multiplicative noise.
method Analyzes the sample complexity of policy gradient methods, addressing the cusp obstruction and using symmetry to control divergent parts of the gradient.
result Proves that projected mini-batch policy gradient attains total sample complexity of O(1/η) when noise density is known and O(η^(-(2s+1)/(2s))) when estimated, for C^s noise densities with s ≥ 2.
This paper presents a novel approach for approximate integration over the uncertainty of noise and signal variances in Gaussian process (GP) regression. Our efficient and straightforward approach can also be applied to integration over input dependent noise variance (heteroscedasticity) and input dependent signal varia…
Study variance-optimal hedging of forward curve derivatives under stochastic volatility.
problem Variance-optimal hedging of forward curve derivatives with stochastic volatility.
method Assumes HJM-Musiela dynamics modulated by stochastic covariance, uses Galtchouk-Kunita-Watanabe projection.
result Density of finite-maturity strategies, convergence of finite-rank projections, decomposition of hedging error.
We study the variance of the REINFORCE policy gradient estimator in environments with continuous state and action spaces, linear dynamics, quadratic cost, and Gaussian noise. These simple environments allow us to derive bounds on the estimator variance in terms of the environment and noise parameters. We compare the pr…
Regularization helps resolve ambiguity in mean-variance models, improving predictive uncertainty quantification.
problem Signal-to-noise ambiguity in overparameterized mean-variance models.
method Statistical field theory framework to explain phase transition.
result Regularization reduces variability and improves predictive uncertainty quantification.
New method reduces inference variance for faster optimization.
problem High variance in black-box variational inference.
method Joint control variate addressing both data subsampling and Monte Carlo noise.
result Significantly reduced gradient variance, leading to faster optimization.
Simultaneous orthogonal matching pursuit (SOMP) and block OMP (BOMP) are two widely used techniques for sparse support recovery in multiple measurement vector (MMV) and block sparse (BS) models respectively. For optimal performance, both SOMP and BOMP require \textit{a priori} knowledge of signal sparsity or noise vari…
Paper proposes robust estimators for heavy-tailed data with infinite variance.
problem Developing robust estimators for heavy-tailed data with infinite variance.
method Proposes two robust estimators: ridge log-truncated M-estimator and elastic net log-truncated M-estimator.
result Demonstrates robustness of log-truncated estimations over standard estimations through simulations and real data analysis.
Simplified analysis of diffusion models using discrete random variables.
problem Theoretical analysis of diffusion models is complex and requires rigorous proofs.
method Simplified framework for analyzing Euler--Maruyama discretization of VP-SDEs using Grönwall's inequality.
result Standard Gaussian noise can be replaced by discrete random variables without sacrificing convergence guarantee.
Noise injection before gradient steps helps in regularization for neural networks.
problem Improving generalization in overparametrized neural networks.
method Injecting small noise perturbations before computing gradient steps, especially in layer-wise fashion.
result Small noise perturbations can explicitly regularize neural networks without variance explosion.
Proposes ENVAR for causal discovery in structural VAR models with equal noise variance.
problem Challenges in causal discovery from multivariate time series with contemporaneous effects.
method Introduces observational equivalence and the observational alignment discrepancy for structural VAR models with equal noise variance.
result Shows that multiple structural VAR parameterizations can induce the same stationary observed process law.
Method estimates noise variance in Gaussian process regression.
problem Estimating noise variance in Gaussian process regression models.
method Reduces hyperparameter space, uses marginal likelihood function, derives bounds and asymptotes.
result Computational advantages and robustness compared to traditional methods.
Derives scaling limits and fluctuations for SGD in high dimensions.
problem Understanding SGD behavior in high-dimensional settings with varying noise levels.
method Interacting particle system approach, treating SGD iterates as such, with covariance structure considered.
result Precise three-step phase transition observed in SGD behavior: ballistic, diffusive, then random.
We analyze the Levy processes produced by means of two interconnected classes of non stable, infinitely divisible distribution: the Variance Gamma and the Student laws. While the Variance Gamma family is closed under convolution, the Student one is not: this makes its time evolution more complicated. We prove that -- a…
Paper develops methods for statistical inference in SGD with infinite variance.
problem Challenges in statistical inference for SGD with infinite variance.
method Model-agnostic methodology based on weak convergence and subsampling calibration.
result Asymptotically valid confidence regions for SGD in both finite and infinite variance regimes.
Privacy-preserving SGD with heavy-tailed noise achieves differential privacy guarantees.
problem Privacy preservation in noisy SGD with heavy-tailed noise.
method Differential privacy guarantees for SGD with heavy-tailed noise.
result SGD with heavy-tailed perturbations achieves ( 0 , O ( 1 / n ) ) (0, O(1/n)) ( 0 , O ( 1/ n )) -DP. New algorithm reduces regret for linear bandits with unknown noise variance.
problem Finding optimal actions in linear bandits with varying noise variance.
method Adaptive algorithm with Freedman-type concentration inequality and multi-layer structure.
result Achieves i l d e O ( d ∑ k = 1 K σ k 2 + d ) ilde{O}(d \sqrt{\sum_{k = 1}^K σ_k^2} + d) i l d e O ( d ∑ k = 1 K σ k 2 + d ) regret for linear bandits. New method reduces variance in stochastic optimization with high confidence.
problem Achieving high-probability guarantees in stochastic optimization with weaker noise assumptions.
method Stochastic proximal point method combining proximal subproblem solver and probability booster.
result Demonstrates convergence with low sample complexity under bounded variance assumptions.
CARV reduces compute cost for downstream pipelines using diffusion models.
problem High variance in Monte Carlo estimators from diffusion models limits compute efficiency.
method CARV uses hierarchical MC estimation with amortized upstream computation and stratified-inverse-CDF.
result CARV delivers 2-3x effective compute multipliers without changing the objective.
Proposes incorporating noise sources in machine learning evaluation for more reliable conclusions.
problem Inadequate handling of nondeterminism in machine learning research leads to unreliable results.
method Uses linear mixed effects models (LMEMs) and generalized likelihood ratio tests (GLRT) to analyze performance evaluation scores and assess performance differences.
result Demonstrates how to incorporate various sources of noise and data properties into statistical significance testing and reliability analysis.
SGD transitions between maxima and minima with varying time scales.
problem Understanding SGD's behavior near critical points in noisy landscapes.
method Analyzing SGD convergence and escape dynamics in 1D landscapes with infinite- and finite-variance noise.
result SGD reliably moves to the basin's minimum unless close to a local maximum, where it can linger.
New algorithm resists corruption in linear contextual bandits.
problem Adversarial corruption in linear contextual bandits.
method Variance-aware algorithm with multi-level partition and adaptive confidence sets.
result Regret bound of i l d e O ( C 2 d ∑ t = 1 T σ t 2 + C 2 R d T ) ilde{O}(C^2d\sqrt{\sum_{t = 1}^T σ_t^2} + C^2R\sqrt{dT}) i l d e O ( C 2 d ∑ t = 1 T σ t 2 + C 2 R d T ) . PS-IG improves feature attribution by reducing noise and variance.
problem Improving feature attribution in machine learning models.
method Path-sampled integrated gradients (PS-IG) computes expected value over sampled baselines.
result PS-IG reduces attribution variance by a factor of 1/3 under uniform sampling.
Measures three types of noise in LLM evaluations.
problem Separating signal from noise in LLM experiments.
method Defined and measured three types of noise: prediction, data, and total noise. Proposed the all-pairs paired method for statistical power.
result Total noise level is characteristic and predictable across all model pairs.
PEGR improves deep learning models' robustness against noisy data.
problem Learning signals from noisy data in deep learning models.
method Per-example gradient regularization (PEGR) to suppress noise.
result PEGR enhances test error and robustness against noise perturbations.
This paper studies distributed estimation and support recovery for high-dimensional linear regression model with heavy-tailed noise. To deal with heavy-tailed noise whose variance can be infinite, we adopt the quantile regression loss function instead of the commonly used squared loss. However, the non-smooth quantile …
ALPCAH improves PCA for noisy data by estimating sample-wise noise variances.
problem Noisy data with varying noise levels in different samples.
method Sample-wise heteroscedastic PCA with tail singular value regularization.
result Improves subspace basis estimation for low-rank data.
Paper improves sparse linear bandits by accounting for noise variance.
problem Sparse linear bandits with unknown noise variance.
method Develops a general framework to convert variance-aware algorithms to sparse linear bandits.
result Achieves $\widetilde{\mathcal O}\left(\sqrt{d\sum_{t=1}^T σ_t^2} + 1
ight)$ regret, interpolating between worst-case and benign settings.
Introduces TPV to analyze model robustness without labels.
problem Analyzing post-training robustness of machine learning models.
method Parameter perturbations and test prediction variance (TPV) as a unifying framework.
result TPV connects various perturbations under a single lens, providing insights into model stability.
Paper improves confidence intervals and variance estimation for deep learning models.
problem Improving confidence intervals and variance estimation in deep learning models.
method Residual-based framework for conditional variance estimation; robust bootstrap procedure for confidence intervals.
result First non-asymptotic bounds for variance estimation using ReLU networks.