The paper extends confidence sequences for infinite variance data.
problem Addressing confidence sequences for distributions with infinite variance.
method Establishing lower bounds and deriving tight confidence sequences for relaxed bounded pth-moment distributions. result Derived confidence sequences are tighter than those using Dubins-Savage inequality.
The paper studies stochastic gradient descent with infinite variance gradients.
problem Theoretical properties of SGD with infinite variance gradients.
method Establish asymptotic behavior of SGD with infinite variance gradients.
result Asymptotic distribution of SGD is characterized as a stationary distribution of an Ornstein-Uhlenbeck process driven by a stable Lévy process.
Paper proposes robust estimators for heavy-tailed data with infinite variance.
problem Developing robust estimators for heavy-tailed data with infinite variance.
method Proposes two robust estimators: ridge log-truncated M-estimator and elastic net log-truncated M-estimator.
result Demonstrates robustness of log-truncated estimations over standard estimations through simulations and real data analysis.
Paper develops methods for statistical inference in SGD with infinite variance.
problem Challenges in statistical inference for SGD with infinite variance.
method Model-agnostic methodology based on weak convergence and subsampling calibration.
result Asymptotically valid confidence regions for SGD in both finite and infinite variance regimes.
Study shows Stochastic Mirror Descent optimizes convex problems with infinite noise variance.
problem Optimizing convex problems with infinite noise variance.
method Stochastic Mirror Descent algorithm with uniformly convex mirror maps.
result Demonstrates convergence rate quantified in terms of iterations, dimensionality, and geometric parameters.
New convergence rates for SGD under heavy-tailed noise with infinite variance.
problem Convergence analysis of SGD under heavy-tailed noise with infinite variance.
method Identifying a condition on the Hessian and providing a convergence rate for the distance to the global optimum.
result SGD can converge to the global optimum under heavy-tailed noise with infinite variance.
Stochastic neural networks with infinite width become deterministic, reducing training variance.
problem Understanding how stochasticity in neural networks affects learning and regularization.
method Theoretical analysis of stochastic neural networks with infinite width.
result As the width of an optimized stochastic neural network increases, its predictive variance on the training set decreases to zero.
Paper proposes a self-supervised method to denoise autoregressive signals with heavy-tailed noise.
problem Denoising autoregressive signals corrupted by heavy-tailed noise.
method Self-supervised learning approach without requiring full noise distribution knowledge.
result Strong denoising performance compared to baseline methods, especially for impulsive noise.
The study reveals a transition in neural network performance from infinite-width to variance-limited behavior as dataset size increases.
problem Understanding the transition from infinite-width to variance-limited behavior in neural networks.
method Empirical study of the transition from infinite-width to variance-limited behavior as a function of sample size and network width.
result The critical sample size \( P^* \) is approximately \( \sqrt{N} \) for polynomial regression with ReLU networks.
Bayesian deep neural networks converge to processes with α-stable marginals under infinite variance weights.
problem Representation learning in deep kernel processes is hindered by deterministic covariance kernels.
method Showed convergence to α-stable processes with conditionally Gaussian representations in infinite-width networks.
result Conditional random covariance kernels can be recursively linked, even if the process is α-stable.
New unbiased variance estimator for random forests using Hoeffding decomposition.
problem Uncertainty quantification in random forests with large kernel sizes and small sample sizes.
method Proposes a new Hoeffding decomposition view for variance estimation, establishing unbiased estimators and ratio consistency.
result Establishes the ratio consistency of the proposed variance estimator, justifying confidence interval coverage rates.
New algorithm reduces regret in infinite MDPs with optimal variance-dependent bounds.
problem Infinite horizon MDPs lack optimal algorithms with low regret.
method Developed a UCB-style algorithm for average-reward and γ-regret.
result Achieved optimal variance-dependent regret bounds for both objectives.
Neural SDEs reduce variance in stochastic simulations.
problem Efficiency of Monte Carlo simulations in finance.
method Use neural SDEs with control variates parameterized by neural networks.
result Prove optimality conditions for variance reduction in SDEs with infinite activity.
This paper analyzes M-estimators under infinite-variance noise in high dimensions.
problem High-dimensional M-estimation with infinite-variance noise.
method Study of the Fenchel conjugate domain and its impact on risk.
result Exact risk of M-estimators under infinite-variance noise is derived.
Study minimax regret in bilateral trade with heavy-tailed valuations.
problem Minimizing regret in bilateral trade with infinite variance valuations.
method Extended self-bounding property, truncated-mean estimation, epoch-based algorithm.
result Achieves regret bound of O(T1−2β(p−1)/(βp+d(p−1))) under specific conditions. In this note we provide detailed derivations of two versions of small-variance asymptotics for hierarchical Dirichlet process (HDP) mixture models and the HDP hidden Markov model (HDP-HMM, a.k.a. the infinite HMM). We include derivations for the probabilities of certain CRP and CRF partitions, which are of more general…
Infinite horizon off-policy policy evaluation is a highly challenging task due to the excessively large variance of typical importance sampling (IS) estimators. Recently, Liu et al. (2018a) proposed an approach that significantly reduces the variance of infinite-horizon off-policy evaluation by estimating the stationar…
A new method detects anomalies in multivariate streams without unit dependence.
problem Detect anomalies in multivariate streams without unit dependence.
method Proposes SigMahaKNN combining variance norm and path signature.
result SigMahaKNN detects anomalies better than existing methods.
We study the problems related to the estimation of the Gini index in presence of a fat-tailed data generating process, i.e. one in the stable distribution class with finite mean but infinite variance (i.e. with tail index α∈(1,2)). We show that, in such a case, the Gini coefficient cannot be reliably estimated usin…
The posterior variance of Gaussian processes is a valuable measure of the learning error which is exploited in various applications such as safe reinforcement learning and control design. However, suitable analysis of the posterior variance which captures its behavior for finite and infinite number of training data is …
Bayesian ReLU nets fix asymptotic overconfidence with infinite features.
problem Bayesian ReLU nets can be asymptotically overconfident far from training data.
method Extend finite ReLU BNNs with infinite ReLU features via a Gaussian process.
result The resulting model is asymptotically maximally uncertain far from the data.
We consider the off-policy estimation problem of estimating the expected reward of a target policy using samples collected by a different behavior policy. Importance sampling (IS) has been a key technique to derive (nearly) unbiased estimators, but is known to suffer from an excessively high variance in long-horizon pr…
Study on fluctuations in neural network kernels and predictions, focusing on finite width effects.
problem Characterizing fluctuations in finite width neural networks.
method Dynamical mean field theory analysis of wide but finite feature learning neural networks.
result Fluctuations in kernels and predictions are dynamically coupled, leading to reduced variance in feature learning regimes.
Before training a neural net, a classic rule of thumb is to randomly initialize the weights so the variance of activations is preserved across layers. This is traditionally interpreted using the total variance due to randomness in both weights \emph{and} samples. Alternatively, one can interpret the rule of thumb as pr…
Paper shows MoM is optimal under adversarial contamination for certain distributions.
problem Optimality of MoM under adversarial contamination.
method Upper and lower bounds for MoM's error under adversarial contamination.
result MoM is (minimax) optimal for distributions with finite variance and infinite variance with finite absolute moments.
Stochastic optimization algorithms with variance reduction have proven successful for minimizing large finite sums of functions. Unfortunately, these techniques are unable to deal with stochastic perturbations of input data, induced for example by data augmentation. In such cases, the objective is no longer a finite su…
Investment and consumption strategy for risk-averse agents with Epstein-Zin utility.
problem Optimal investment and consumption strategy for Epstein-Zin utility.
method Detailed introduction to Epstein-Zin utility, existence and uniqueness proof, verification argument.
result Existence and uniqueness of optimal solution for Epstein-Zin utility under certain parameter restrictions.
This paper removes the finite variance assumption for deep convolutional neural networks.
problem Removing the finite variance assumption for deep convolutional neural networks.
method Assuming iid parameters distributed according to a stable distribution, the paper shows that the infinite-channel limit of a deep feed-forward convolutional neural network is a multivariate stable stochastic process.
result The infinite-channel limit of a deep feed-forward convolutional neural network, under suitable scaling, is a multivariate stable stochastic process.
Optimizes learning Hilbert-Schmidt operators between Sobolev spaces.
problem Statistical limits of learning mappings between infinite-dimensional function spaces.
method Minimax optimal regularization and multilevel training.
result Multilevel kernel operator learning achieves optimal learning rate.
This paper analyzes bias-variance trade-off for clipped SFOMs, improving complexity guarantees for heavy-tailed noise.
problem Improving complexity guarantees for stochastic optimization methods with heavy-tailed noise.
method Novel analysis of bias-variance trade-off in gradient clipping for clipped SFOMs.
result Improved complexity guarantees for clipped SFOMs across various tail indices, including infinite mean noise.
The paper studies a rebalanced dataset for imbalanced classification using Centered Random Forests.
problem Imbalanced classification where one class is underrepresented.
method Theoretical analysis of Centered Random Forests (CRF) with rebalanced datasets and debiasing techniques.
result Theoretical Central Limit Theorem (CLT) for the infinite CRF and debiased estimator IS-ICRF.
New method reduces density estimation variance for multivariate data.
problem Efficient multivariate density estimation with reduced dimensionality.
method Variance-Reduced Sketching (VRS) framework for multivariate density estimation.
result VRS framework significantly improves density estimation over existing methods.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
problem Understanding why large batch sizes work in DP-SGD.
method Decomposed total gradient variance into subsampling and noise-induced variances, proving batch size independence in the limit.
result Large batch sizes reduce effective total gradient variance, improving privacy in DP-SGD.
Bayesian neural networks approximate Student-t processes in the infinite-width limit.
problem Modeling uncertainty in neural networks with greater flexibility.
method Extending asymptotic properties of Gaussian processes to Student-t processes in the infinite-width limit of BNNs.
result Posterior BNNs converge to Student-t processes in the infinite-width limit.
The paper extends logistic regression for unbounded majority classes and derives asymptotic properties.
problem Infinitely imbalanced logistic regression inference.
method Derive a second order expansion for slope parameter under unbounded majority class.
result The second order term converges to a normal distribution with a variance depending only on the minority class's mean.
Naive approaches to amortized inference in probabilistic programs with unbounded loops can produce estimators with infinite variance. This is particularly true of importance sampling inference in programs that explicitly include rejection sampling as part of the user-programmed generative procedure. In this paper we de…
The paper discusses the importance of infinite-mean models in finance and risk management.
problem Classic statistical models assume finite mean or variance, which is not suitable for heavy-tailed data.
method Discussion and recent results on infinite-mean models in economics and finance.
result Classic statistical results for finite-mean models often fail or flip for infinite-mean models.
Theory explains neural network scaling with dataset and model size.
problem Neural network scaling laws with dataset and model size.
method Identified variance-limited and resolution-limited scaling behaviors.
result Four scaling regimes explained: infinite data, infinite width, resolution-limited, and large width.
We consider a market impact game for n risk-averse agents that are competing in a market model with linear transient price impact and additional transaction costs. For both finite and infinite time horizons, the agents aim to minimize a mean-variance functional of their costs or to maximize the expected exponential u…
We estimate risk measures in Markov cost processes with lower and upper bounds.
problem Estimating risk measures in infinite-horizon discounted costs within Markov processes.
method Truncation scheme and lower/upper bounds for CVaR and variance estimation.
result Upper and lower bounds for CVaR and variance estimation match up to logarithmic factors.
We develop robust pricing and hedging of a weighted variance swap when market prices for a finite number of co--maturing put options are given. We assume the given prices do not admit arbitrage and deduce no-arbitrage bounds on the weighted variance swap along with super- and sub- replicating strategies which enforce t…
This work investigates training infinite mixtures with maximum likelihood for improved uncertainty quantification.
problem Improving uncertainty quantification in neural networks.
method Investigates training infinite mixtures with maximum likelihood instead of variational inference.
result The proposed method leads to stochastic networks with increased predictive variance, improved robustness, and higher entropy on out-of-distribution data.
A Monte Carlo method for pairs trading on mean-reverting spreads with Lévy processes.
problem Trading on mean-reverting spreads with flexible models.
method Monte Carlo simulation with variance gamma and alpha-gamma driving processes.
result Optimal trading strategies are affected by model parameters and correlation.
The paper prices energy spread options using a complex stochastic model.
problem Pricing energy spread options with specific stochastic dynamics.
method Uses an exponential Ornstein-Uhlenbeck process driven by variance gamma processes, applying the Esscher transform and FFT method.
result Derives an analytical formula for pricing forwards and spread options.
Variance reduction has been commonly used in stochastic optimization. It relies crucially on the assumption that the data set is finite. However, when the data are imputed with random noise as in data augmentation, the perturbed data set be- comes essentially infinite. Recently, the stochastic MISO (S-MISO) algorithm i…
We wish to compute the gradient of an expectation over a finite or countably infinite sample space having K≤∞ categories. When K is indeed infinite, or finite but very large, the relevant summation is intractable. Accordingly, various stochastic gradient estimators have been proposed. In this paper, we de…
New method for Bayesian neural networks with unbounded weights.
problem Posterior inference for Bayesian neural networks with unbounded weights.
method Conditionally Gaussian representation for efficient posterior inference.
result Interpretable and computationally efficient procedure for posterior inference.
We consider a stochastic volatility model with jumps where the underlying asset price is driven by the process sum of a 2-dimensional Brownian motion and a 2-dimensional compensated Poisson process. The market is incomplete, resulting in infinitely many equivalent martingale measures. We find the set equivalent marting…