Increasing variance of losses improves learning with noisy labels.
problem Learning with noisy labels and the need to penalize variance of losses.
method Designing regularizers based on the label noise transition matrix to increase variance of losses.
result Increasing variance of losses significantly improves generalization ability.
We analyze bias-variance of margin losses.
problem Understanding model overfitting/underfitting.
method Bias-variance decomposition for strictly convex margin losses.
result Expected risk decomposes into central model risk and data variation.
This paper calculates worst-case target semi-variances for uncertain losses.
problem Managing risk when loss distribution is uncertain and only partial information is known.
method Derives worst-case target semi-variances for symmetric or non-negative losses under uncertainty sets representing investor's undesirable scenarios.
result Closed-form expressions for worst-case target semi-variances are derived.
Introduces MWLD to measure loss inequality across groups.
problem Machine learning's focus on average loss can lead to large group loss discrepancies.
method Defines MWLD, relates it to fairness and robustness, and provides estimation methods.
result MWLD can be estimated efficiently under certain weighting functions and reduces loss variance without significant accuracy loss.
VarGrad reduces variance in ELBO gradient estimation for variational inference.
problem Improving the variance of gradient estimators in variational inference.
method VarGrad uses a new log-variance loss to estimate the ELBO gradient, achieving lower variance than the score function method.
result VarGrad offers a lower variance gradient estimator compared to other methods.
Unified theory explains diversity in ensemble learning.
problem Explaining diversity in ensemble learning across various scenarios.
method Developed a framework revealing diversity as a hidden dimension in bias-variance decomposition.
result Proved exact bias-variance-diversity decompositions for multiple losses in regression and classification.
Paper unifies bias and variance models for classification.
problem Different frameworks for bias and variance in classification.
method Unified Tumer & Ghosh and James approaches.
result Closed form relationships between 0/1 loss and squared error loss.
Improved CNN for HCCR with new loss function and ranking method.
problem Loss of inter-class information in traditional CNN models for HCCR.
method Combining cross entropy with a new similarity ranking function (Average variance similarity) as loss function.
result New loss function (SoftMax cross entropy with Average variance similarity) achieves highest accuracy in HCCR.
We analyze distributed algorithms for minimizing losses with large, disjoint data.
problem Distributed implementation of stochastic variance reduced methods for large, disjoint data.
method General framework for distributing stochastic variance reduced methods in a master/slave model.
result Linear convergence of distributed algorithms for minimizing strongly convex losses.
Study large deviations in life insurance portfolios without identical distributions.
problem Large deviations in life insurance portfolios with bounded losses and variances.
method Upper bound from standard large deviations, counterexample for full large deviation principle.
result Exponential bound for average loss exceeding a threshold.
The paper analyzes the bias-variance tradeoff for Bregman divergences.
problem Understanding the bias-variance tradeoff for Bregman divergences.
method Analyzes the bias-variance tradeoff through operations in dual space.
result Derives several results including a generalized law of total variance and ensembling operations.
Simplifies risk minimization combining mean and standard deviation.
problem Minimizing mean and standard deviation under heavy-tailed losses.
method Adapting robust mean estimation technique to include standard deviation.
result Simple approach performs as well or better than alternative risk criteria.
Integrates uncertainty of loss landscape into stochastic optimization.
problem Improving convergence and generalization in stochastic optimization.
method Incorporates variance of stochastic loss function into momentum updates.
result Improved convergence rates on MNIST and CIFAR-10 datasets.
Paper analyzes high-dimensional portfolio risks and finds empirical out-of-sample relative loss is more reliable.
problem Analyzing risks in high-dimensional portfolios using empirical variance.
method Derives asymptotic behavior of out-of-sample variance and relative loss in high-dimensional settings.
result Empirical out-of-sample relative loss is more reliable than variance in high-dimensional portfolios.
This paper analyzes M-estimators under infinite-variance noise in high dimensions.
problem High-dimensional M-estimation with infinite-variance noise.
method Study of the Fenchel conjugate domain and its impact on risk.
result Exact risk of M-estimators under infinite-variance noise is derived.
Vector embedding is a foundational building block of many deep learning models, especially in natural language processing. In this paper, we present a theoretical framework for understanding the effect of dimensionality on vector embeddings. We observe that the distributional hypothesis, a governing principle of statis…
In this paper, we study an insurer's reinsurance-investment problem under a mean-variance criterion. We show that excess-loss is the unique equilibrium reinsurance strategy under a spectrally negative Lévy insurance model when the reinsurance premium is computed according to the expected value premium principle. Furthe…
Improves deep learning performance on noisy datasets using inverse-variance weighting.
problem Heteroscedastic regression with varying noise levels.
method Batch Inverse-Variance (BIV) loss function for neural networks.
result Significantly improves network performance on noisy datasets compared to other methods.
This work addresses unstable MeanFlow training by optimizing a coefficient in the loss function.
problem Unstable training of MeanFlow models with non-decreasing loss and unbounded gradient variance.
method Established a theory attributing the instability to misuse of the conditional velocity field, derived the optimal coefficient, and showed practical realizations.
result Optimal coefficient yields up to 54% improvement in sample quality and monotone FID trend.
Transformer-based models overfit financial time series data, leading to increased prediction variance.
problem Forecast collapse of transformer-based models under squared loss in financial time series.
method Theoretical analysis and numerical experiments on high-frequency EUR/USD exchange rate data.
result Increased model expressivity in Transformer-based models leads to spurious fluctuations without reducing bias, resulting in higher prediction variance.
A new gradient tree boosting framework reduces variance and accelerates performance.
problem High variance in stochastic gradient boosting.
method Combining gradient tree boosting with importance sampling and a regularizer.
result Achieves a linear convergence rate on logistic loss and 2.5x--18x acceleration on LogitBoost and LambdaMART.
This work broadens calibeating to various proper losses using Bregman divergence.
problem Calibration for a wide range of proper losses.
method Regret minimization and Bregman divergence approach.
result U-calibration results for a family of Tsallis losses with logarithmic regret and dimension independence.
This work generalizes calibeating for a broader range of proper losses using Bregman divergence.
problem Calibration for a wide range of proper losses beyond Brier and log loss.
method Regret minimization based on Bregman divergence for a family of proper losses.
result U-calibration results for a family of Tsallis losses with logarithmic regret and dimension independence.
This work classifies SOC loss functions based on their gradient properties.
problem Optimizing noisy systems in stochastic optimal control.
method Grouping loss functions into classes with the same gradient expectation.
result Different loss functions have the same optimization landscape but differ in gradient variance.
Normalization techniques play an important role in supporting efficient and often more effective training of deep neural networks. While conventional methods explicitly normalize the activations, we suggest to add a loss term instead. This new loss term encourages the variance of the activations to be stable and not va…
Optimizes risk sharing with multiple models under uncertainty.
problem Risk sharing with multiple models under ambiguity.
method Constructs a mean-variance criterion using chi-squared divergence, adapts monotone preferences, and uses dual representation.
result Characterizes optimal risk sharing contract and agent's wealth process.
MeanFlow training is unstable due to misusing conditional velocity, leading to variance issues.
problem Unstable training of MeanFlow due to variance problems.
method Theoretical analysis and derivation of optimal coefficient in closed form.
result The optimal coefficient in MeanFlow training minimizes variance but not necessarily quality.
Improved diffusion bridge sampling with rKL-LD loss.
problem Improving sampling from unnormalized distributions using diffusion bridges.
method Employing the rKL-LD loss instead of the Log Variance (LV) loss for diffusion bridges.
result rKL-LD consistently outperforms LV loss in diffusion bridges.
The paper studies risk-sensitive learning schemes and provides learning bounds for empirical OCE minimizers.
problem Risk-sensitive learning aims to minimize risk-averse measures of loss.
method Proposes learning bounds for empirical OCE minimizers based on Rademacher average and variance.
result Provides two learning bounds on the performance of empirical OCE minimizers.
We propose a generic framework to calibrate accuracy and confidence of a prediction in deep neural networks through stochastic inferences. We interpret stochastic regularization using a Bayesian model, and analyze the relation between predictive uncertainty of networks and variance of the prediction scores obtained by …
This work improves structured prediction by learning the balance between signal and random noise.
problem Structured prediction with random perturbations.
method Learning the variance of randomized structured predictors to balance signal and noise.
result Learning the balance improves structured prediction effectiveness.
In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we revea…
The paper explores trade-offs between regret and variance in online learning algorithms.
problem Investigating the trade-offs between regret and variance in online learning.
method Analysis of the Exponentially Weighted Average (EWA) algorithm and its variants.
result A variant of EWA either achieves negative regret or guarantees a logarithmic bound on both variance and regret.
Improved SVRG with a coefficient reduces training loss in deep learning.
problem Demonstrating SVRG's effectiveness in deep learning.
method Introduced a multiplicative coefficient α to control SVRG's variance reduction strength and decay it linearly.
result α-SVRG consistently reduces training loss compared to baseline and standard SVRG across various model architectures and datasets.
New algorithm optimizes multi-armed bandit performance in stochastic and adversarial settings.
problem Optimizing multi-armed bandit performance in both stochastic and adversarial environments.
method Follow-the-regularized-leader method with adaptive learning rates.
result First BOBW algorithm with gap-variance-dependent regret bounds in adversarial settings.
New algorithm tackles adversarial bandits with arbitrary strategies.
problem Adversarial bandit problem against arbitrary strategies.
method Adopted master-base framework using online mirror descent method (OMD). Proposed adaptive learning rates for OMD.
result Achieved improved regret bounds compared to previous methods.
Unified analysis of stochastic gradient methods for convex and smooth optimization.
problem Minimizing composite convex and smooth functions.
method Unified convergence analysis of various stochastic gradient methods.
result Unified convergence rates for a variety of methods including proximal SGD, variance reduced methods, quantization, and coordinate descent.
New risk class defined based on loss location and deviation.
problem Risk assessment in loss distributions.
method Wrapper around smooth loss functions, M-estimators, stochastic gradient methods.
result Finite-sample stationarity guarantees for stochastic gradient methods.
Over-parameterized models reduce Out-of-Distribution (OOD) generalization loss.
problem Understanding how over-parameterized models handle non-trivial distributional shifts.
method Investigating random feature models and examining non-trivial natural distributional shifts.
result Increasing model parameterization reduces OOD loss.
Methodology to analyze traffic accidents using microscopic models.
problem Understanding and predicting traffic accidents and their impact.
method Developed a statistical approach using microscopic traffic models and SUMO.
result Approximate distribution of total losses as a mean-variance mixture.
Paper shows ERM's suboptimality due to bias, not variance.
problem Understanding why ERM fails to achieve optimal rates.
method Probabilistic and admissibility proofs for ERM in various settings.
result ERM's suboptimality is due to bias, not variance.
This paper analyzes and compares different Automated Market Maker mechanisms.
problem Impermanent loss in Constant Function Market Makers.
method Mean-Variance analysis of liquidity providers' profit and loss, comparison of different mechanisms.
result Optimized oracle-based mechanisms outperform Constant Function Market Makers.
Optimizing noise variance in VAEs balances reconstruction quality and prior regularisation.
problem Balancing reconstruction quality and prior regularisation in VAEs.
method Learning the noise variance in the Gaussian likelihood to balance the ELBO loss.
result Optimizing noise variance improves VAE-generated sample quality and uncertainty.
A new method improves SNPE for intractable likelihood models.
problem Simulation-based models with intractable likelihoods.
method Adaptive calibration kernel and variance reduction techniques.
result The proposed method provides a better approximation of the posterior.
A new algorithm reduces bias and variance in distributionally robust optimization.
problem Distributionally robust optimization with bias and variance issues.
method Prospect, a stochastic gradient-based algorithm that reduces hyperparameter tuning.
result Prospect achieves linear convergence and 2-3x faster convergence on various benchmarks.
Paper proves robust M-estimators' coordinates' normality in high dimensions.
problem High-dimensional robust M-estimators' asymptotic normality.
method Develops Stein formulae for high-dimensional random vectors on the sphere.
result Asymptotic normality holds for most coordinates of robust M-estimators with convex penalty.
We analyze the variance of Fisher information estimators in deep learning models.
problem Understanding the variance of Fisher information in deep learning models.
method Investigated two unbiased and consistent estimators of Fisher information matrix.
result The variance of estimators is influenced by the model's parametric structure.
New method allocates capital based on tail central moments for financial risk assessment.
problem Inability of CTE-based capital allocation to reflect tail behavior of losses.
method Developed TCM-based capital allocation for normal mean-variance mixture distributions.
result TCM-based method captures tail risk contributions not detected by CTE.