Proposes reducing random error in stochastic optimization by variance regularization.
problem Random error accumulation in stochastic optimization algorithms.
method Regularizes learning-rate based on mini-batch variances.
result Speeds up convergence and stabilizes stochastic optimization.
Dealing with high variance is a significant challenge in model-free reinforcement learning (RL). Existing methods are unreliable, exhibiting high variance in performance from run to run using different initializations/seeds. Focusing on problems arising in continuous control, we propose a functional regularization appr…
Proposes RVP to address theoretical concerns of V-REx for OOD generalization.
problem Theoretical concerns about V-REx's motivation and utility.
method Risk Variance Penalization (RVP) modifies V-REx's regularization.
result RVP discovers a robust predictor and finds invariant predictors under certain conditions.
Regularization helps resolve ambiguity in mean-variance models, improving predictive uncertainty quantification.
problem Signal-to-noise ambiguity in overparameterized mean-variance models.
method Statistical field theory framework to explain phase transition.
result Regularization reduces variability and improves predictive uncertainty quantification.
The paper analyzes how re-weighting helps in reducing variance in high-dimensional kernel methods under covariate shifts.
problem The challenge of high-dimensional kernel methods under covariate shifts and the role of re-weighting.
method Derives asymptotic expansion of high-dimensional kernels under covariate shifts, analyzes bias-variance decomposition, and characterizes the regularized kernel.
result Re-weighting helps in decreasing variance and can be seen as a data-dependent regularization.
RMDA trains structured neural networks with regularization and variance reduction.
problem Training structured neural networks with desired properties.
method RMDA algorithm for structured NNs with regularization and variance reduction.
result RMDA achieves desired structures identical to regularizer's at stationary points.
The optimization of the variance supplemented by a budget constraint and an asymmetric ℓ1 regularizer is carried out analytically by the replica method borrowed from the theory of disordered systems. The asymmetric regularizer allows us to penalize short and long positions differently, so the present treatment in…
Noise injection before gradient steps helps in regularization for neural networks.
problem Improving generalization in overparametrized neural networks.
method Injecting small noise perturbations before computing gradient steps, especially in layer-wise fashion.
result Small noise perturbations can explicitly regularize neural networks without variance explosion.
This paper balances bias and variance in adaptive importance sampling using mirror descent.
problem Large variance in adaptive importance sampling weights.
method Regularization strategy with power raised importance weights connected to mirror descent.
result The regularization parameter balances bias and variance.
Increasing variance of losses improves learning with noisy labels.
problem Learning with noisy labels and the need to penalize variance of losses.
method Designing regularizers based on the label noise transition matrix to increase variance of losses.
result Increasing variance of losses significantly improves generalization ability.
We propose a sample efficient stochastic variance-reduced cubic regularization (Lite-SVRC) algorithm for finding the local minimum efficiently in nonconvex optimization. The proposed algorithm achieves a lower sample complexity of Hessian matrix computation than existing cubic regularization based methods. At the heart…
Paper optimizes MVE network convergence and regularization.
problem Optimizing Mean Variance Estimation networks for better performance.
method Presented two key insights: warm-up period for mean optimization and separate regularization of mean and variance.
result Warm-up period and separate regularization improve MVE network performance.
This paper presents a bias-variance tradeoff of graph Laplacian regularizer, which is widely used in graph signal processing and semi-supervised learning tasks. The scaling law of the optimal regularization parameter is specified in terms of the spectral graph properties and a novel signal-to-noise ratio parameter, whi…
A large portfolio of independent returns is optimized under the variance risk measure with a ban on short positions. The no-short selling constraint acts as an asymmetric ℓ1 regularizer, setting some of the portfolio weights to zero and keeping the out of sample estimator for the variance bounded, avoiding the di…
Several applications of Reinforcement Learning suffer from instability due to high variance. This is especially prevalent in high dimensional domains. Regularization is a commonly used technique in machine learning to reduce variance, at the cost of introducing some bias. Most existing regularization techniques focus o…
Proposes volumization for neural networks to control bias-variance tradeoff.
problem Improving generalization and preventing memorization in neural networks.
method Defines a physical volume for weights, interpolating between L2 and L∞ regularization.
result Volumization interpolates between weight decay and clipping, improving generalization.
We consider the problem of streaming kernel regression, when the observations arrive sequentially and the goal is to recover the underlying mean function, assumed to belong to an RKHS. The variance of the noise is not assumed to be known. In this context, we tackle the problem of tuning the regularization parameter ada…
Improved portfolio optimization method reduces risk and improves performance.
problem Minimizing risk in large portfolios with limited data.
method Combines Tikhonov regularization and direct shrinkage of portfolio weights.
result Significantly reduces out-of-sample variance and Sharpe ratio compared to existing methods.
Dynamic CBDT improves treatment effect estimation in clinical data.
problem Estimating heterogeneous treatment effects in observational data with high accuracy and interpretability.
method Dynamic Regularized Causal Boosted Decision Trees (CBDT) integrating variance regularization and calibration.
result Significantly improved estimation accuracy and reliable coverage of true treatment effects.
A new tradeoff between regularization and sharpness improves model performance in overparameterized settings.
problem Improving model performance in overparameterized settings with minimum-norm interpolators.
method Proposes a regularization-sharpness tradeoff for overparameterized linear regression with an ℓ^p penalty.
result Empirical validation shows the tradeoff terms can distinguish performant linear interpolators.
Double descent risk in L2-regularized models explained and mitigated.
problem Risk of overparameterized models in machine learning.
method Analysis of L2-regularized models, two-layer neural networks, and CNNs.
result Double descent risk in L2-regularized models can be explained and mitigated by adjusting regularization strengths.
Statistical characteristics of deep network representations, such as sparsity and correlation, are known to be relevant to the performance and interpretability of deep learning. When a statistical characteristic is desired, often an adequate regularizer can be designed and applied during the training phase. Typically, …
New approach to portfolio optimization shows entropy regularization is ineffective.
problem Entropy regularization in mean-variance portfolio optimization under drift uncertainty.
method Combining Bayesian filtering and stochastic policy optimization.
result Entropy regularization does not accelerate learning about unknown drift.
New algorithms reduce regret in online MDPs by adapting to data and variance.
problem Adapting to both adversarial and stochastic environments in online MDPs.
method Develops algorithms based on global optimization and policy optimization, using optimistic follow-the-regularized-leader with log-barrier regularization.
result Achieves refined data-dependent and variance-dependent regret bounds.
New analysis reveals optimal regularization for ESNs, avoiding double descent.
problem Characterizing and optimizing Echo State Networks (ESNs) for precise bias-variance.
method Random matrix theory applied to ESNs in a teacher-student setting.
result ESNs achieve lower MSE with limited training samples and teacher memory.
Bayesian Markowitz portfolio problem shows entropy regularization is ineffective.
problem Entropy regularization in Bayesian Markowitz portfolio optimization.
method Combines continuous-time Bayesian filtering with stochastic policy optimization.
result Entropy regularization does not accelerate learning of unknown drift.
PEGR improves deep learning models' robustness against noisy data.
problem Learning signals from noisy data in deep learning models.
method Per-example gradient regularization (PEGR) to suppress noise.
result PEGR enhances test error and robustness against noise perturbations.
FlexAE addresses bias-variance trade-off in RAEs by learning latent priors.
problem Improving generation quality of deterministic AE models.
method Introducing flexibly learnable latent priors in WAEs to optimize the latent distribution.
result FlexAE achieves state-of-the-art performance in AE-based generative models.
The paper analyzes high-dimensional kernel regression, showing different risk curves based on data and regularization.
problem Characterizing generalization properties of high-dimensional kernel ridge regression.
method Bias-variance decomposition of the expected excess risk, considering different regularization schemes and data eigen-profiles.
result The risk curve of kernel regression can be double-descent-like, bell-shaped, or monotonic, depending on n, d, and regularization level.
This paper analyzes M-estimators under infinite-variance noise in high dimensions.
problem High-dimensional M-estimation with infinite-variance noise.
method Study of the Fenchel conjugate domain and its impact on risk.
result Exact risk of M-estimators under infinite-variance noise is derived.
Paper tackles bias-variance trade-off in missing data, proposing a dynamic framework.
problem Missing data in practical applications deteriorates model performance.
method Develops a fine-grained dynamic learning framework to jointly optimize bias and variance.
result Theoretical and empirical validation of joint bias-variance optimization.
Kernel-smoothed scores improve diffusion models by reducing memorization.
problem Diffusion models can memorize training data, leading to biased samples.
method Interpret empirical score as noisy version of true score, kernel-smoothed.
result Kernel-smoothing reduces variance and improves generalization.
VRSMD improves SMD convergence and has implicit regularization.
problem Efficiently estimating models with large datasets.
method Variance reduction in stochastic mirror descent.
result VRSMD converges to the minimum mirror interpolant.
A new algorithm reduces bias and variance in distributionally robust optimization.
problem Distributionally robust optimization with bias and variance issues.
method Prospect, a stochastic gradient-based algorithm that reduces hyperparameter tuning.
result Prospect achieves linear convergence and 2-3x faster convergence on various benchmarks.
Normalization techniques play an important role in supporting efficient and often more effective training of deep neural networks. While conventional methods explicitly normalize the activations, we suggest to add a loss term instead. This new loss term encourages the variance of the activations to be stable and not va…
We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning. We propose two new stochastic gradient methods that are based on stochastic dual averaging method with variance reduction. Our methods generate a sparser solution than the…
The paper extends a variance gamma model to quadratic functions, reducing arbitrage and computational costs.
problem Creating an arbitrage-free interpolation for option pricing models.
method Generalizing the local variance gamma model to a piecewise quadratic local variance function.
result The quadratic model results in an arbitrage-free interpolation of class C3, reducing knots and computational cost.
This paper addresses the problem of segmenting a time-series with respect to changes in the mean value or in the variance. The first case is when the time data is modeled as a sequence of independent and normal distributed random variables with unknown, possibly changing, mean value but fixed variance. The main assumpt…
We approach the continuous-time mean-variance (MV) portfolio selection with reinforcement learning (RL). The problem is to achieve the best tradeoff between exploration and exploitation, and is formulated as an entropy-regularized, relaxed stochastic control problem. We prove that the optimal feedback policy for this p…
Paper solves a complex stopping problem using regularization and HJB equations.
problem Time-inconsistent mean-variance optimal stopping problem
method Vanishing regularization method to derive HJB equations and prove existence of solutions
result Formally recovers variational inequalities for original problem
Differentiable PF via entropy-regularized OT for better inference.
problem Non-differentiability of traditional PF resampling methods.
method Entropy-regularized optimal transport for differentiable resampling.
result Convergent differentiable PF method with improved gradient estimates.
This paper develops a new theory for ensemble learning beyond variance reduction.
problem Ensemble learning's effectiveness for stable estimators is not fully explained by variance reduction.
method Develops a general weighting theory for ensemble learning, formalizing ensembles as linear operators and introducing geometric and spectral constraints.
result Structured weights can outperform uniform averaging by reshaping approximation geometry and redistributing spectral complexity.
Bayesian methods reduce variance in subspace identification for small data sets.
problem High variance in traditional subspace identification methods for large models or small sample sizes.
method Investigation of Bayesian estimation solutions (regularized and shrinkage estimators) for subspace identification.
result Bayesian estimators reduce estimation risk by up to 40% compared to traditional methods.
This paper improves generative models by using data scaling and theoretical analysis.
problem Challenges in selecting noise distributions for stable learning in generative models.
method Introduces Scale-GAN, which uses data scaling and variance-based regularization.
result Data scaling controls the bias-variance trade-off and improves stability and accuracy.
New L1 regularization controls neural network generalization error and sparsifies input dimensions.
problem Selecting the optimal number of hidden neurons in neural networks.
method Theoretical analysis of L1 regularization in two-layer neural networks. result Appropriate L1 regularization leads to near minimax optimal generalization risk bounds. Stochastic neural networks with infinite width become deterministic, reducing training variance.
problem Understanding how stochasticity in neural networks affects learning and regularization.
method Theoretical analysis of stochastic neural networks with infinite width.
result As the width of an optimized stochastic neural network increases, its predictive variance on the training set decreases to zero.
SAPPHIRE tackles ill-conditioned rERM problems with faster convergence.
problem Ill-conditioned objectives and non-smooth regularizers in rERM.
method Sketch-based preconditioning and scaled proximal mapping.
result Achieves condition-number-free linear convergence.
A simple method treats heteroscedastic variance variatively, improving model calibration and sample quality.
problem Brittle optimization impacts model likelihoods for mean and variance estimation.
method Proposes a variational approach to heteroscedastic variance, improving predictive mean and variance calibration.
result The proposed method significantly improves parameter calibration and sample quality for regression and VAEs.