The paper analyzes the bias-variance tradeoff for Bregman divergences.
problem Understanding the bias-variance tradeoff for Bregman divergences.
method Analyzes the bias-variance tradeoff through operations in dual space.
result Derives several results including a generalized law of total variance and ensembling operations.
Neural networks exhibit unimodal variance with model complexity, improving generalization.
problem The classical bias-variance trade-off does not apply to neural networks, leading to better generalization with larger models.
method Measured bias and variance of neural networks, confirmed empirically and theoretically.
result Neural networks show unimodal variance, leading to a double descent risk curve.
Deep learning models show bias and variance are aligned, not in trade-off.
problem The classical bias-variance trade-off in deep learning models.
method Empirical evidence and theoretical analysis of bias and variance in deep learning models.
result Squared bias is approximately equal to variance for correctly classified sample points in deep learning models.
The paper explores the trade-off between bias and variance in high-dimensional models.
problem Understanding the unavoidable trade-off between bias and variance in high-dimensional statistical models.
method Proposes a general strategy to obtain lower bounds on the variance of estimators with a specified bias, and applies it to various statistical models.
result Shows the extent to which the bias-variance trade-off is unavoidable and quantifies the performance loss for methods that do not balance it.
New insights into bias and variance in over-parameterized models.
problem Understanding bias and variance in over-parameterized models.
method Analytic expressions derived from statistical physics for two minimal models.
result Over-parameterized models can overfit even in noiseless conditions.
Ensembles improve classifier performance by reducing bias, not variance.
problem Improving classifier performance through ensemble methods.
method Extended bias-variance decomposition for classification tasks, introducing dual reparameterization.
result Ensembling reduces bias in classifiers, contrary to the traditional view.
The bias-variance tradeoff tells us that as model complexity increases, bias falls and variances increases, leading to a U-shaped test error curve. However, recent empirical results with over-parameterized neural networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decrea…
Generalizes bias-variance decomposition for Bregman divergences.
problem No specific problem stated; generalization of bias-variance for Bregman divergences.
method Provided a generalization of the bias-variance decomposition for Bregman divergences.
result A clear, standalone derivation of the bias-variance decomposition for Bregman divergences.
Paper tackles bias-variance trade-off in missing data, proposing a dynamic framework.
problem Missing data in practical applications deteriorates model performance.
method Develops a fine-grained dynamic learning framework to jointly optimize bias and variance.
result Theoretical and empirical validation of joint bias-variance optimization.
Paper explores robust regression methods and their bias-variance trade-off.
problem Understanding the trade-off between robust estimation and optimization methods.
method Examines traditional outlier-resistant robust estimation and robust optimization.
result Both methods follow converse strategies due to a bias-variance trade-off.
Adversarial training leads to large generalization gap, decomposed into bias and variance.
problem Understanding the large generalization gap in adversarially trained models.
method Bias-Variance decomposition of test risk as a function of adversarial perturbation radius.
result Bias increases monotonically with adversarial perturbation radius and is dominant in test risk.
We analyze bias-variance of margin losses.
problem Understanding model overfitting/underfitting.
method Bias-variance decomposition for strictly convex margin losses.
result Expected risk decomposes into central model risk and data variation.
SkMM selects data for finetuning by balancing bias and variance.
problem Balancing bias and variance in high-dimensional finetuning.
method Gradient sketching for bias reduction and moment matching for variance reduction.
result Gradient sketching selects samples efficiently and accurately.
Study explores bias-variance in adversarial machine learning.
problem Understanding adversarial machine learning's impact on bias and variance.
method Investigates bias-variance trade-offs in deep neural networks using MSE and cross-entropy.
result Derives bias-variance trade-offs for classification and regression.
The main goal of this thesis is to point out that the bias-variance tradeoff is not always true (e.g. in neural networks). We advocate for this lack of universality to be acknowledged in textbooks and taught in introductory courses that cover the tradeoff. We first review the history of the bias-variance tradeoff, its …
Study shows exponential error reduction in multiclass classification without bias-variance trade-off.
problem Multiclass classification with margin conditions.
method Analysis of classification error under hard-margin conditions.
result Exponential decrease in classification error without bias-variance trade-off.
New method reduces model bias and variance by adjusting training sample weights based on label uncertainty.
problem Tradeoff between model bias and variance in classification models.
method Estimate label uncertainty, adjust training sample weights, and fine-tune decision boundary.
result Improves model performance and reduces variance in physical activity recognition.
Safe-FinRL uses DRL for high-frequency stock trading, reducing bias and variance.
problem Challenges in applying DRL to high-frequency stock trading, especially bias and variance issues.
method Safe-FinRL separates financial time series into near-stationary short environments and uses Trace-SAC with a general retrace operator.
result Safe-FinRL reduces bias and variance significantly in near-stationary financial environments.
Unified theory explains diversity in ensemble learning.
problem Explaining diversity in ensemble learning across various scenarios.
method Developed a framework revealing diversity as a hidden dimension in bias-variance decomposition.
result Proved exact bias-variance-diversity decompositions for multiple losses in regression and classification.
Bayesian imputation optimizes bias-variance tradeoff in time-series data.
problem Look-ahead bias in imputation of missing time-series data.
method Wasserstein interpolation for Bayesian posterior consensus distribution.
result Optimal control of look-ahead bias and variance in imputation.
Deep learning models can have low bias and variance, contrary to classical theory.
problem Understanding the performance of deep learning models at high complexity.
method Developed a fine-grained bias-variance decomposition for random feature kernel regression, analyzing the effects of sampling, initialization, and labels.
result The variance terms exhibit non-monotonic behavior and can diverge at the interpolation boundary, even in the absence of label noise.
Optimal feature transfer identified through bias-variance analysis.
problem Optimizing feature transfer in transfer learning.
method Simple linear model with fine-grained bias-variance decomposition.
result Optimal pretrained feature transform is naturally sparse.
New estimator reduces bias and variance in tensor and matrix denoising.
problem Optimal bias-variance tradeoff in matrix and tensor estimation.
method One-step variant of higher-order SVD (HOSVD) estimator.
result Achieves optimal bias-variance tradeoff in both matrix and tensor settings.
Paper unifies bias and variance models for classification.
problem Different frameworks for bias and variance in classification.
method Unified Tumer & Ghosh and James approaches.
result Closed form relationships between 0/1 loss and squared error loss.
New insights into bias-variance tradeoff for data-driven optimization under local misspecification.
problem Understanding the relative performance of SAA, IEO, and ETO under local misspecification.
method Developed a local misspecification perspective using contiguity theory in statistics.
result Explicit expressions for decision bias and geometric understanding of variance.
The paper analyzes how re-weighting helps in reducing variance in high-dimensional kernel methods under covariate shifts.
problem The challenge of high-dimensional kernel methods under covariate shifts and the role of re-weighting.
method Derives asymptotic expansion of high-dimensional kernels under covariate shifts, analyzes bias-variance decomposition, and characterizes the regularized kernel.
result Re-weighting helps in decreasing variance and can be seen as a data-dependent regularization.
New estimator reduces bias-variance tradeoff in Markovian interference experiments.
problem Estimating impact of interventions in systems with limited resources.
method Differences-In-Q (DQ) estimator for on-policy policy evaluation.
result DQ estimator has exponentially smaller variance than off-policy methods.
Analyzes bias-variance in overparameterized linear models using random features.
problem Understanding bias-variance trade-off in overparameterized models.
method Zero-temperature cavity method and random matrix theory.
result Three phase transitions in the linear random features model.
This work introduces a bias-variance decomposition for proper scores, improving uncertainty estimation in predictive models.
problem Reliable uncertainty estimation for predictions in safety-critical applications, especially under domain drift.
method Developed a general bias-variance decomposition for proper scores, introducing the Bregman Information as the variance term.
result The decomposition provides novel formulations for different predictive tasks, including classification and model ensembles.
Hierarchical probabilistic models are able to use a large number of parameters to create a model with a high representation power. However, it is well known that increasing the number of parameters also increases the complexity of the model which leads to a bias-variance trade-off. Although it is a classical problem, t…
There exist a number of reinforcement learning algorithms which learnby climbing the gradient of expected reward. Their long-runconvergence has been proved, even in partially observableenvironments with non-deterministic actions, and without the need fora system model. However, the variance of the gradient estimator ha…
Random forests reduce bias and variance, especially in low SNR settings.
problem Reducing bias and variance in machine learning models, particularly in low SNR scenarios.
method Empirical study of random forests and bagging ensembles, focusing on the importance of mtry tuning. result Random forests reduce both bias and variance, outperforming bagging ensembles in high SNR settings.
The paper analyzes and mitigates biases in scalable Gaussian Process methods.
problem Modeling biases in scalable Gaussian Process methods.
method Randomized truncation estimators to eliminate bias in exchange for increased variance.
result Randomized truncation estimators meaningfully outperform biased counterparts with minimal additional computation.
Bias - variance decomposition of the expected error defined for regression and classification problems is an important tool to study and compare different algorithms, to find the best areas for their application. Here the decomposition is introduced for the survival analysis problem. In our experiments, we study bias -…
Gradient-based methods for optimisation of objectives in stochastic settings with unknown or intractable dynamics require estimators of derivatives. We derive an objective that, under automatic differentiation, produces low-variance unbiased estimators of derivatives at any order. Our objective is compatible with arbit…
This paper balances bias and variance in adaptive importance sampling using mirror descent.
problem Large variance in adaptive importance sampling weights.
method Regularization strategy with power raised importance weights connected to mirror descent.
result The regularization parameter balances bias and variance.
Depth uncertainty networks don't improve with bias correction, contrary to expectations.
problem Improving performance in active learning with overparameterised models like NNs.
method Depth uncertainty networks, compared to underparameterised models, show no improvement in performance with bias correction.
result Depth uncertainty networks do not improve with bias correction, unlike underparameterised models.
Random forest performance depends on SNR and covariate characteristics.
problem Understanding when random forests perform well.
method Systematic analysis of out-of-sample MSE for different SNR scenarios.
result Randomization effectiveness depends on SNR and covariate characteristics.
New algorithm offers costless model selection in contextual bandits.
problem Minimizing cumulative regret in stochastic contextual bandits.
method Gradually increasing class complexity and adapting to the simplest class with dominant estimation variance.
result Costless model selection is feasible under certain conditions, providing improved regret guarantees.
In this note, we study the relationship between the variational gap and the variance of the (log) likelihood ratio. We show that the gap can be upper bounded by some form of dispersion measure of the likelihood ratio, which suggests the bias of variational inference can be reduced by making the distribution of the like…
Boundary effects inflate variance in Gaussian processes, leading to acquisition bias.
problem Boundary-induced acquisition bias in Gaussian processes.
method Traced root cause to geometric mechanism of kernel truncation at domain boundaries.
result Boundary effects create distortion that worsens with dimensionality, affecting acquisition behavior.
Societal bias towards certain communities is a big problem that affects a lot of machine learning systems. This work aims at addressing the racial bias present in many modern gender recognition systems. We learn race invariant representations of human faces with an adversarially trained autoencoder model. We show that …
BR-SNIS reduces bias in self-normalized IS without increasing variance.
problem Bias in self-normalized IS.
method Iterated sampling-importance resampling (ISIR) to form a bias-reduced estimator.
result Significant reduction in bias without increasing variance.
Analyzes how diffusion models learn, revealing a spectral bias in structure mastery.
problem Understanding the learning dynamics and bias in diffusion models.
method Developed an analytical framework using a Gaussian-equivalence principle to solve gradient-flow dynamics and integrate probability-flow ODEs.
result Exposes a universal inverse-variance spectral law: high-variance structure is mastered faster than low-variance detail.
We consider the least-squares regression problem and provide a detailed asymptotic analysis of the performance of averaged constant-step-size stochastic gradient descent (a.k.a. least-mean-squares). In the strongly-convex case, we provide an asymptotic expansion up to explicit exponentially decaying terms. Our analysis…
VRER selectively reuses past observations to reduce variance in policy optimization.
problem Lack of effective experience replay for accelerating policy optimization in complex systems.
method Variance Reduction Experience Replay (VRER) framework that selectively reuses informative samples.
result VRER reduces gradient variance and improves policy learning over state-of-the-art algorithms.
In this paper we propose using the principle of boosting to reduce the bias of a random forest prediction in the regression setting. From the original random forest fit we extract the residuals and then fit another random forest to these residuals. We call the sum of these two random forests a \textit{one-step boosted …
Ensemble methods for classification and clustering have been effectively used for decades, while ensemble learning for outlier detection has only been studied recently. In this work, we design a new ensemble approach for outlier detection in multi-dimensional point data, which provides improved accuracy by reducing err…