Paper tackles overestimation bias in continuous control, improving performance by 25%.
problem Overestimation bias in off-policy learning.
method Truncated Quantile Critics (TQC) combines distributional representation, truncation, and ensembling of critics.
result TQC outperforms state-of-the-art methods by 25% on the Humanoid environment.
Q-Learning overestimation bias influenced by learning rate, discount factor, and reward signal.
problem Overestimation bias in Q-Learning algorithm.
method Investigated the influence of learning rate, discount factor, and reward signal on Q-Learning's overestimation bias. Tuned parameters and used an exponential moving average of reward signal.
result Q-Learning can achieve more accurate value estimates by tuning parameters and using an exponential moving average of reward signal.
Automates bias control in reinforcement learning algorithms.
problem Overestimation bias in reinforcement learning algorithms.
method Data-driven approach for automatic selection of bias control hyperparameters.
result Significant reduction in the number of interactions while maintaining performance.
We study the phenomenon of bias amplification in classifiers, wherein a machine learning model learns to predict classes with a greater disparity than the underlying ground truth. We demonstrate that bias amplification can arise via an inductive bias in gradient descent methods that results in the overestimation of the…
A widely applicable Bayesian information criterion (Watanabe, 2013) is applicable for both regular and singular models in the model selection problem. This criterion tends to overestimate the log marginal likelihood. We identify an overestimating term of a widely applicable Bayesian information criterion. Adjustment of…
EMIX minimizes surprise in multi-agent reinforcement learning.
problem Surprise and approximation bias in multi-agent reinforcement learning.
method Energy-based MIXer (EMIX) for minimizing surprise across multiple agents.
result EMIX demonstrates consistent stable performance in challenging StarCraft II scenarios.
Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.
problem Underestimation bias in double Q-learning leading to non-optimal fixed points.
method Proposes a simple approach using approximate dynamic programming to bound the target value.
result Significant improvement in learning performance over baseline algorithms in Atari benchmark tasks.
Many real world tasks require multiple agents to work together. Multi-agent reinforcement learning (RL) methods have been proposed in recent years to solve these tasks, but current methods often fail to efficiently learn policies. We thus investigate the presence of a common weakness in single-agent RL, namely value fu…
A novel Q-learning variant reduces underestimation bias in deep actor-critic methods for reinforcement learning.
problem Underestimation bias in deep actor-critic methods for reinforcement learning.
method Introduces a parameter-free Q-learning variant that combines maximum and minimum operators to bound value estimates.
result Improves state-of-the-art performance on OpenAI Gym tasks.
A new Q-learning variant reduces underestimation bias in deep reinforcement learning.
problem Underestimation bias in deep reinforcement learning policies.
method Introducing a novel, parameter-free Deep Q-learning variant.
result Significantly outperforms existing approaches and improves state-of-the-art performance.
Investigates offline RL in factorisable action spaces, overcoming overestimation bias.
problem Overestimation bias in value estimates for unseen state-action pairs.
method Value-decomposition approach in DecQN, adapted for factorised discrete action spaces.
result Demonstrates the effectiveness of factorised approach in offline RL.
SPQR improves Q-ensemble diversity in reinforcement learning.
problem Overestimation bias in Q-learning for complex tasks.
method Introduces SPQR for Q-ensemble independence regularization.
result SPQR outperforms baseline algorithms in online and offline RL benchmarks.
In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the cr…
New technique prevents Q-learning collapse by maximizing diversity among ensembles.
problem Value function collapse in ensemble Q-learning.
method Maximizing representation diversity through regularization.
result Regularized approach significantly outperforms existing methods.
New method corrects bias in estimating entropic risk for better decision-making.
problem Underestimation of entropic risk when data are limited.
method Parametric bootstrap procedure to overestimate entropic risk.
result Corrected method provides better risk estimates, leading to improved decision-making.
Improved RL policies from offline data with relaxed BC constraints.
problem Overestimation bias in offline RL due to lack of interaction with environment.
method Introducing a policy constraint via behavioural cloning (BC) and adjusting the balance between RL and BC.
result Refined policies outperform baseline and match/exceed complex alternatives.
New theory shows how learning algorithms can create a bias towards negative outcomes.
problem Negativity bias in adaptive learning algorithms.
method Generalization of the Hot Stove Effect to settings with negative estimates leading to smaller sample sizes.
result Negativity bias persists even when negative estimates do not lead to avoidance.
Machine learning models predict brain age with systematic bias, corrected in this study.
problem Systematic bias in machine learning regression models for brain age prediction.
method General constrained optimization approach to correct bias.
result Our method effectively eliminates the bias from brain age predictions.
Importance-weighting is a popular and well-researched technique for dealing with sample selection bias and covariate shift. It has desirable characteristics such as unbiasedness, consistency and low computational complexity. However, weighting can have a detrimental effect on an estimator as well. In this work, we empi…
MFVI can overestimate predictive variance compared to the exact posterior
problem MFVI underestimates posterior variance
method Analyzing conjugate Bayesian Linear Regression
result MFVI can overestimate predictive variance compared to the exact posterior
The breakthrough of deep Q-Learning on different types of environments revolutionized the algorithmic design of Reinforcement Learning to introduce more stable and robust algorithms, to that end many extensions to deep Q-Learning algorithm have been proposed to reduce the variance of the target values and the overestim…
Method learns neural network to overestimate reference function with guarantees.
problem Learning a neural network to overestimate a reference function on a given domain.
method Two-step process: constructing Majoring Points and optimizing a neural network.
result The learned neural network overestimates the reference function on the domain.
Performance of investment managers are evaluated in comparison with benchmarks, such as financial indices. Due to the operational constraint that most professional databases do not track the change of constitution of benchmark portfolios, standard tests of performance suffer from the "look-ahead benchmark bias," when t…
Second-order methods fail to fully quantify epistemic uncertainty, leading to biased predictions.
problem Incomplete quantification of epistemic uncertainty in machine learning models.
method Analysis of existing second-order uncertainty estimation methods.
result Current methods overestimate aleatoric uncertainty and underestimate epistemic uncertainty, leading to biased predictions.
New CV method reduces bias in spatial prediction models.
problem Bias in standard cross-validation due to uneven sampling.
method Target-Weighted Cross-Validation (TWCV) framework.
result Weighted CV approaches reduce bias in prediction error.
Simplifies RL training with fewer techniques, reducing bias and instability.
problem Training instabilities and high sample complexity in RL.
method Introduced a simple deterministic policy gradient, used propensity estimation, and delayed policy updates.
result Improved performance and reduced sample complexity through these techniques.
The paper examines various RL algorithms to address overestimation and noise issues.
problem Overestimation and noise in deep reinforcement learning algorithms.
method Analysis of DQN, double DQN, DDPG, TD3, and hill climbing algorithms.
result Optimal noise settings for TD3 in specific environments.
Bayesian models overestimate clusters, but practical summaries can correct this.
problem Bayesian mixture models overestimate the number of clusters.
method Simulations and gene expression data analysis using MCMC summarisation.
result Overestimation is limited in finite samples and can be corrected, but misspecification leads to significant overestimation.
Develops a new method for uncertainty quantification in high-dimensional learning.
problem Challenges in uncertainty quantification in high-dimensional regression or learning problems.
method Data-driven approach for UQ that corrects bias terms from training data.
result Non-asymptotic confidence intervals that avoid overestimating uncertainty.
We correct a mistake in the published version of our paper. Our new conclusion is that the "implied leverage effect" for single stocks is underestimated by option markets for short maturities and overestimated for long maturities, while it is always overestimated for OEX options, except for the shortest maturities wher…
The paper examines skill estimation and variance under model misspecification in IRT.
problem Underestimation and overestimation of skills when non-compensatory model is misspecified as compensatory.
method Theoretical approach to analyze underestimation and overestimation of skills and variance.
result Overestimation of skills occurs around the origin and asymptotic variance differs under model misspecification.
Study shows statistical biases can mislead transformer models, impairing their generalization.
problem Statistical biases in transformers affect their ability to generalize.
method Evaluated transformer models on synthetic algorithmic tasks with varying statistical biases.
result Statistical biases lead to overestimation of transformer models' generalization capabilities.
New model improves volatility forecasting by reducing overestimation and underestimation.
problem SVR-GARCH model overestimates or underestimates volatility, hindering peak or trough behaviors.
method Proposes blending ARCH and augmented blending-ARCH models to improve volatility forecasting.
result Empirical results show improved volatility forecasting ability.
Study finds racial bias in pulse oximeter readings has minimal impact on ICU ventilation rates.
problem Racial disparities in pulse oximeter readings affect clinical decisions in ICU settings.
method Causal inference using path-specific effects and doubly robust estimator.
result Minimal impact of racial discrepancies on invasive ventilation rates, but more pronounced on ventilation duration.
A key problem in research on adversarial examples is that vulnerability to adversarial examples is usually measured by running attack algorithms. Because the attack algorithms are not optimal, the attack algorithms are prone to overestimating the size of perturbation needed to fool the target model. In other words, the…
GUM tackles MARL by avoiding overestimation through state-marginal restriction.
problem Overestimation of values in large joint state-action spaces.
method Greedy UnMixing through state-marginal restriction and unmixing.
result Superior performance compared to existing Q-learning and general MARL algorithms.
Q-Distribution Guided Q-Learning corrects overestimation of uncertain OOD actions in offline RL.
problem Overestimation of Q-values for out-of-distribution actions in offline reinforcement learning.
method QDQ applies a pessimistic adjustment to Q-values in uncertain OOD regions based on a consistency model.
result QDQ improves performance on the D4RL benchmark and achieves significant improvements across many tasks.
Catastrophic forgetting is a critical challenge in training deep neural networks. Although continual learning has been investigated as a countermeasure to the problem, it often suffers from the requirements of additional network components and the limited scalability to a large number of tasks. We propose a novel appro…
Compensation methods correct overestimation of adversarial robustness in neural networks.
problem Overestimation of adversarial robustness using first-order attack methods.
method Proposed compensation methods address inaccurate gradient computation and reduce backpropagations.
result Empirical evaluation of adversarial robustness is improved with these methods.
AEA dynamically aggregates ensemble targets for actor-critic learning.
problem Static ensemble aggregation methods struggle with overestimation bias and variance.
method Adaptive Ensemble Aggregation (AEA) dynamically constructs ensemble-based targets.
result AEA converges to optimal variance reduction and maximal Fisher information.
Develops a fair relational model learning algorithm.
problem Fairness in machine learning models for relational data.
method Fair-A3SL, a fairness-aware structure learning algorithm for relational structures.
result Demonstrates effectiveness in learning fair, interpretable, and expressive structures.
Deep Reinforcement Learning improves with Weighted Q-Learning to reduce bias and uncertainty.
problem Overestimation and high variance in Q-Learning cause learning algorithms to diverge in complex environments.
method Deep Weighted Q-Learning (Deep WQL) uses Dropout and Monte Carlo sampling to approximate WQL's weights and reduce bias.
result Deep WQL reduces bias and improves performance on benchmarks compared to existing methods.
We study the consistency of sample mean-variance portfolios of arbitrarily high dimension that are based on Bayesian or shrinkage estimation of the input parameters as well as weighted sampling. In an asymptotic setting where the number of assets remains comparable in magnitude to the sample size, we provide a characte…
The study aims to prevent unfair content presentation in recommender systems.
problem Over- and under-presentation of content leads to biased user preference estimates.
method Two models are considered: one that ignores systematic and limited exposure, and another that conditions on limited exposure.
result Ignoring systematic presentations overestimates promoted options and underestimates censored alternatives.
Markov random fields (MRFs) are difficult to evaluate as generative models because computing the test log-probabilities requires the intractable partition function. Annealed importance sampling (AIS) is widely used to estimate MRF partition functions, and often yields quite accurate results. However, AIS is prone to ov…
In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. Conventionally, the loss function is defined as the temporal difference between the action value and…
Machine learning confound removal biases results, leading to misleading predictions.
problem Common confound removal methods in machine learning lead to misleading predictions.
method Featurewise removal of confound variance by linear regression before applying ML.
result This common deconfounding approach can leak information, amplifying null or moderate effects.
AFLite filters dataset biases to improve model generalization.
problem Overfitting to dataset biases degrades model performance on out-of-distribution samples.
method AFLite adversarially filters dataset biases, reducing their impact on model performance.
result Filtered datasets yield better generalization to out-of-distribution tasks.