Reduces overestimation bias in multi-agent RL, improving performance.
problem Value function overestimation bias in multi-agent RL.
method Double centralized critics to reduce overestimation bias.
result Significant improvement in performance on mixed tasks.
In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the cr…
Q-Learning overestimation bias influenced by learning rate, discount factor, and reward signal.
problem Overestimation bias in Q-Learning algorithm.
method Investigated the influence of learning rate, discount factor, and reward signal on Q-Learning's overestimation bias. Tuned parameters and used an exponential moving average of reward signal.
result Q-Learning can achieve more accurate value estimates by tuning parameters and using an exponential moving average of reward signal.
Q-Distribution Guided Q-Learning corrects overestimation of uncertain OOD actions in offline RL.
problem Overestimation of Q-values for out-of-distribution actions in offline reinforcement learning.
method QDQ applies a pessimistic adjustment to Q-values in uncertain OOD regions based on a consistency model.
result QDQ improves performance on the D4RL benchmark and achieves significant improvements across many tasks.
In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. Conventionally, the loss function is defined as the temporal difference between the action value and…
New dropout technique reduces variance and overestimation in deep Q-Learning.
problem Reduction of variance and overestimation in deep Q-Learning.
method Using Dropout techniques to reduce variance and overestimation in deep Q-Learning.
result Demonstrated effectiveness in enhancing stability and reducing both variance and overestimation.
Method learns neural network to overestimate reference function with guarantees.
problem Learning a neural network to overestimate a reference function on a given domain.
method Two-step process: constructing Majoring Points and optimizing a neural network.
result The learned neural network overestimates the reference function on the domain.
A novel Q-learning variant reduces underestimation bias in deep actor-critic methods for reinforcement learning.
problem Underestimation bias in deep actor-critic methods for reinforcement learning.
method Introduces a parameter-free Q-learning variant that combines maximum and minimum operators to bound value estimates.
result Improves state-of-the-art performance on OpenAI Gym tasks.
GUM tackles MARL by avoiding overestimation through state-marginal restriction.
problem Overestimation of values in large joint state-action spaces.
method Greedy UnMixing through state-marginal restriction and unmixing.
result Superior performance compared to existing Q-learning and general MARL algorithms.
EMIX minimizes surprise in multi-agent reinforcement learning.
problem Surprise and approximation bias in multi-agent reinforcement learning.
method Energy-based MIXer (EMIX) for minimizing surprise across multiple agents.
result EMIX demonstrates consistent stable performance in challenging StarCraft II scenarios.
A new Q-learning variant reduces underestimation bias in deep reinforcement learning.
problem Underestimation bias in deep reinforcement learning policies.
method Introducing a novel, parameter-free Deep Q-learning variant.
result Significantly outperforms existing approaches and improves state-of-the-art performance.
Paper analyzes finite-time convergence of double Q-learning.
problem Overestimation issue in Q-learning.
method Finite-time analysis of double Q-learning.
result Convergence to ε-accurate neighborhood in finite iterations.
New technique prevents Q-learning collapse by maximizing diversity among ensembles.
problem Value function collapse in ensemble Q-learning.
method Maximizing representation diversity through regularization.
result Regularized approach significantly outperforms existing methods.
Investigates offline RL in factorisable action spaces, overcoming overestimation bias.
problem Overestimation bias in value estimates for unseen state-action pairs.
method Value-decomposition approach in DecQN, adapted for factorised discrete action spaces.
result Demonstrates the effectiveness of factorised approach in offline RL.
Selective state-adaptive regularization improves offline RL performance.
problem Extrapolation errors and value overestimation in static dataset RL.
method State-adaptive regularization coefficients trust Bellman-driven results selectively.
result Significant improvement in performance on D4RL benchmark.
Paper tackles overestimation bias in continuous control, improving performance by 25%.
problem Overestimation bias in off-policy learning.
method Truncated Quantile Critics (TQC) combines distributional representation, truncation, and ensembling of critics.
result TQC outperforms state-of-the-art methods by 25% on the Humanoid environment.
Catastrophic forgetting is a critical challenge in training deep neural networks. Although continual learning has been investigated as a countermeasure to the problem, it often suffers from the requirements of additional network components and the limited scalability to a large number of tasks. We propose a novel appro…
Paper addresses underestimation bias in double Q-learning, proposing a method to improve learning performance.
problem Underestimation bias in double Q-learning leading to non-optimal fixed points.
method Proposes a simple approach using approximate dynamic programming to bound the target value.
result Significant improvement in learning performance over baseline algorithms in Atari benchmark tasks.
MFVI can overestimate predictive variance compared to the exact posterior
problem MFVI underestimates posterior variance
method Analyzing conjugate Bayesian Linear Regression
result MFVI can overestimate predictive variance compared to the exact posterior
The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on explorati…
Markov random fields (MRFs) are difficult to evaluate as generative models because computing the test log-probabilities requires the intractable partition function. Annealed importance sampling (AIS) is widely used to estimate MRF partition functions, and often yields quite accurate results. However, AIS is prone to ov…
This paper makes one step forward towards characterizing a new family of \textit{model-free} Deep Reinforcement Learning (DRL) algorithms. The aim of these algorithms is to jointly learn an approximation of the state-value function (V), alongside an approximation of the state-action value function (Q). Our analysis…
Support vector data description (SVDD) is a popular technique for detecting anomalies. The SVDD classifier partitions the whole space into an inlier region, which consists of the region near the training data, and an outlier region, which consists of points away from the training data. The computation of the SVDD class…
A widely applicable Bayesian information criterion (Watanabe, 2013) is applicable for both regular and singular models in the model selection problem. This criterion tends to overestimate the log marginal likelihood. We identify an overestimating term of a widely applicable Bayesian information criterion. Adjustment of…
SPQR improves Q-ensemble diversity in reinforcement learning.
problem Overestimation bias in Q-learning for complex tasks.
method Introduces SPQR for Q-ensemble independence regularization.
result SPQR outperforms baseline algorithms in online and offline RL benchmarks.
The paper examines various RL algorithms to address overestimation and noise issues.
problem Overestimation and noise in deep reinforcement learning algorithms.
method Analysis of DQN, double DQN, DDPG, TD3, and hill climbing algorithms.
result Optimal noise settings for TD3 in specific environments.
Investors mispricing volatility and jump sensitivity in Delta hedging models still super-replicate the true claim.
problem Investors misestimate volatility and jump sensitivity in Delta hedging models.
method Analyzes the robustness of Delta hedging in jump-diffusion models, proving stochastic flow properties and convexity of value functions.
result An erroneously computed Delta strategy super-replicates the true claim in expectation under a wide class of models.
Leasing is a popular channel to market new cars. Pricing a leasing contract is complicated because the leasing rate embodies an expectation of the residual value of the car after contract expiration. To aid lessors in their pricing decisions, the paper develops resale price forecasting models. A peculiarity of the leas…
Bayesian models overestimate clusters, but practical summaries can correct this.
problem Bayesian mixture models overestimate the number of clusters.
method Simulations and gene expression data analysis using MCMC summarisation.
result Overestimation is limited in finite samples and can be corrected, but misspecification leads to significant overestimation.
CQL learns conservative Q-functions to improve offline RL performance.
problem Leveraging large, static datasets in reinforcement learning without further interaction.
method Conservative Q-learning (CQL) which learns a conservative Q-function to lower-bound policy values.
result CQL substantially outperforms existing offline RL methods, often achieving 2-5 times higher final returns.
We correct a mistake in the published version of our paper. Our new conclusion is that the "implied leverage effect" for single stocks is underestimated by option markets for short maturities and overestimated for long maturities, while it is always overestimated for OEX options, except for the shortest maturities wher…
The paper examines skill estimation and variance under model misspecification in IRT.
problem Underestimation and overestimation of skills when non-compensatory model is misspecified as compensatory.
method Theoretical approach to analyze underestimation and overestimation of skills and variance.
result Overestimation of skills occurs around the origin and asymptotic variance differs under model misspecification.
We consider tackling a single-agent RL problem by distributing it to n learners. These learners, called advisors, endeavour to solve the problem from a different focus. Their advice, taking the form of action values, is then communicated to an aggregator, which is in control of the system. We show that the local plan…
Study shows statistical biases can mislead transformer models, impairing their generalization.
problem Statistical biases in transformers affect their ability to generalize.
method Evaluated transformer models on synthetic algorithmic tasks with varying statistical biases.
result Statistical biases lead to overestimation of transformer models' generalization capabilities.
New model improves volatility forecasting by reducing overestimation and underestimation.
problem SVR-GARCH model overestimates or underestimates volatility, hindering peak or trough behaviors.
method Proposes blending ARCH and augmented blending-ARCH models to improve volatility forecasting.
result Empirical results show improved volatility forecasting ability.
A key problem in research on adversarial examples is that vulnerability to adversarial examples is usually measured by running attack algorithms. Because the attack algorithms are not optimal, the attack algorithms are prone to overestimating the size of perturbation needed to fool the target model. In other words, the…
BCPO optimizes offline RL policies by converting uncertainty into conservative bounds.
problem Offline RL's fragility under distribution shifts and model errors.
method Bayesian approach with credible lower bounds and KL regularization.
result BCPO yields an uncertainty-calibrated policy that avoids exploiting model errors.
We methodologically address the problem of Q-value overestimation in deep reinforcement learning to handle high-dimensional state spaces efficiently. By adapting concepts from information theory, we introduce an intrinsic penalty signal encouraging reduced Q-value estimates. The resultant algorithm encompasses a wide r…
We focus on variational inference in dynamical systems where the discrete time transition function (or evolution rule) is modelled by a Gaussian process. The dominant approach so far has been to use a factorised posterior distribution, decoupling the transition function from the system states. This is not exact in gene…
Compensation methods correct overestimation of adversarial robustness in neural networks.
problem Overestimation of adversarial robustness using first-order attack methods.
method Proposed compensation methods address inaccurate gradient computation and reduce backpropagations.
result Empirical evaluation of adversarial robustness is improved with these methods.
New theory shows how learning algorithms can create a bias towards negative outcomes.
problem Negativity bias in adaptive learning algorithms.
method Generalization of the Hot Stove Effect to settings with negative estimates leading to smaller sample sizes.
result Negativity bias persists even when negative estimates do not lead to avoidance.
A new method for SVGD reduces variance in high dimensions.
problem High-dimensional variance in SVGD.
method Grassmann Stein Variational Gradient Descent (GSVGD) projects onto arbitrary subspaces and uses coupled Grassmann-valued diffusion.
result GSVGD explores high-dimensional problems with intrinsic low-dimensional structure efficiently.
The study aims to prevent unfair content presentation in recommender systems.
problem Over- and under-presentation of content leads to biased user preference estimates.
method Two models are considered: one that ignores systematic and limited exposure, and another that conditions on limited exposure.
result Ignoring systematic presentations overestimates promoted options and underestimates censored alternatives.
Automates bias control in reinforcement learning algorithms.
problem Overestimation bias in reinforcement learning algorithms.
method Data-driven approach for automatic selection of bias control hyperparameters.
result Significant reduction in the number of interactions while maintaining performance.
Support vector data description (SVDD) is a popular anomaly detection technique. The SVDD classifier partitions the whole data space into an inlier region, which consists of the region near the training data, and an outlier region, which consists of points away from the training data. The computation of the SVDD classi…
Machine learning models predict brain age with systematic bias, corrected in this study.
problem Systematic bias in machine learning regression models for brain age prediction.
method General constrained optimization approach to correct bias.
result Our method effectively eliminates the bias from brain age predictions.
Enhanced tracking control for AUVs with improved policy gradient method.
problem Trajectory tracking problem for underactuated AUVs with unknown dynamics and constrained inputs.
method Hybrid actors-critics architecture with multiple actors and critics, Pseudo Q-learning, and deterministic policy gradient.
result High-level tracking control accuracy and stable learning of AUVs.
Proposes a conservative LR estimator for infrequent data near a frequency threshold.
problem Overestimation of likelihood ratios for infrequent data near a frequency threshold.
method Conservative likelihood ratio estimator for frequencies slightly above a threshold.
result Improves prediction accuracy in named entity context prediction.