Paper analyzes distributional reinforcement learning with value function approximation, introducing Bellman unbiasedness and a new algorithm.
problem Improving reinforcement learning by capturing environmental stochasticity and addressing infinite dimensionality.
method Introduces Bellman unbiasedness and proposes SF-LSVI algorithm for provably efficient distributional reinforcement learning.
result Achieves a tight regret bound of O(d_E H^3/2 √K) for distributional reinforcement learning.
Researchers calculated EVaR for various distributions using Lambert function.
problem Difficulty in finding analytical representation of EVaR measure.
method Used Lambert function to calculate EVaR for multiple distributions.
result Successfully calculated EVaR for 7 specific distributions.
Researchers identify valid auxiliary functions for extreme value distributions and their max-domains of attraction.
problem Characterize valid auxiliary functions for extreme value distributions and their max-domains of attraction.
method Introduced 'universal' auxiliary functions valid for both VR and vMR representations, identified sets of valid auxiliary functions, and proposed a method for finding appropriate auxiliary functions.
result Characterized valid auxiliary functions for both VR and vMR representations for the entire MDA distribution families.
Develops robust MDPs for unknown disturbances with performance guarantees.
problem Unknown disturbance distribution in MDPs.
method Empirical distribution, sublevel set of distance function, weak convergence, concentration inequality.
result Robust optimal value function converges to true optimal value function with increasing sample sizes.
The paper establishes a Poisson integral formula for bounded pluriharmonic functions on Teichmüller space.
problem Analyzing bounded pluriharmonic functions on Teichmüller space.
method Establishing a Poisson integral formula.
result A Poisson integral formula for bounded pluriharmonic functions on Teichmüller space.
In traditional reinforcement learning, an agent maximizes the reward collected during its interaction with the environment by approximating the optimal policy through the estimation of value functions. Typically, given a state s and action a, the corresponding value is the expected discounted sum of rewards. The optima…
Policy evaluation is a key process in reinforcement learning. It assesses a given policy using estimation of the corresponding value function. When using a parameterized function to approximate the value, it is common to optimize the set of parameters by minimizing the sum of squared Bellman Temporal Differences errors…
New method uses extreme value theory to estimate neural network errors.
problem Quantifying the error of neural networks, especially for large values.
method Applying extreme value theory to approximate the distribution of error.
result Developed a new estimator for the shape parameter of the Pareto distribution.
Distributional Reinforcement Learning (RL) differs from traditional RL in that, rather than the expectation of total returns, it estimates distributions and has achieved state-of-the-art performance on Atari Games. The key challenge in practical distributional RL algorithms lies in how to parameterize estimated distrib…
The paper proposes methods to find a shared active subspace for multivariate vector-valued functions.
problem Minimizing the deviation between function evaluations in the original and reconstructed spaces.
method Manipulating gradients or SPD matrices to identify a shared structure.
result Summing SPD matrices often identifies the best shared active subspace.
Time-variant value function transfer method for RL.
problem Transfer learning with time-variant task distributions.
method Variational approach leveraging temporal structure.
result The proposed method outperforms time-invariant approach in experiments.
Randomized value functions offer a promising approach towards the challenge of efficient exploration in complex environments with high dimensional state and action spaces. Unlike traditional point estimate methods, randomized value functions maintain a posterior distribution over action-space values. This prevents the …
Optimizes risk measures given known marginal distributions of two unknown factors.
problem Determining an upper bound for spectral risk measures with unknown joint distribution.
method Introduces Maximum Spectral Measure (MSP) as a worst-case risk measure, formulated as an optimization problem with a more general objective function.
result Characterizes the continuity properties of the optimal value function and optimal solution set with respect to marginal distributions.
Bayesian reinforcement learning (BRL) offers a decision-theoretic solution for reinforcement learning. While "model-based" BRL algorithms have focused either on maintaining a posterior distribution on models or value functions and combining this with approximate dynamic programming or tree search, previous Bayesian "mo…
The paper simulates Lévy processes and their extremum and hitting time.
problem Simulating Lévy processes and their extremum and hitting time accurately and efficiently.
method Using characteristic functions and conditional characteristic functions, with conformal deformations and precalculated values on multi-grids.
result Accurate and fast simulation of Lévy processes and their extremum and hitting time.
A nonparametric approach for policy learning for POMDPs is proposed. The approach represents distributions over the states, observations, and actions as embeddings in feature spaces, which are reproducing kernel Hilbert spaces. Distributions over states given the observations are obtained by applying the kernel Bayes' …
The paper proves the convergence of Q-value for Gaussian rewards.
problem Existing proofs cannot guarantee convergence of the Q-function for Gaussian rewards.
method Using the central limit theorem and relaxing the condition to E[r(s,a)2]<∞. result Proves the convergence of the Q-function under the condition of E[r(s,a)2]<∞. Previous work in hierarchical reinforcement learning has faced a dilemma: either ignore the values of different possible exit states from a subroutine, thereby risking suboptimal behavior, or represent those values explicitly thereby incurring a possibly large representation cost because exit values refer to nonlocal a…
QR-MIX models joint state-action values as a distribution to handle randomness in MARL.
problem Randomness in rewards and observations leads to randomness in long-term returns in MARL.
method QR-MIX uses quantile regression and combines it with QMIX and IQN to model joint state-action values as a distribution.
result QR-MIX outperforms QMIX in the StarCraft Multi-Agent Challenge (SMAC) environment.
Many biological characteristics of evolutionary interest are not scalar variables but continuous functions. Here we use phylogenetic Gaussian process regression to model the evolution of simulated function-valued traits. Given function-valued data only from the tips of an evolutionary tree and utilising independent pri…
Paper introduces a new robust loss function for RL.
problem Heuristic selection of threshold parameters in quantile Huber loss.
method Derived from Wasserstein distance, captures noise in quantile values.
result Enhances robustness against outliers and enables parameter adjustment.
This work optimizes bid strategies for online auctions using measure-valued optimization.
problem Optimizing bid strategies in first-price auctions to maximize expected surplus.
method Formulates the problem as convex optimization over the joint distribution of shading parameters, adapts the distribution after each auction using a Wasserstein-proximal update.
result The proposed algorithm encourages bids on values with high expected surplus.
Paper quantifies distortion risk measures' robustness to distributional uncertainty.
problem Quantifying risk measures' robustness to distributional uncertainty.
method Employing isotonic projections, the paper derives bounds on distortion risk measures' values.
result Sharp bounds on distortion risk measures' values are provided, especially for Value-at-Risk and Range-Value-at-Risk.
Paper proposes a new framework for individualized treatment rules that generalize better across different distributions.
problem Existing individualized treatment rules may not generalize well when training and testing distributions differ.
method Distributionally robust individualized treatment rules (DR-ITR) framework that maximizes worst-case value function across close distributions.
result Calibrated DR-ITR outperforms standard ITR in generalizability.
Proposes a method to handle missing inputs in Bayesian optimization.
problem Missing values in historical data and function evaluations.
method Impute missing values using probability distributions and develop a new acquisition function.
result Improves performance of Bayesian optimization by handling missing inputs effectively.
The paper derives risk measures for metalog distributions.
problem Deriving risk measures for metalog distributions.
method Closed-form expressions for Conditional Value at Risk and first-order partial moments.
result First-order partial moments are convex with respect to metalog parameters.
Q-Distribution Guided Q-Learning corrects overestimation of uncertain OOD actions in offline RL.
problem Overestimation of Q-values for out-of-distribution actions in offline reinforcement learning.
method QDQ applies a pessimistic adjustment to Q-values in uncertain OOD regions based on a consistency model.
result QDQ improves performance on the D4RL benchmark and achieves significant improvements across many tasks.
Enhances financial market valuation and trading algorithms using distributional value functions.
problem Accurate valuation and optimal trading decisions in financial markets.
method Combines predictive knowledge and deep reinforcement learning to introduce CDG-Model, a flexible framework for financial market valuation and trading.
result Improved market valuation and trading performance through enhanced feature creation and integration of real-world market factors.
A new method for incorporating preferences in multi-objective Bayesian optimization.
problem Incorporating preferences in computationally expensive multi-objective optimization problems.
method Building independent surrogate models on each objective function and using Generalised value distribution to approximate the scalarizing function.
result The proposed multi-surrogate approach outperforms the mono-surrogate approach on benchmark and real-world problems.
Sharp bounds for distortion risk metrics under uncertain distributions.
problem Modeling risk metrics under distributional uncertainty.
method Established bounds for distortion risk metrics using specific features of underlying distributions.
result Identified worst- and best-case values of distortion risk metrics.
In reinforcement learning an agent interacts with the environment by taking actions and observing the next state and reward. When sampled probabilistically, these state transitions, rewards, and actions can all induce randomness in the observed long-term return. Traditionally, reinforcement learning algorithms average …
New method uses DistRL to estimate entire payoff distribution for financial derivatives.
problem Traditional methods focus on expected option value; this tackles risk-aware pricing.
method Reinterprets and proposes a framework using Distributional Reinforcement Learning (DistRL).
result Demonstrates enhanced risk-aware pricing and uncertainty quantification on Asian options.
We study the model of interacting agents proposed by Chatterjee et al that allows agents to both save and exchange wealth. Closed equations for the wealth distribution are developed using a mean field approximation. We show that when all agents have the same fixed savings propensity, subject to certain well defined app…
It is well known that quantifying uncertainty in the action-value estimates is crucial for efficient exploration in reinforcement learning. Ensemble sampling offers a relatively computationally tractable way of doing this using randomized value functions. However, it still requires a huge amount of computational resour…
WES improves neural network regression by stretching distribution error.
problem Improving prediction performance in neural-network-based regression.
method Proposed weighted empirical stretching (WES) loss function.
result WES outperforms existing loss functions, especially in extreme domains.
We present a method to compute the Shapley values of reconstruction errors of principal component analysis (PCA), which is particularly useful in explaining the results of anomaly detection based on PCA. Because features are usually correlated when PCA-based anomaly detection is applied, care must be taken in computing…
We consider an investor, whose portfolio consists of a single risky asset and a risk free asset, who wants to maximize his expected utility of the portfolio subject to the Value at Risk assuming a heavy tail distribution of the stock prices return. We use Markov Decision Process and dynamic programming principle to get…
Paper identifies unobserved variables from observable data.
problem Missing variables in empirical studies.
method Function mapping from observables to unobservables based on joint distribution.
result Uniqueness of latent values in each observation.
This work provides guarantees for off-policy function estimation under realizability assumptions.
problem Estimating the value function of a policy under user-specified error-measuring distributions.
method The approach involves imposing a flexible regularization on the MIS objectives to account for an arbitrary user-specified distribution.
result Exact characterization of the optimal dual solution that determines the data-coverage assumption in the case of value-function learning.
A method uses neural networks to approximate sampling distributions of test statistics.
problem Accurate modeling of p-value functions or cdfs for correct confidence set coverage.
method Uses neural networks to model the cdf of test statistics, approximating sampling distributions.
result Neural network approximations of sampling distributions are effective and simple.
It will be discussed the statistics of the extreme values in time series characterized by finite-term correlations with non-exponential decay. Precisely, it will be considered the results of numerical analyses concerning the return intervals of extreme values of the fluctuations of resistance and defect-fraction displa…
Motivated by the need for effectively summarising, modelling, and forecasting the distributional characteristics of intra-daily returns, as well as the recent work on forecasting histogram-valued time-series in the area of symbolic data analysis, we develop a time-series model for forecasting quantile-function-valued (…
Wasserstein gradient boosting predicts probability distributions for supervised learning.
problem Distribution-valued supervised learning where outputs are probability distributions.
method Fits a new weak learner to Wasserstein gradients of loss functionals of probability distributions.
result Superior performance in probabilistic prediction compared to existing methods.
In this note we prove the a pointwise ergodic theorem for functions taking values in a separable complete CAT(0)-space, analogous to Lindenstrauss' pointwise ergodic theorem for real-valued integrable functions on a probability space subject to a probability-preserving action of an amenable l.c.s.c. group, where in the…
Similarity measure for Gaussian process predictive distributions.
problem Comparing predictive distributions of Gaussian processes for correlated functions.
method Developed a similarity metric to compare predictive distributions of Gaussian processes.
result Gaussian process predictive distributions can be compared and one is enough to model two correlated functions.
PBVFs generalize across policies using learned value functions.
problem RL algorithms forget information about old policies when updating value functions to track the learned policy.
method Introduce Parameter-Based Value Functions (PBVFs) that include policy parameters in their inputs, enabling them to generalize across different policies.
result PBVFs enable zero-shot learning of new policies that outperform any policy seen during training.
The goal of regression analysis is to predict the value of a numeric outcome variable y given a vector of joint values of other (predictor) variables x. Usually a particular x-vector does not specify a repeatable value for y, but rather a probability distribution of possible y--values, p(y|x). This distribution has a l…
The paper extends risk measures to two-step approximations and studies log-concave distributions.
problem Extending classical risk measures to two-step approximations.
method Optimization problem for determining optimal regime thresholds and values for log-concave distributions.
result Conditions for the uniqueness of regime changing in log-concave distributions.