A new algorithm improves offline reinforcement learning robustness.
problem Finding optimal policies in perturbed environments from offline data.
method Doubly Pessimistic Model-based Policy Optimization (P^2MPO) framework.
result Proves sample efficiency with robust partial coverage data.
Proposes a new Bayesian learning method for optimal treatment regimes.
problem Sub-optimal policies in offline data due to lack of exploration.
method Integrates pessimism principle with Thompson sampling and Bayesian machine learning.
result Derives a credible set that uniformly lower bounds the optimal Q-function.
The paper introduces Bellman-consistent pessimism to improve offline reinforcement learning without overly pessimistic bias.
problem Offline reinforcement learning's challenge of discovering good policies without exhaustive exploration.
method Introduces Bellman-consistent pessimism for function approximation, improving sample complexity and adaptability.
result Improves sample complexity by O(d) in the action space finite case, and automatically adapts to bias-variance tradeoff. USAC balances pessimism and optimism in actor-critic training for better exploration and performance.
problem Excessive pessimism limits exploration, while excessive optimism leads to high-risk behaviors.
method Utility Soft Actor-Critic (USAC) dynamically adapts exploration based on critic uncertainty.
result USAC consistently outperforms state-of-the-art algorithms in continuous control tasks.
Unified framework for analyzing pessimism in off-policy learning with regularized importance sampling.
problem High variance in importance weighting for off-policy learning.
method Unified PAC-Bayesian study of pessimism with regularized importance sampling.
result Derivation of a tractable PAC-Bayesian generalization bound for common importance weight regularizations.
The paper improves Q-learning by incorporating pessimism for better sample efficiency.
problem Improving sample efficiency in asynchronous Q-learning with non-i.i.d. data.
method Developed an algorithmic framework that incorporates the principle of pessimism into asynchronous Q-learning, penalizing infrequently-visited state-action pairs based on suitable lower confidence bounds (LCBs).
result Achieved near-optimal sample complexity, providing theoretical support for the use of pessimism in non-i.i.d. data.
New method optimizes offline linear bandits using different confidence sets.
problem Optimizing offline learning for linear contextual bandits.
method Introduces a family of pessimistic learning rules based on ℓp confidence sets. result The π^∞ rule achieves minimax performance and strictly dominates other predictors. Pessimistic RL algorithm improves offline RL performance.
problem Insufficient dataset coverage in offline RL.
method Proposes a pessimistic variant of value iteration (PEVI) with a penalty function.
result Establishes upper bound on suboptimality for general MDPs, matching lower bound.
PASTA optimizes assortment selection using pessimism principle.
problem Optimizing assortment selection with limited data coverage.
method Pessimistic Assortment Optimization (PASTA) based on the principle of pessimism.
result PASTA correctly identifies optimal assortment with minimal data coverage.
Pessimistic Q-learning improves sample efficiency in offline reinforcement learning.
problem Insufficient coverage and sample scarcity in offline reinforcement learning datasets.
method Pessimistic Q-learning algorithm for offline reinforcement learning, focusing on variance reduction.
result Near-optimal sample complexity achieved with the proposed algorithm.
New model explains price dynamics of Bitcoin with psychological factors.
problem Understanding price variations in cryptocurrency markets with psychological factors.
method Extended agent-based model with heterogeneous psychological parameters.
result Model shows diverse dynamics based on psychological correlation.
Bayesian approach to portfolio selection reduces pessimism in frequent trading.
problem Tackling the challenge of estimating drift in Merton's portfolio selection model.
method Bayesian distributionally robust control with nonlinear Wasserstein projections.
result Reduced pessimism and improved performance in frequent rebalancing compared to existing methods.
Study examines how pandemic anxiety affects financial market trust.
problem Anxiety during pandemic and trust in financial markets.
method Used Google search volume and stock market data to create mood indicators.
result Different clusters of countries and markets in terms of pessimism and optimism emerged.
New assumptions and algorithm solve offline two-player zero-sum Markov games.
problem Solving offline two-player zero-sum Markov games under insufficient assumptions.
method Proposed unilateral concentration assumption and pessimism-type algorithm.
result Algorithm efficiently learns Nash equilibrium under unilateral concentration.
New method reduces over-pessimism in Bayesian control under parameter uncertainty.
problem Over-pessimism in Bayesian control due to misspecified priors.
method Distributionally robust Bayesian control (DRBC) with strong duality and optimization.
result Validated algorithm on synthetic and real data, reducing over-pessimism.
Latent variable models improve RL by facilitating efficient learning and exploration.
problem Improving sample efficiency in reinforcement learning.
method Representation view of latent variable models for state-action value functions, incorporating kernel embeddings and UCB exploration.
result Established sample complexity of the proposed approach in online and offline settings, demonstrated superior performance in benchmarks.
Distributionally Robust Supervised Learning (DRSL) is necessary for building reliable machine learning systems. When machine learning is deployed in the real world, its performance can be significantly degraded because test data may follow a different distribution from training data. DRSL with f-divergences explicitly …
LoCo-RLHF models diverse human feedback with contextual information.
problem Heterogeneous human feedback from diverse contexts and preferences.
method Low-rank contextual preference model, PRS policy.
result LoCo-RLHF achieves tighter sub-optimality gap than existing methods.
New method tackles MDPs by learning normalized representations efficiently.
problem Curse of dimensionality in MDPs.
method Contrastive representation learning for linear MDPs.
result First practical method with strong theoretical guarantees and empirical performance.
RH-UCRL combines pessimism and optimism for robust RL.
problem Ensuring reliable performance in real-world RL tasks with worst-case scenarios.
method RH-UCRL is a model-based RL algorithm that optimizes between an agent and an adversary, distinguishing between epistemic and aleatoric uncertainty.
result RH-UCRL achieves near-optimal sample complexity guarantees and outperforms other robust RL algorithms in adversarial environments.
This work improves policy evaluation and selection using logarithmic smoothing for pessimistic off-policy estimation.
problem Offline evaluation and selection of policies from past data.
method Develops novel concentration bounds and a logarithmically smoothed estimator (LS) for improved policy selection and learning.
result The logarithmically smoothed estimator (LS) provides tighter bounds and better policy selection and learning.
Paper tackles efficient IRL in offline settings with polynomial samples and runtime.
problem Efficiently learning reward functions from expert demonstrations in offline settings.
method Adapting the pessimism principle for offline RL, achieving strong guarantees.
result Achieved efficient IRL in offline and online settings with polynomial samples and runtime.
New algorithm for offline RL with linear approx in MDPs and MGs, nearly optimal.
problem Offline RL with linear function approximation in MDPs and MGs.
method Pessimism-based algorithm with uncertainty decomposition via reference function.
result Nearly minimax optimal performance in offline RL for MDPs and MGs.
Algorithm learns optimal dynamic mechanisms from data.
problem Designing optimal mechanisms for dynamic settings with unknown reward functions.
method Offline reinforcement learning with pessimism principle.
result Learned mechanisms are efficient, individually rational, and truthful.
Unified approach to RLHF tackles uncertainty in reward function.
problem Uncertainty in reward function learned from human feedback.
method Value-incentivized preference optimization (VPO) that regularizes the reward function with value function.
result Theoretical and practical guarantees for both online and offline RLHF settings.
Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.
problem Finding an approximate Nash equilibrium in offline Markov games with non-uniform coverage.
method Pessimistic Minimax Value Iteration (PMVI) constructs pessimistic value function estimates and solves NEs.
result Established a nearly minimax optimal result for offline Markov games with function approximation.
New method boosts skill learning by encouraging optimistic exploration.
problem Intrinsic reward for exploration is inherently pessimistic.
method Derive an information gain auxiliary objective involving an ensemble of discriminators and rewarding policy disagreement.
result Improves skill learning in grid worlds and Atari games.
This paper develops a model of reference-dependent assessment of subjective beliefs in which loss-averse people optimally choose the expectation as the reference point to balance the current felicity from the optimistic anticipation and the future disappointment from the realisation. The choice of over-optimism or over…
New algorithm CROP achieves asymptotic optimality with bounded regret.
problem Optimistic algorithms fail to achieve asymptotic instance-dependent regret optimality.
method CRush Optimism with Pessimism (CROP) algorithm that eliminates optimistic hypotheses.
result CROP achieves constant-factor asymptotic optimality and bounded regret.
Proposes DRRO to mitigate over-optimization in RLHF from human feedback.
problem Over-optimization due to reward misspecification in RLHF.
method Wasserstein distributionally robust regret optimization (DRRO).
result DRRO mitigates over-optimization more effectively than existing baselines.
In the presence of model risk, it is well-established to replace classical expected values by worst-case expectations over all models within a fixed radius from a given reference model. This is the "robustness" approach. We show that previous methods for measuring this radius, e.g. relative entropy or polynomial diverg…
New algorithm improves inference-time alignment without reward hacking.
problem Improving quality of responses from language models with limited compute.
method Inference-time alignment, focusing on extttInferenceTimePessimism algorithm. result Optimal performance and scaling-monotonicity of extttInferenceTimePessimism. Paper tackles RLHF with DCPPO method, proving near-optimal suboptimality.
problem Challenges in offline RLHF with limited human feedback and bounded rationality.
method DCPPO method involving three stages: MLE, reward function recovery, and pessimistic value iteration.
result DCPPO's suboptimality almost matches classical pessimistic offline RL in terms of distribution shift and dimension.
Study human-machine interaction with private info using offline RL.
problem Confounding bias and distributional mismatch in offline RL for human-guided interaction.
method Developed a novel identification result and OPE method to address confounding bias, and used pessimism to tackle distributional mismatch.
result Policy pair converges to optimal one at satisfactory rate under mild assumptions.
A Klein surface is a surface with a dianalytic structure. A double of a Klein surface X is a Klein surface Y such that there is a degree two morphism (of Klein surfaces) Y→X. There are many doubles of a given Klein surface and among them the so-called natural doubles which are: the complex double, the …
We define a general notion of abstract double Lie algebroid. We show (1) that the double Lie algebroid of a double Lie groupoid is a double Lie algebroid in this sense; (2) that the double cotangent constructed from Lie algebroid structures on a vector bundle A and its dual A* is a double Lie algebroid if and only if (…
The word `double' was used by Ehresmann to mean `an object X in the category of all X'. Double categories, double groupoids and double vector bundles are instances, but the notion of Lie algebroid cannot readily be doubled in the Ehresmann sense, since a Lie algebroid bracket cannot be defined diagrammatically. In this…
We define double principal bundles (DPBs), for which the frame bundle of a double vector bundle, double Lie groups and double homogeneous spaces are basic examples. It is shown that a double vector bundle can be realized as the associated bundle of its frame bundle. Also dual structures, gauge transformations and conne…
Generalizes Hecke algebra for double torus, linking to skein algebra.
problem Understanding algebraic structures on double torus.
method Introducing Heegaard dual operators and Dehn twists.
result Established relationship between Hecke algebra and skein algebra.
Introduces Poisson double algebroids and their relation to Lie 2-bialgebras.
problem Developing Lie theory for Poisson double structures.
method Introducing Poisson double algebroids and double Lie bialgebroids, and relating them through differentiation and integration.
result Revisits Lie 2-bialgebras using Poisson double structures.
Survey of global geometry for double field theory.
problem Global description of double field theory geometry.
method Review of Courant algebroids, metric algebroids, AKSZ construction, para-Hermitian geometry.
result Global description of doubled geometry and topological models.
This paper establishes an equivalence between transitive double Lie algebroids and core diagrams.
problem Understanding and characterizing transitive double Lie algebroids.
method Using core diagrams and equivalence of transitive core diagrams with transitive double Lie groupoids.
result Transitive double Lie algebroids are completely determined by their core diagrams.
We define an abstract notion of double Lie algebroid, which includes as particular cases: (1) the double Lie algebroid of a double Lie groupoid in the sense of the author, such as the iterated tangent bundle of an ordinary manifold, and various iterated tangent/cotangent constructions in symplectic and Poisson geometry…
We develop new algebraic methods refining the Witt group of linking forms and Ranicki's torsion algebraic L-groups into double Witt groups and double L-groups. At each prime ideal of the underlying ring, our double Witt groups capture infinitely many more integral signatures of the linking form than the single Witt gro…
A theory of double affine and special double affine bundles, i.e. differential manifolds with two compatible (special) affine bundle structures, is developed as an affine counterpart of the theory of double vector bundles. The motivation and basic examples come from Analytical Mechanics, where double affine bundles hav…
Existence of double bubbles with high constant mean curvatures in Riemannian manifolds.
problem Existence of double bubbles with high constant mean curvatures in Riemannian manifolds.
method Perturbations of geodesic standard double bubbles centered at critical points of the ambient scalar curvature and aligned along eigen-vectors of the ambient Ricci tensor, with general multiplicity results via Lusternik-Schnirelman theory.
result Existence of double bubbles with high constant mean curvatures in Riemannian manifolds.
We show that if K is any knot whose Ozsvath-Szabo concordance invariant tau(K) is positive, the all-positive Whitehead double of any iterated Bing double of K is topologically but not smoothly slice. We also show that the all-positive Whitehead double of any iterated Bing double of the Hopf link (e.g., the all-positive…
For compact real manifolds, a new double conformal invariant is constructed using the Wodzicki residue and the d operator in the framework of Connes. In the flat case, we compute this double conformal invariant, and in some special cases, we also compute this double conformal invariants. For complex manifolds, a new …