Bayesian optimization learns DM preferences for multi-outcome experiments.
problem Optimizing expensive experiments with unknown utility functions and multiple outcomes.
method Alternates preference learning and Bayesian optimization, using pairwise comparisons.
result Preference exploration strategies improve Bayesian optimization performance.
Optimal Kelly strategy for multi-outcome parlay bets proven using implicit cash approach.
problem Finding optimal Kelly stakes for multi-outcome parlay bets.
method Eventwise Kelly strategy followed by outer product for full menu of bets. Uses implicit cash viewpoint.
result Optimal Kelly stakes for parlay bets factorize across events, with active leg criterion.
Recent advances in contextual bandit optimization and reinforcement learning have garnered interest in applying these methods to real-world sequential decision making problems. Real-world applications frequently have constraints with respect to a currently deployed policy. Many of the existing constraint-aware algorith…
MO-CBO optimizes multiple outcomes in causal systems with minimal data.
problem Optimizing multiple outcomes in causal systems with limited data.
method Decomposes MO-CBO into multi-objective optimization tasks and uses relative hypervolume improvement for sequential intervention balancing.
result MO-CBO outperforms traditional multi-objective Bayesian optimization in causal settings.
Support selection and eventwise decoupling for simultaneous bets proven.
problem Optimizing expected utility for simultaneous independent events with multiple outcomes.
method Proved a support theorem for a broad class of strictly increasing strictly concave utilities, identifying the exact active support and proving independence from utility function.
result The exact active support is the eventwise union of single-event supports, independent of the utility function.
A new method prioritizes and recycles experiences for better reinforcement learning.
problem Improving reinforcement learning efficiency by prioritizing and recycling experiences.
method Double-prioritized state-recycled (DPSR) experience replay.
result DPSR achieved state-of-the-art results in Atari games, outperforming original and prioritized methods.
This project combines recent advances in experience replay techniques, namely, Combined Experience Replay (CER), Prioritized Experience Replay (PER), and Hindsight Experience Replay (HER). We show the results of combinations of these techniques with DDPG and DQN methods. CER always adds the most recent experience to th…
ReaPER improves learning efficiency by prioritizing reliable experiences.
problem Inefficient sampling of past experiences in reinforcement learning.
method Introducing a novel measure of reliability to prioritize experiences in PER.
result ReaPER outperforms PER in various environments, including Atari-10.
This paper describes an improvement in Deep Q-learning called Reverse Experience Replay (also RER) that solves the problem of sparse rewards and helps to deal with reward maximizing tasks by sampling transitions successively in reverse order. On tasks with enough experience for training and enough Experience Replay mem…
Unified model optimizes experiment performance and reduces duration.
problem Balancing reward maximization and experiment termination.
method Unified model that considers both within-experiment and post-experiment outcomes.
result Familiar algorithms can optimize a broad class of objectives with proper parameter adjustment.
An important component of many Deep Reinforcement Learning algorithms is the Experience Replay which serves as a storage mechanism or memory of made experiences. These experiences are used for training and help the agent to stably find the perfect trajectory through the problem space. The classic Experience Replay howe…
New method designs experiments robustly for nonlinear estimation, improving parameter knowledge.
problem Designing robust experiments for nonlinear estimation under parametric uncertainty.
method Multi-stage robust optimization framework for sequential experiments.
result Identifies experiments better conducted early for improved parameter knowledge.
Experience reuse is key to sample-efficient reinforcement learning. One of the critical issues is how the experience is represented and stored. Previously, the experience can be stored in the forms of features, individual models, and the average model, each lying at a different granularity. However, new tasks may requi…
Optimal tests developed for sequential experiments with asymptotic properties.
problem Performing hypothesis tests after sequential experiments without prior design.
method Analyze asymptotic properties of sequential experiments; develop tests for Gaussian process observations.
result Asymptotic power function of any test can be matched by a specific test in a limit experiment.
Bayesian optimization for long-term outcomes using fast and slow experiments.
problem Optimizing long-term system effects with short-term misleading results.
method Combining fast and slow experiments for Bayesian optimization.
result Sequential optimization over large action spaces in a short time.
Two methods estimate effect size for online experiments, improving accuracy and efficiency.
problem Determining the correct effect size for online experiment duration.
method Two approaches: hierarchical models and utility theory.
result Proposed methods outperform baseline approaches in accuracy and efficiency.
Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences uniformly or utilize a rule-based replay strategy, which may be sub-optimal. In this work, we consid…
Study finds real-world datasets contain natural experiments that can improve model performance.
problem Detecting natural experiments in real-world datasets for causal inference.
method Synthetic graph simulation and feature selection based on causal links.
result Real-world datasets contain natural experiments that can be exploited for improved model performance.
We reduce variance in monetization metrics for ranking experiments.
problem Heavy-tailed monetization metrics lead to unreliable conclusions in A/B experiments.
method Post-stratification combined with CUPED.
result Significant reduction in variance and improved decision stability.
New framework for choosing optimal proxy metrics from past experiments.
problem Difficult to measure long-term treatment effects in experiments.
method Statistical framework to define and construct optimal proxy metrics.
result Optimal proxy metric depends on experiment's sample size.
Experience replay is widely used in deep reinforcement learning algorithms and allows agents to remember and learn from experiences from the past. In an effort to learn more efficiently, researchers proposed prioritized experience replay (PER) which samples important transitions more frequently. In this paper, we propo…
LiDER refreshes past experiences in RL by dreaming about them.
problem Improving data efficiency in off-policy RL algorithms.
method Refreshing past experiences in a replay buffer using the current policy.
result LiDER consistently improves performance in Atari games.
We analyze the sample complexity of learning from multiple experiments where the experimenter has a total budget for obtaining samples. In this problem, the learner should choose a hypothesis that performs well with respect to multiple experiments, and their related data distributions. Each collected sample is associat…
The paper improves reinforcement learning stability and efficiency with a new theoretical framework.
problem Stability and efficiency in reinforcement learning, especially in data-scarce scenarios.
method Theoretical framework using resampled U- and V-statistics to model experience replay, applied to policy evaluation and kernel ridge regression. result Significant improvements in stability and efficiency, particularly in data-scarce scenarios.
Optimal adaptive experiment for choosing best treatment with binary outcomes.
problem Choosing the best treatment from binary options in an adaptive experiment.
method Adaptive experiment with two phases: treatment allocation and choice. Neyman allocation method used.
result Neyman allocation is minimax and Bayes optimal, matching lower bounds for regret.
We study the task of retrieving relevant experiments given a query experiment. By experiment, we mean a collection of measurements from a set of `covariates' and the associated `outcomes'. While similar experiments can be retrieved by comparing available `annotations', this approach ignores the valuable information ava…
Expands Bayesian experiment design framework to account for model discrepancies.
problem Model misspecification in Bayesian optimal experiment design.
method Introduces Expected General Information Gain and Expected Discriminatory Information criteria.
result Demonstrates improved robustness and detection capabilities in experiment design.
ER-GNN uses experience replay to prevent GNNs from forgetting previous tasks.
problem Catastrophic forgetting in GNNs when learning multiple tasks sequentially.
method Experience Replay framework to store and replay knowledge from previous tasks.
result ER-GNN effectively mitigates catastrophic forgetting in GNNs.
The paper addresses statistical inference issues in adaptive experiments.
problem Statistical inference problems in adaptive experiments.
method Explains and fixes statistical inference issues in adaptive experiments using various methods.
result Various methods to stabilize inferences and recover asymptotic normality.
Online review communities are dynamic as users join and leave, adopt new vocabulary, and adapt to evolving trends. Recent work has shown that recommender systems benefit from explicit consideration of user experience. However, prior work assumes a fixed number of discrete experience levels, whereas in reality users gai…
New insights into experience replay in RL algorithms.
problem Understanding the impact of replay capacity and replay ratio in Q-learning.
method Systematic and extensive analysis of experience replay in Q-learning methods, focusing on replay capacity and replay ratio.
result Greater replay capacity significantly improves performance for certain algorithms, while other techniques offer limited benefit.
Policy gradient reinforcement learning (RL) algorithms have achieved impressive performance in challenging learning tasks such as continuous control, but suffer from high sample complexity. Experience replay is a commonly used approach to improve sample efficiency, but gradient estimators using past trajectories typica…
We address the problem of retrieving relevant experiments given a query experiment, motivated by the public databases of datasets in molecular biology and other experimental sciences, and the need of scientists to relate to earlier work on the level of actual measurement data. Since experiments are inherently noisy and…
Paper uses transfer learning and Bayesian optimization to reduce DNA sequence design experiments.
problem Designing many similar DNA sequences for specific applications is expensive and time-consuming.
method Combines transfer learning with Bayesian optimization to reduce experiment count.
result Total number of experiments can be significantly reduced by sharing information between tasks.
The paper clusters hypergraphs to find diverse and experienced groups based on past experiences.
problem Finding diverse and experienced groups with respect to past experiences.
method Regularized edge-based hypergraph clustering objective with a 2-approximation algorithm.
result Demonstrates an efficient 2-approximation algorithm for clustering hypergraphs.
The scientific method relies on the iterated processes of inference and inquiry. The inference phase consists of selecting the most probable models based on the available data; whereas the inquiry phase consists of using what is known about the models to select the most relevant experiment. Optimizing inquiry involves …
The logic of uncertainty is not the logic of experience and as well as it is not the logic of chance. It is the logic of experience and chance. Experience and chance are two inseparable poles. These are two dual reflections of one essence, which is called co~event. The theory of experience and chance is the theory of c…
iDAD uses neural networks to quickly adapt experiments without likelihoods.
problem Performing adaptive experiments in real-time with implicit models.
method iDAD learns a design policy network upfront to make quick design decisions.
result iDAD can make design decisions in milliseconds, unlike traditional BOED approaches.
CEA augments reinforcement learning by generating counterfactual experiences.
problem Challenges in reinforcement learning, especially out-of-distribution and inefficient exploration.
method CEA uses variational autoencoders to model state transitions and introduces randomness for non-stationarity. It expands learning data through counterfactual inference.
result CEA outperforms SOTA algorithms in diverse environments.
IMeL turns RL into SL by interpolating improved experiences.
problem Improving reinforcement learning efficiency and scalability.
method IMeL uses a reservoir of experiences and a NN regressor for interpolation.
result IMeL achieves preliminary results and proposes itself as a baseline.
A framework for efficient multi-objective optimization using entropy search.
problem Optimizing expensive black-box functions with multiple objectives.
method Output space entropy search (OSE) to minimize resource cost.
result Improves efficiency and accuracy in multi-objective optimization.
New BO methods exploit parallel experiments, reducing search time and improving solution quality.
problem Limitation of Bayesian optimization in exploiting parallel experiments.
method Propose new parallel BO paradigms that exploit the structure of the system to partition the design space.
result Significantly reduce search time and increase probability of finding global solutions.
Random forests classify Pokemon names based on evolutionary status.
problem Classifying Pokemon names into pre-evolution and post-evolution categories.
method Trained random forests using sound symbolism from Pokemon names, conducted experiments with human participants.
result Random forests outperform human participants in classifying Pokemon names.
We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of off-policy learning where agents learn from other agents behaviour. We employ …
VRER selectively reuses past observations to reduce variance in policy optimization.
problem Lack of effective experience replay for accelerating policy optimization in complex systems.
method Variance Reduction Experience Replay (VRER) framework that selectively reuses informative samples.
result VRER reduces gradient variance and improves policy learning over state-of-the-art algorithms.
Paper proposes a method to use in silico experiments with foundation models to reduce sample size.
problem Costly and uncertain randomized experiments.
method Integrates predictions from multiple foundation models with experimental data.
result Estimator offers substantial precision gains, equivalent to a 20% reduction in sample size.
A-ICP selects experiments to learn causal effects efficiently.
problem Learning causal effects from observational data is difficult.
method Active learning framework based on Invariant Causal Prediction.
result Proposes intervention selection policies to reveal direct causes.
Inertial confinement fusion (ICF) experiments are designed using computer simulations that are approximations of reality, and therefore must be calibrated to accurately predict experimental observations. In this work, we propose a novel nonlinear technique for calibrating from simulations to experiments, or from low fi…