New method designs experiments robustly for nonlinear estimation, improving parameter knowledge.
problem Designing robust experiments for nonlinear estimation under parametric uncertainty.
method Multi-stage robust optimization framework for sequential experiments.
result Identifies experiments better conducted early for improved parameter knowledge.
Bayesian optimization for long-term outcomes using fast and slow experiments.
problem Optimizing long-term system effects with short-term misleading results.
method Combining fast and slow experiments for Bayesian optimization.
result Sequential optimization over large action spaces in a short time.
Paper uses transfer learning and Bayesian optimization to reduce DNA sequence design experiments.
problem Designing many similar DNA sequences for specific applications is expensive and time-consuming.
method Combines transfer learning with Bayesian optimization to reduce experiment count.
result Total number of experiments can be significantly reduced by sharing information between tasks.
Optimal tests developed for sequential experiments with asymptotic properties.
problem Performing hypothesis tests after sequential experiments without prior design.
method Analyze asymptotic properties of sequential experiments; develop tests for Gaussian process observations.
result Asymptotic power function of any test can be matched by a specific test in a limit experiment.
Combines multi-fidelity and asynchronous batch methods for faster experimental design.
problem Designing optimal experimental setups for battery performance.
method Algorithm combining multi-fidelity and asynchronous batch Bayesian Optimization.
result Algorithm outperforms single-fidelity batch and multi-fidelity sequential methods.
Unified model optimizes experiment performance and reduces duration.
problem Balancing reward maximization and experiment termination.
method Unified model that considers both within-experiment and post-experiment outcomes.
result Familiar algorithms can optimize a broad class of objectives with proper parameter adjustment.
New framework for choosing optimal proxy metrics from past experiments.
problem Difficult to measure long-term treatment effects in experiments.
method Statistical framework to define and construct optimal proxy metrics.
result Optimal proxy metric depends on experiment's sample size.
Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences uniformly or utilize a rule-based replay strategy, which may be sub-optimal. In this work, we consid…
Olympus benchmarks optimization algorithms for noisy experiments.
problem Benchmarking optimization algorithms on realistic experimental scenarios is challenging.
method Introduces Olympus, a software package for benchmarking optimization algorithms on synthetic experiments.
result Mitigates barriers in benchmarking optimization algorithms on realistic experimental scenarios.
Randomized experiments are the gold standard for evaluating the effects of changes to real-world systems. Data in these tests may be difficult to collect and outcomes may have high variance, resulting in potentially large measurement error. Bayesian optimization is a promising technique for efficiently optimizing multi…
VRER selectively reuses past observations to reduce variance in policy optimization.
problem Lack of effective experience replay for accelerating policy optimization in complex systems.
method Variance Reduction Experience Replay (VRER) framework that selectively reuses informative samples.
result VRER reduces gradient variance and improves policy learning over state-of-the-art algorithms.
Bayesian model ranks treatments in multi-response experiments.
problem Identifying the best treatment among competing ideal properties.
method Bayesian approach with Markov Chain Monte Carlo algorithm.
result Reliable inference of treatment ranks in practice.
Optimal adaptive experiment for choosing best treatment with binary outcomes.
problem Choosing the best treatment from binary options in an adaptive experiment.
method Adaptive experiment with two phases: treatment allocation and choice. Neyman allocation method used.
result Neyman allocation is minimax and Bayes optimal, matching lower bounds for regret.
A framework for efficient multi-objective optimization using entropy search.
problem Optimizing expensive black-box functions with multiple objectives.
method Output space entropy search (OSE) to minimize resource cost.
result Improves efficiency and accuracy in multi-objective optimization.
Motivated by the widespread adoption of large-scale A/B testing in industry, we propose a new experimentation framework for the setting where potential experiments are abundant (i.e., many hypotheses are available to test), and observations are costly; we refer to this as the experiment-rich regime. Such scenarios requ…
We study the problem of causal structure learning over a set of random variables when the experimenter is allowed to perform at most M experiments in a non-adaptive manner. We consider the optimal learning strategy in terms of minimizing the portions of the structure that remains unknown given the limited number of e…
NUBO simplifies Bayesian optimization for researchers.
problem Optimizing expensive functions like experiments and simulations.
method Bayesian optimization using Gaussian processes and acquisition functions.
result Transparency and user-friendly design for easy access.
Bayesian Optimization (BO) is a data-efficient method for global black-box optimization of an expensive-to-evaluate fitness function. BO typically assumes that computation cost of BO is cheap, but experiments are time consuming or costly. In practice, this allows us to optimize ten or fewer critical parameters in up to…
This study optimizes covariate density and propensity score for efficient ATE estimation.
problem Efficiently estimating average treatment effects (ATEs) with minimal variance.
method Adaptive experiment optimizing both covariate density and propensity score.
result Proposed method minimizes the semiparametric efficiency bound for ATE estimation.
BoFire optimizes chemistry experiments using Bayesian Optimization.
problem Effective deployment of Bayesian Optimization in the chemical industry.
method Combines Bayesian Optimization with DoE strategies, providing a rich feature-set.
result BoFire enables seamless integration into RESTful APIs for real-world use.
BoTier optimizes experiments by balancing multiple objectives hierarchically.
problem Balancing multiple competing objectives in scientific experiments.
method Composite objective that flexibly represents a hierarchy of preferences over outcomes and parameters.
result Demonstrates robust applicability across various use cases and seamless integration.
Optimizes expensive experiments by incorporating expert knowledge.
problem Expensive experiments require minimizing the number of trials.
method Bayesian optimization with posterior sampling of expert knowledge.
result Demonstrates significant efficiency gains in experiments and hyperparameter tuning.
VRER selectively reuses samples to improve policy optimization in complex systems.
problem Lack of effective reuse of historical samples in reinforcement learning.
method Variance reduction based experience replay (VRER) framework.
result VRER accelerates policy optimization and enhances performance.
Neural Optimal Design of Experiments improves inverse problem solving efficiency.
problem Optimal experimental design in inverse problems.
method Jointly trains a reconstruction model and design variables in a single loop.
result Significantly reduces computational complexity and improves reconstruction accuracy.
Paper develops an efficient algorithm for experiment design using synthetic controls.
problem Designing optimal experiments for treatment effect estimation.
method Solves a phase synchronization problem via a normalized generalized power method.
result First global optimality guarantee for experiment design with pre-treatment data.
New BO methods exploit parallel experiments, reducing search time and improving solution quality.
problem Limitation of Bayesian optimization in exploiting parallel experiments.
method Propose new parallel BO paradigms that exploit the structure of the system to partition the design space.
result Significantly reduce search time and increase probability of finding global solutions.
Bayesian optimization learns DM preferences for multi-outcome experiments.
problem Optimizing expensive experiments with unknown utility functions and multiple outcomes.
method Alternates preference learning and Bayesian optimization, using pairwise comparisons.
result Preference exploration strategies improve Bayesian optimization performance.
Equal experience fairness in recommender systems reduces bias.
problem Bias in data leads to unfair item recommendations.
method Introduces equal experience fairness notion and optimization framework.
result Mitigates unfairness in recommendations with minimal accuracy loss.
New job recommendation system improves job seekers' welfare through field experiments.
problem Current job recommendation systems focus on clicks and applications, not job seekers' welfare.
method Developed a job-search model with two dimensions: utility and success probability. Conducted field experiments to validate model predictions.
result Welfare-optimal job recommendation algorithms outperform existing approaches and perform close to the benchmark.
Paper optimizes experimental design for estimating treatment effect.
problem Estimating treatment effect with heterogeneous subjects and treatments.
method Adaptive experimental design incorporating bandit learning.
result Demonstrates optimality of proposed adaptive experiment framework.
CO-BED optimizes experiments using Bayesian methods and information theory.
problem Optimizing experiments in a context-dependent manner.
method Formalizes contextual optimization with Bayesian experimental design, employing information-theoretic principles and black-box variational methods.
result CO-BED provides a general solution for contextual optimization problems.
Paper proposes a new method for efficient hyperparameter optimization.
problem Challenging task of optimizing hyperparameters in machine learning.
method Sequential Uniform Design (SeqUD) strategy for adaptive and efficient exploration of hyperparameter space.
result The proposed SeqUD strategy outperforms existing methods in hyperparameter optimization.
New methods optimize experiment selection for sequential data, improving model accuracy.
problem Optimizing experiment selection for sequential data in multidimensional cases.
method Adopting greedy experiment selection methods for maximum likelihood estimation.
result Proposed methods produce consistent and asymptotically normal estimators.
Proposes a model to optimize feedback for content creators on social media.
problem Nurturing content creation on social media platforms.
method Modeling feedback distribution to incentivize creators and optimize newsfeed experience.
result Optimized newsfeed for content creators without compromising user experience.
Transformer RL optimizes A/B testing for time series experiments.
problem Challenges in applying A/B testing to time series experiments, especially with limited history and strong assumptions.
method Transformer reinforcement learning approach that conditions allocation on full history and optimizes MSE without restrictive assumptions.
result Consistently outperforms existing designs in synthetic, simulator, and real-world data.
LM optimization outperforms other methods in deep learning tasks but at high computational cost.
problem Finding efficient optimization methods for deep learning models.
method Comparing first-order (CG, SGD, LM, L-BFGS) and higher-order optimization functions.
result Levemberg-Marquardt (LM) optimization significantly improves convergence but at a high computational cost.
Algorithm selects optimal experiments in Markov chains to learn unknown quantities.
problem Designing efficient experiments in Markov chains to learn about unknown quantities.
method Proposes extsc{markov-design} algorithm for sequential policy selection.
result Algorithm provably converges to optimal measurement allocation.
Pessimistic estimator improves multi-objective policy optimization.
problem Optimizing multi-objective policies from existing data.
method Pessimistic estimator based on inverse propensity scores (IPS).
result Pessimistic estimator outperforms naive IPS estimator in theory and experiments.
Online field experiments are the gold-standard way of evaluating changes to real-world interactive machine learning systems. Yet our ability to explore complex, multi-dimensional policy spaces - such as those found in recommendation and ranking problems - is often constrained by the limited number of experiments that c…
Estimating arbitrary quantities of interest (QoIs) that are non-linear operators of complex, expensive-to-evaluate, black-box functions is a challenging problem due to missing domain knowledge and finite budgets. Bayesian optimal design of experiments (BODE) is a family of methods that identify an optimal design of exp…
This work uses RL to optimize batch experiments in SDOE.
problem Maximizing knowledge with limited resources and constraints.
method Reinforcement Learning for batch-sampling in Bayesian SDOE.
result The proposed algorithm optimizes experiment selection for multiple tasks.
In this paper, we study the design and analysis of experiments conducted on a set of units over multiple time periods where the starting time of the treatment may vary by unit. The design problem involves selecting an initial treatment time for each unit in order to most precisely estimate both the instantaneous and cu…
A new method designs batches for Bayesian optimization more efficiently.
problem Efficiently designing batches for Bayesian optimization to reduce total time.
method Minimal Terminal Variance (MTV) acquisition function, optimizing I-optimality criterion.
result MTV designs batches more efficiently than other methods, as shown by numerical experiments.
Bayesian Optimization improves data mixture selection for large language models.
problem Optimizing the training data mixtures for large language models.
method Viewed as a black-box hyperparameter optimization problem, using Bayesian Optimization.
result Consistently strong results with speed-ups of over 500%.
The paper proposes an efficient nested simulation design using likelihood ratio method.
problem Designing nested simulations with fixed outer scenarios and minimizing simulation effort.
method Proposes a bi-level optimization problem to decide inner replications and pooling strategies.
result Optimized design achieves $\cO(Γ^{-1})$ mean squared error of estimators.
Optimizes recommender selection online with D-optimal design.
problem Finding the optimal recommender in online exploration-exploitation.
method Leverages D-optimal design from statistics to maximize information gain.
result Achieves maximum information gain during online exploration.
This paper introduces a Bayesian framework for optimizing online experiments to maximize profit.
problem Statistical flaws and reliance on proxy metrics in A/B tests compromise their effectiveness.
method Hierarchical Bayesian model for estimating conversion probability and monetary value, decision-theoretic stopping rule.
result The framework ensures experiments conclude when no variant offers a significant profit improvement, conserving resources.
Optimal experiments tighten causal effect bounds efficiently.
problem Selecting experiments to tighten causal effect bounds from observational data.
method Formalized as max-potency problem, NP-hard. Polynomial-programming framework with graphical pruning criteria.
result Pruning criteria reduce search space significantly, enabling efficient experiment selection.