The paper improves A/B testing for non-Gaussian data, ensuring reliable results with large sample sizes.
problem Inaccurate A/B testing results due to non-normal data and unequal sample sizes.
method Derives explicit formulas for minimum sample size and introduces an Edgeworth-based correction.
result Corrected method improves reliability of A/B testing in real-world conditions.
A family of maximum mean discrepancy (MMD) kernel two-sample tests is introduced. Members of the test family are called Block-tests or B-tests, since the test statistic is an average over MMDs computed on subsets of the samples. The choice of block size allows control over the tradeoff between test power and computatio…
Improved A/B testing by leveraging system similarities.
problem Traditional A/B testing ignores potential system similarities.
method Off-policy estimation to exploit system propensities.
result Improved A/B testing estimators achieve better accuracy.
Machine learning method characterizes network interference in A/B tests.
problem Compromised A/B test reliability due to network interference.
method Causal network motifs and machine learning models.
result Outperforms conventional methods in characterizing network interference.
Unified framework for robust A/B testing under model misspecification.
problem Improving sample efficiency in A/B testing with model misspecification.
method Unified framework for contextual bandit and dynamic settings, proving worst-case mean squared error bounds.
result Empirically validated approach using synthetic and real-world datasets.
Improves A/B testing power using a two-armed bandit framework.
problem Comparing outcomes under a new policy to a control.
method Doubly robust estimation, two-armed bandit framework, permutation-based method.
result Superior performance in A/B testing compared to existing methods.
This paper analyzes switchback experiments in A/B testing, revealing key factors affecting their effectiveness.
problem Understanding the effectiveness of different switchback designs in A/B testing.
method Comprehensive comparative analysis of various switchback designs in Markovian environments, covering state-of-the-art RL estimators.
result The effectiveness of switchback designs depends on the size of the carryover effect and reward autocorrelations.
The paper optimizes A/B tests by balancing lift and cost in large-scale settings.
problem Balancing lift and cost in A/B tests for large-scale experimentation.
method Empirical Bayes approach using a greedy knapsack algorithm to rank experiments based on lift-to-cost ratio, incorporating local false discovery rate (lfdr).
result The proposed method maximizes expected profit while controlling false discovery rate, demonstrating superior performance in large-scale settings.
Boosts A/B test precision using auxiliary data from historical users.
problem Small sample sizes and imprecise estimates in A/B tests.
method Coupling design-based causal estimation with machine-learning models of historical user data.
result Effect estimates using auxiliary data are roughly equivalent to increasing sample size by 20%, or up to 50-80% in some cases.
Improves A/B testing by detecting minor treatment effects.
problem Challenges in identifying small average treatment effects.
method Maximum probability-driven two-armed bandit (TAB) process with weighted mean volatility statistic.
result Significant improvement in A/B testing with reduced experimental costs.
New framework minimizes interference and selection bias in network A/B testing.
problem Interference and selection bias in network A/B testing.
method Proposes a principled framework that jointly minimizes interference and selection bias using edge spillover probability and cluster matching.
result Significantly lower error in causal effect estimation compared to existing solutions.
Transformer RL optimizes A/B testing for time series experiments.
problem Challenges in applying A/B testing to time series experiments, especially with limited history and strong assumptions.
method Transformer reinforcement learning approach that conditions allocation on full history and optimizes MSE without restrictive assumptions.
result Consistently outperforms existing designs in synthetic, simulator, and real-world data.
Before A/B testing online a new version of a recommender system, it is usual to perform some offline evaluations on historical data. We focus on evaluation methods that compute an estimator of the potential uplift in revenue that could generate this new technology. It helps to iterate faster and to avoid losing money b…
A/B testing refers to the task of determining the best option among two alternatives that yield random outcomes. We provide distribution-dependent lower bounds for the performance of A/B testing that improve over the results currently available both in the fixed-confidence (or delta-PAC) and fixed-budget settings. When…
Improves A/B testing for long-term outcomes in dynamic systems.
problem Estimating long-term effects from short-term A/B testing data.
method Develops optimal inference techniques and localized information sharing methods.
result New estimator reduces variance linearly with test arms and matches lower bounds.
No algorithm outperforms uniform sampling in A/B testing.
problem Identifying the best arm in A/B testing with fixed budget.
method Introducing consistent and stable algorithms, deriving lower bounds, and proving optimality of uniform sampling.
result No algorithm performs better than uniform sampling in A/B testing.
New metrics boost A/B-test power by up to 210%.
problem High cost and type-II errors in A/B-tests.
method Learn metrics from short-term signals to maximize power.
result Statistical power increased by up to 210%.
A/B testing, or online experiment is a standard business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries. Major challenges arise in online experiments of two-sided marketplace platforms (e.g., Uber) where there is only one unit that receives a sequence of t…
A/B testing improves marketing decisions by selecting effective stratification variables.
problem Improving the sensitivity of A/B testing through stratified sampling.
method Designing an algorithm to select a subset of stratification variables for variance reduction.
result The subset selection method outperforms other variance reduction techniques in A/B testing.
Deep network optimizes ad bidding for first-price auctions.
problem Optimizing bid prices for first-price auctions in online advertising.
method Introduced a deep distribution network for optimal bidding.
result Algorithm outperforms previous methods in terms of surplus and eCPX metrics.
Optimizes non-linear outcomes from summed contributions.
problem Maximizing a non-linear function of summed small contributions.
method Derives a scalable descent algorithm leveraging concentration properties.
result Directly optimizes for stated objective, e.g., A/B test success criterion.
This paper studies the geometry of minimum-volume confidence sets for multinomial parameters.
problem Determining if minimum-volume confidence sets for multinomial outcomes are disjoint.
method Enumerating and covering the continuous regions of the exact p-value function to study the geometry of minimum-volume confidence sets.
result The geometry of minimum-volume confidence sets for multinomial parameters is studied, providing insights into their structure and properties.
Paper certifies intersection of minimum-volume confidence sets for multinomial outcomes.
problem Certifying intersection of minimum-volume confidence sets for multinomial outcomes.
method Exploits likelihood ordering to induce halfspace constraints, enabling adaptive geometric partitioning and computable bounds on p-values.
result Efficient and provably sound algorithm for certifying intersection, disjointness, or indeterminate result.
State of the art online learning procedures focus either on selecting the best alternative ("best arm identification") or on minimizing the cost (the "regret"). We merge these two objectives by providing the theoretical analysis of cost minimizing algorithms that are also delta-PAC (with a proven guaranteed bound on th…
The paper provides bounds on the CDF of a variable under nonstationary conditions.
problem Estimating the complete distribution of a random variable under nonstationary conditions.
method Time-uniform and value-uniform bounds on the CDF of the running averaged conditional distribution.
result Presented computationally efficient bounds that are always valid and sometimes trivial.
Randomized trials, also known as A/B tests, are used to select between two policies: a control and a treatment. Given a corresponding set of features, we can ideally learn an optimized policy P that maps the A/B test data features to action space and optimizes reward. However, although A/B testing provides an unbiased …
New method optimizes multiple objectives in A/B testing for AI and clinical trials.
problem Minimizing cumulative regret, maximizing CATE, and ensuring differential privacy in large-scale experiments.
method ConSE and DP-ConSE algorithms for sequential segmentation and elimination, achieving Pareto-optimal frontier.
result Privacy comes 'for free' in our framework, with only asymptotically negligible costs to regret and accuracy.
The rollout of new versions of a feature in modern applications is a manual multi-stage process, as the feature is released to ever larger groups of users, while its performance is carefully monitored. This kind of A/B testing is ubiquitous, but suboptimal, as the monitoring requires heavy human intervention, is not gu…
New framework estimates long-term outcomes from short-term data.
problem Estimating long-term outcomes from short-term data.
method Reward function decomposition-based framework (LOPE).
result LOPE outperforms existing methods, especially when surrogacy is violated.
Research suggests using deep learning for better recommendation systems.
problem Recommender systems rely on proxies for A/B testing, leading to random success.
method Advocates for using deep learning to improve recommendation performance.
result Deep learning can potentially optimize reward in recommendation systems.
This paper introduces a Bayesian framework for optimizing online experiments to maximize profit.
problem Statistical flaws and reliance on proxy metrics in A/B tests compromise their effectiveness.
method Hierarchical Bayesian model for estimating conversion probability and monetary value, decision-theoretic stopping rule.
result The framework ensures experiments conclude when no variant offers a significant profit improvement, conserving resources.
Bayesian optimization for long-term outcomes using fast and slow experiments.
problem Optimizing long-term system effects with short-term misleading results.
method Combining fast and slow experiments for Bayesian optimization.
result Sequential optimization over large action spaces in a short time.
This paper unifies two types of statistical methods for estimating treatment effects.
problem Isolating online A/B-tests and off-policy evaluation.
method Establishes formal equivalence between online Difference-in-Means and off-policy Inverse Propensity Scoring methods.
result Standard online methods are mathematically equivalent to off-policy methods with optimal control variates.
The ability to perform offline A/B-testing and off-policy learning using logged contextual bandit feedback is highly desirable in a broad range of applications, including recommender systems, search engines, ad placement, and personalized health care. Both offline A/B-testing and off-policy learning require a counterfa…
Synthetic control method improves policy evaluation in high-dimensional settings.
problem Evaluating the impact of new policies in large-scale applications.
method Two-phase approach: nearest neighbor matching followed by supervised learning.
result The method successfully improves estimate accuracy in large-scale experiments.
This paper proposes a new AED framework for multi-metric experiments with fixed budget.
problem Statistical power challenges in testing multiple metrics simultaneously.
method Two-phase structure: adaptive exploration followed by validation. SHRVar algorithm with relative-variance-based sampling.
result Achieves provable error probability that decreases exponentially.
Optimizing an interactive system against a predefined online metric is particularly challenging, when the metric is computed from user feedback such as clicks and payments. The key challenge is the counterfactual nature: in the case of Web search, any change to a component of the search engine may result in a different…
The paper examines A/B tests in recommendation systems to detect biased algorithm comparisons due to shared data.
problem Bias in comparing recommendation algorithms due to shared data.
method Formalized as a multi-armed bandit problem, analyzed the sign of difference-in-means estimator vs true GTE.
result Data sharing can lead to biased comparisons of recommendation algorithms, and a detection procedure is proposed.
We propose an alternative framework to existing setups for controlling false alarms when multiple A/B tests are run over time. This setup arises in many practical applications, e.g. when pharmaceutical companies test new treatment options against control pills for different diseases, or when internet companies test the…
This study uses OPE methods to quickly assess auction policies.
problem Rapid decision-making in dynamic auction environments.
method Off-Policy Evaluation and counterfactual methods.
result Improved policy selection and optimization.
Machine learning experiments often mislead due to unmet assumptions.
problem Machine learning experiments with pooled data may not meet necessary assumptions for unbiased causal effect estimation.
method Analysis of assumptions required for unbiased causal effect estimation in machine learning experiments.
result Practical applications of A/B-tests with machine learning models may not yield unbiased estimates of causal effect.
Paper improves PBO using Skew Gaussian Processes for better optimization.
problem Optimizing with preference judgments, especially in A/B tests and recommender systems.
method Uses Skew Gaussian Processes to model preference function and exact posterior inference.
result Exact SkewGP posterior leads to better optimization results than Laplace approximation.
LOLA uses LLMs to optimize content delivery, outperforming traditional methods.
problem Identifying the most engaging headlines for user engagement.
method LOLA integrates LLMs with adaptive experimentation to optimize content delivery.
result LOLA outperforms traditional methods in optimizing user engagement.
We consider the estimation of heterogeneous treatment effects with arbitrary machine learning methods in the presence of unobserved confounders with the aid of a valid instrument. Such settings arise in A/B tests with an intent-to-treat structure, where the experimenter randomizes over which user will receive a recomme…
Paper evaluates dynamic QTE for ridesharing data.
problem Assessing QTE in ridesharing with skewed outcomes.
method Developed VCDP models to estimate dynamic CQTE.
result Dynamic CQTE equals sum of individual CQTEs.
This paper tackles inventory control with general arrival dynamics and post-processing, improving profitability.
problem Inventory control with arbitrary arrival dynamics and post-processing constraints.
method Formulated as an exogenous decision process, incorporating deep generative models for arrivals, and applying supervised learning techniques.
result Improves profitability over production baselines and real-world A/B test data.
New framework for evaluating ad auctions using stochastic modeling.
problem Challenges in evaluating deterministic ad auctions.
method Repurposed bid landscape model to approximate propensity scores, enabling robust OPE estimators.
result Remarkable alignment with online A/B test results, achieving 92% MDA in CTR prediction.
We generalize to the finite-state case the notion of the extreme effect variable Y that accumulates all the effect of a variant variable V observed in changes of another variable X. We conduct theoretical analysis and turn the problem of finding of an effect variable into a problem of a simultaneous decomposition…