Fast Selection from Multiple Treatments: A Sequential Method for Principled Digital Experimentation

Digital experimentation faces a sample size planning problem: the typically small treatment effects require a very large number of participants to be reliably detected, and yet the large number of simultaneous experiments being conducted by many platforms means that the number of available users to participate is often limited. Meanwhile, common practice often involves rules of thumb such as setting experiments to run for two weeks, which can result in substantial over- or undersampling, depending on unknown parameters of the distribution of the focal outcome variable. To address this problem, we propose a novel framework and method for sequentially sampled experiments with multiple treatment arms, where the objective is to determine the best treatment arm and whether that arm provides a practically significant improvement over the current business-as-usual platform condition. Our approach is unique among sequentially sampled best-arm identification methods in allowing experimenters to specify separate error rates for different failure conditions and separate effect-size margins for selection and testing tasks. We provide extensive conceptual and empirical comparisons with a suite of comparison methods that can serve as a useful guide to practitioners. Our method is freely available in the R package seqbest.

• 저널Information Systems Research
• 토픽Advanced Causal Inference Techniques · Statistics and Probability
• DOI10.1287/isre.2024.1537
• 저자Will Stamey, Ken Kelley, Bhargab Chattopadhyay, Tathagata Bandyopadhyay