Empirical Bayes Estimation of Treatment Effects with Many A/B Tests: An Overview
- (pp. 43-47)
Abstract
The use of large-scale experimentation to screen product innovations is increasingly common. This is a practical guide on how to use treatment effect estimates from a large number of experiments to improve estimates of the effects of each experiment. When thousands of new features are A/B tested by internet companies, the winners tend to be a combination of good features and features that got lucky experimental draws. Empirical Bayes methods are a commonly used tool in statistics to separate good features from lucky draws. We give a user-friendly overview of both classic and recent approaches to this problem.Citation
Azevedo, Eduardo M., Alex Deng, José L. Montiel Olea, and E. Glen Weyl. 2019. "Empirical Bayes Estimation of Treatment Effects with Many A/B Tests: An Overview." AEA Papers and Proceedings, 109: 43-47. DOI: 10.1257/pandp.20191003Additional Materials
JEL Classification
- C11 Bayesian Analysis: General
- C21 Single Equation Models; Single Variables: Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions
- C31 Multiple or Simultaneous Equation Models: Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions; Social Interaction Models
- C51 Model Construction and Estimation
- O31 Innovation and Invention: Processes and Incentives