Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Reply
American Economic Review
vol. 112,
no. 9, September 2022
(pp. 3137-39)
Abstract
In Brodeur, Cook, and Heyes (2020) we present evidence that instrumental variable (and to a lesser extent difference-in-difference) articles are more p-hacked than randomized controlled trial and regression discontinuity design articles. We also find no evidence that (i) articles published in the top five journals are different; (ii) the "revise and resubmit" process mitigates the problem; (iii) things are improving through time. Kranz and Pütz (2022) apply a novel adjustment to address rounding errors. They successfully replicate our results with the exception of our shakiest finding: after adjusting for rounding errors, bunching of test statistics for difference-in-difference articles is now smaller around the 5 percent level (and coincidentally larger at the 10 percent level).Citation
Brodeur, Abel, Nikolai Cook, and Anthony Heyes. 2022. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Reply." American Economic Review, 112 (9): 3137-39. DOI: 10.1257/aer.20220277Additional Materials
JEL Classification
- A14 Sociology of Economics
- C12 Hypothesis Testing: General
- C52 Model Evaluation, Validation, and Selection