Evidence Use in Policymaking
Paper Session
Saturday, Jan. 7, 2023 2:30 PM - 4:30 PM (CST)
- Chair: Eva Vivalt, University of Toronto
How Do Policy-makers Update Their Beliefs?
Abstract
We present results from experiments run in collaboration with the World Bank and Inter-American Development Bank on how policy-makers, policy practitioners, and researchers update their beliefs in response to results from academic studies. Initially, policy-makers both believe development programs will have more positive results and are more certain about it than policy practitioners and researchers, despite reporting less familiarity with the programs. When participants are presented with the results of impact evaluations, we find evidence of asymmetric updating on good news and insensitivity to confidence intervals across all groups, but few differences between groups. We also observe a form of overconfidence. Finally, we show that research findings affect a real-life allocation decision.Bottlenecks for Evidence Adoption
Abstract
Governments increasingly use RCTs to test innovations before scale up. Yet, we know little about whether and how they incorporate the results of the experiments. We follow up with 67 US city departments which collectively ran 73 RCTs in collaboration with a national Nudge Unit to improve city communications using nudges. The city departments adopt the nudge treatment in follow-on communication in 27% of the 73 RCTs. As potential determinants of adoption we consider (i) the strength of the evidence, (ii) features of the organization, such as “state capacity” of the city and whether the staff member working on the RCT is still involved, and (iii) features of the treatment, such as whether it was implemented as part of pre-existing communication. We find (i) a limited impact of strength of the evidence and (ii) some impact of city features, especially the retention of the original staff member. By far, the largest predictor of adoption is (iii) whether the communication was pre-existing, as opposed to a new communication. We consider two main interpretations of this finding: organizational inertia, in that changes to pre-existing communications are more naturally folded into the year-to-year city communication, and costs, since new communications may require additional funding. We find the same pattern for electronic communications, with zero marginal costs, supporting the organizational inertia explanation. We compare the findings to the predictions of experts and practitioners. Forecasters over-estimate the extent of evidence-based adoption and, while they anticipate the importance of inertia, they do not foresee the channel through which it would operate. We thus stress the importance of creating a path to adoption as part of the RCT design when the goal is implementation into policy.Training Policymakers in Econometrics
Abstract
The credibility revolution triggered a paradigm shift in economics. This paper examines its causal effects on deputy ministers in a “mastering metrics” training program. We separated the demand for econometrics training from its impact with a simplified Becker Degroot Marshak mechanism. Policymakers could choose a high or low probability for randomly receiving a popular econometrics or a self-help placebo book. After receiving the book, policymakers participated in an intense training workshop that included watching lecture videos made by the authors of the book, summarizing each chapter, discussing, presenting, and applying the book’s concepts in their policymaking. Three results emerge. First, we document large persistent effects. After six months, treated individuals' ratings on the importance of quantitative analysis increase by 50%. Treated individuals' performance in national research methods and public policy exams improves by 0.5-0.8 sigma. Text analysis of their writings reflect an increase in perceived importance of causal inference. Second, treated individuals’ willingness-to-pay for commissioning Randomized Control Trials using public funding increases by 300% and decreases by 50% for correlational studies. Third, treated ministers are twice as likely to choose a policy for which there is RCT evidence. We use click behavior as a behavioral proxy of IV defiers. Few defiers are observed, and they are less affected by treatment. Last, one year after the training, in their official duties, treated ministers are twice as likely to choose and triple the funding for policies for which there is RCT evidence. Overall, we provide experimental evidence that training policymakers in the school of thought associated with the credibility revolution increases demand and responsiveness to causal evidence.Understanding and Increasing Policymakers' Sensitivity to Program Impact
Abstract
Policymakers routinely make high-stakes funding decisions. In two experiments with policymakers in the U.S. government and the general public, we find that valuations of programs are inelastic with respect to program impact. We design and test two decision aids, one which presents programs side-by-side and another which translates multiple features of impact into an aggregate metric. The decision aids increase elasticity by 0.20 on a base of 0.33 among policymakers and by 0.21 on a base of 0.21 among the general public. We provide evidence that the difficulty of assessing complex inputs can help explain the inelasticity of program valuations.Discussant(s)
Yiming Cao
,
Boston University
Linh Tô
,
Boston University
Julius Rueschenpoehler
,
Northwestern University
Michael Price
,
University of Alabama
Eugen Dimant
,
University of Pennsylvania
JEL Classifications
- D7 - Analysis of Collective Decision-Making
- O2 - Development Planning and Policy