Innovative Evaluations for Government Interventions

Paper Session

Saturday, Jan. 7, 2017 2:30 PM – 4:30 PM

Sheraton Grand Chicago, Grant Park
Hosted By: Society of Government Economists
  • Chair: Austin Nichols, Abt Associates

The Effect of Marketing on Demand for OSHA’s On-Site Consultation Program: Experimental Evidence

Randall Juras
,
Abt Associates
Jacob Klerman
,
Abt Associates
Amy Minzner
,
Abt Associates

Abstract

In an effort to help businesses comply with safety and health regulations, the U.S. Occupational Safety and Health Administration (OSHA) offers a free and voluntary consultation service to small establishments in high-hazard industries. In an effort to increase use of this service, we developed several marketing brochures incorporating distinct behavioral theory-based messages and formats that could be mailed and/or emailed to employers. We then used a factorial research design with 18 study arms and a sample size of 34,000 worksites to test the impact of each theory-based message, exemplar, and delivery mode (mail or mail plus email) compared with an existing informational brochure and a no-marketing counterfactual. Each establishment in the treatment group was mailed three brochures in spring 2014, and we assessed the impact on requests for OSC services over the following six months using OSHA administrative data. We find that marketing via brochures is effective overall, nearly doubling the rate of requests for OSC services, but that the behavioral-theory-based messages perform no better than OSHA’s existing informational brochure. These results suggest that it is the informational component of the marketing rather than the behavioral messaging that drives the response.

Providing Free Pregnancy Test Kits to Community Health Workers Increases Distribution of Contraceptives: Results from an Impact Evaluation in Madagascar

Alison Comfort
,
University of California-San Francisco
Slavea Chankova
,
Abt Associates
Randall Juras
,
Abt Associates
Natasha Hsi
,
Abt Associates
Lauren Peterson
,
Abt Associates

Abstract

To improve access to contraceptives in remote and rural areas, sub-Saharan African countries are allowing community health workers (CHWs) to distribute hormonal contraceptives. Before offering hormonal contraceptives, CHWs must determine pregnancy status but often lack a reliable way to do so. No studies have evaluated the impact of providing CHWs with urine pregnancy test kits. We assessed the impact of giving CHWs free pregnancy test kits on the number of new clients purchasing hormonal contraceptives from CHWs. We implemented a randomized experiment in Eastern Madagascar among CHWs who sell injectable and oral hormonal contraceptives. A total of 622 CHWs were stratified by region and randomly assigned at the individual level. Treatment-group CHWs were given free pregnancy tests to distribute (n analyzed=272) and control-group CHWs did not receive the tests (n analyzed=263). We estimated an ordinary least-squares regression model, with the monthly number of new hormonal contraceptive clients per CHW as our primary outcome. We find that providing CHWs with free pregnancy test kits increases the number of new hormonal contraceptive clients. Treatment-group CHWs provide hormonal contraceptives to 3.1 new clients per month, compared to 2.5 in the control group. This difference of 0.7 clients per month (95% confidence interval 0.13–1.18; p=.014) represents a 26% increase. We find that giving CHWs free pregnancy tests is an effective way to increase distribution of hormonal contraceptives. No study has evaluated the impact of giving CHWs free urine pregnancy test kits for distribution to improve provision of hormonal contraceptives. As pregnancy tests become increasingly affordable for health-care systems in developing countries, community-based distribution programs should consider including the tests as a low-cost addition to CHWs’ services.

Designing Evaluations of “Pull” Mechanisms for Agricultural Technology Adoption: Early Lessons from AgResults

Judy Geyer
,
Abt Associates
Tulika Narayan
,
Abt Associates

Abstract

AgResults is a $110 million multilateral initiative funded by Australia, Canada, the United Kingdom, the United States, and the Bill & Melinda Gates Foundation to test the use of “pull mechanisms” to incentivize private companies to develop and disseminate high-impact agricultural innovations that promote food security and benefit smallholder farmers. The pull mechanisms use prizes to monetize positive externalities, mitigate information asymmetries, and motivate innovators’ initial investment. We describe the evaluation framework for this program, which currently includes six distinct pull mechanisms in as many countries. The evaluations, which are ongoing, include a mix of experimental, quasi-experimental, and non-experimental approaches. The overall evaluation framework and learning agenda for AgResults are common across all pull mechanisms to allow for comparison and synthesis on lessons learned on design and implementation. Based on our early process analysis findings and understanding of the economic theory underlying the pull mechanisms, we provide early insights on the suitability of pull mechanisms in various contexts. Specifically, we comment on whether pull mechanisms can address the market failures that have thus far limited the development of market for agricultural innovations, and have an impact on development outcomes.

Which Interventions Work Best for Homeless Families: Three-Year Impacts from the Family Options Study

Daniel Gubits
,
ABT Associates
Marybeth Shinn
,
Vanderbilt University
Michelle Wood
,
ABT Associates
Stephen Bell
,
ABT Associates

Abstract

What housing and service interventions work best to reduce homelessness for families in the U.S.? The Family Options Study randomly assigned 2,282 families recruited in homeless shelters across 12 sites to priority access to one of three active interventions or to usual care in their communities. The interventions were permanent housing subsidies, temporary rapid re-housing subsidies, and transitional housing in supervised programs with intensive psychosocial services. In two waves of follow-up data collected 20 and 37 months later, priority access to permanent subsidies reduced homelessness and food insecurity and improved other aspects of adult and child well-being relative to usual care, at a cost 9 percent higher. The other interventions had little effect. The study provides support for the view that homelessness for most families is an economic problem that permanent subsidies resolve and not the view that families must address psychosocial problems to succeed in housing. It has implications for focusing government resources on this important social problem.
Discussant(s)
Fatih Unlu
,
Abt Associates
Linden McBride
,
Cornell University
Shawn Moulton
,
Abt Associates
Austin Nichols
,
Abt Associates
JEL Classifications
  • C0 - General