This website uses cookies.

By clicking the "Accept" button or continuing to browse our site, you agree to first-party and session-only cookies being stored on your device to enhance site navigation and analyze site performance and traffic. For more information on our use of cookies, please see our Privacy Policy.

  1. Home
  2. ASSA Annual Meeting
  3. 2023
  4. Program
  5. Replications in Economics
« Back to Results

Replications in Economics

Paper Session

Sunday, Jan. 8, 2023 10:15 AM - 12:15 PM (CST)

Hilton Riverside, Royal
Hosted By: American Economic Association
  • Chairs:
    Edward Miguel, UC Berkeley
  • Abel Brodeur, University of Ottawa

Defining Replicability in Economics

Jorg Peters
,
Leibniz Institute for Economic Research and University of Passau
Anna Dreber
,
Stockholm School of Economics
Nathan Fiala
,
University of Connecticut and Leibniz Institute for Economic Research
Magnus Johannesson
,
Stockholm School of Economics
Florian Neubauer
,
University of Connecticut and Leibniz Institute for Economic Research
Julian Rose
,
Leibniz Institute for Economic Research and University of Passau

Abstract

Replication and reanalyses are increasingly important in economics and the social sciences to test for robustness and generalizability, as well as to prevent questionable research practices. Assessing the success or the failure of a replication requires a definition of replicability. In this paper, we establish definitions for different types of replications. We use standardized replications of laboratory experiments as the starting point, for which both binary and non-binary definitions exist that build on the direction, magnitude and statistical significance of effects. Our main goal is to explore definitions for reanalysis and robustness checks of studies outside the laboratory – henceforth, R&R replications. Unlike for the typical laboratory replication, these replications use the same data as in the original study. We use two R&R replications of a Randomized Controlled Trial (Banerjee et al. 2021[1]) and an Instrumental Variable study (Rogowski et al. 2021[2]) to illustrate different definitions based on the fraction of R&R replications that “binary replicates”, the relative effect sizes of the R&R replications, and the joint significance of the R&R replications. We also develop an index incorporating yes/no quality checklists that can be used as a proxy replicability measure for studies that are not directly R&R replicated. In a next step, we apply these definitions to about 40 R&R replication debates published in the American Economic Review as comments and replies between 2010 and 2020. Finally, we carry out ~30 new R&R replications applying these definitions and we run a prediction market on the outcomes of these replications.

Revisiting Habits and Heterogeneity in Demands

Markus Fritsch
,
University of Passau
Andrew Adrian Pua
,
Xiamen University
Joachim Schnurbus
,
University of Passau

Abstract

We revisit a study by Browning and Collado (Journal of Applied Econometrics 2007; 22(3): 625-640), where habits and heterogeneity in demands of Spanish households are analyzed through a linear dynamic panel version of an Engel curve estimated using iterated GMM. They study six categories of consumption goods for durability and habit formation, assess the corresponding demand elasticities, and investigate the related loss from habits encountered by the households. We find that the coefficient estimates and standard errors differ from the reported results when we use their reported instrument set; in particular, we find habit formation in non-durable services and no state dependence in small durables. Our results support their main claim that intertemporal dependence in consumption is not strong enough to resolve existing macro puzzles, despite finding evidence for weak instruments in some specifications.

Revisiting the Effect of Growing Up in a Recession on Attitudes Toward Redistribution

Jan Bietenbeck
,
Lund University
Petra Thiemann
,
Lund University

Abstract

Giuliano and Spilimbergo (2014) show that individuals who experienced a recession when young are more likely to favor redistribution in the short and long run. We revisit their analysis in three ways. First, we conduct a narrow replication in the General Social Survey and the World Values Survey; we successfully replicate the original results for outcomes that directly measure preferences for redistribution, but the results for other outcomes are less clear-cut. Second, adding recent survey waves yields results similar to the narrow replication. Third, a wide replication in a different dataset (International Social Survey Programme) corroborates the original results.

Does Paid Parental Leave Affect Children’s Schooling Outcomes? Replicating Danzer and Lavy (2018)

Claudia Troccoli
,
Aalto University

Abstract

Danzer and Lavy (2018) study how the duration of paid parental leave affects children's educational performance using data from PISA. The extension of the maximum duration from 12 to 24 months had no significant effect on average but the authors highlight the existence of large and significant heterogenous effects that vary in sign depending on the education of mothers. The policy increased the scores obtained by sons of highly educated mothers, as measured in standard deviations, by 0.33 in Reading and 0.40 in Science. On the contrary, sons of lower educated mothers experienced a decrease of 0.27 in Reading and 0.23 in Science. In this article, I replicate their study following the appropriate estimation procedure that takes into account that PISA relies on imputation to derive student scores. I show that the estimates of the effects of the parental leave extension become substantially lower and non-significant.

Institute for Replication

Abel Brodeur
,
University of Ottawa

Abstract

This presentation introduces the Institute for Replication (I4R), a coordinated effort to improve the credibility of science by promoting reproductions and replications. Our team of collaborators supports researchers and aims to improve the credibility of science by (1) reproducing, conducting sensitivity analysis of, and replicating studies published in leading journals; (2) establishing an open-access website (https://i4replication.org/) to serve as a central repository containing the replications, responses by the original authors, and documentation; (3) developing and providing access to educational materials on replication and open science; and (4) preparing standardized file structure, code, and documentation aimed at facilitating reproducibility and replicability by the broader community.

Discussant(s)
Edward Miguel
,
University of California-Berkeley
Fernando Hoces de la Guardia
,
University of California-Berkeley
Abel Brodeur
,
University of Ottawa
Jorg Peters
,
Leibniz Institute for Economic Research and University of Passau
Jan Bietenbeck
,
Lund University
JEL Classifications
  • A1 - General Economics
  • J0 - General