+6 votes
asked ago in General Economics Questions by (1.3k points)
So far in 2018, the AER has published 4 comments with a few more in the queue. There has been 1 published in the other top 5 journals combined (in Econometrica). Although I haven't checked, I believe this discrepancy holds for at least the recent past (5-10 years). I can think of 3 possible reasons and am wondering which it is:

1. The quality of papers is lower in AER relative to the other top journals (at least a fatter left tail of quality). So more opportunities to comment.
2. Other journals either prohibit comment papers or the bar for a comment is much higher (e.g., the comment has to completely overturn a result rather than just causing a meaningful change in the result (which is the AER threshold and is somewhat subjective)).
3. Other journals are willing to publish comments with similar standards for publication, but we're in a weird equilibrium where they haven't published them so people assume they won't. Given that comments tend to be paper and journal specific, I'm not sure I'd ever bother for a non-AER paper since I can't tell what the threshold for publication is or whether it would even be considered (as the issue is never addressed in the submission guidelines on their websites).

I suspect the answer is 2. If so, what is the preferable policy?  I personally like AER's policy and openess to consider comments, which I think is much more in the interest of promoting open science. If 2 is the answer, does anyone affiliated with the other journals care to defend your policies? 1 doesn't seem plausible, although the move to monthly issues and the increase in articles may have changed things. 3 would require everyone act like me and make certain assumptions when the issue isn't specifically addressed which seems unlikely. I'm sure some people would just email someone and ask.

1 Answer

+3 votes
answered ago by (440 points)
All the incentives are against publishing comments. The original authors obviously don't want them. It's an indictment of the editors they did not spot the problem and the comment is very unlikely to generate citations. So the journal is against. In this more competitive age, these factors have come to be more important. So it is probably 2) that applies. Yet it is valuable to know that a paper is wrong or has issues. Page for page, comments are probably more valuable than original papers. So it is to the credit of AER that they still publish them, albeit less often than they used to.
commented ago by (120 points)
The original writer raising this issue referred to only the 2018 (and a very short-window of forthcoming) AER publication of "comment articles".  I think that good sampling theory would say we need longer time-series empirical data on The AER and non-AER top Econ. journals. Only after more careful analysis of such richer sample data can the propositions set forth be soundly addressed scientifically. Am I correct?
commented ago by (1.3k points)
I looked at additional years. In 2016 and 2017, there were 11 comments published in AER and 2 published in the other 4 top 5 journals combined.