Heckman and Moktan on the Top 5

Question

Marginal Revolution links to this provocative paper by Heckman and Moktan: http://www.nber.org/papers/w25093.

From the abstract: "Pursuit of T5 publications has become the obsession of the next generation of economists. However, the T5 screen is far from reliable. A substantial share of influential publications appear in non-T5 outlets."

From the conclusion: "The current practice has weak empirical support if judged by its ability to produce papers that last in terms of citation counts... Academics who impose the T5 standard impose a standard that they themselves do not follow. They primarily publish in, read and cite non-T5 journals, as will the candidates who survive the T5 filter and become tenured faculty. Reliance on the T5 as a screening device raises serious concerns."

Some of the more provocative sections include:
"4.3 The T5 are Not the Journals with the Top Five Impact Factors in Economics"
"5. Openness and Incest"

To what extent is this perceived as a genuine problem by the wider profession? What is the view of those with the power to change things (such as tenure letter writers)? Is there anything that juniors can do about this, and should they? Is the impression correct that this state of affairs is quite unique to economics, and if so, why is this the case?

Answer 1

They are right that Top-5s play a huge role in tenure - being a senior in a Top-5 department I can confirm that. Similarly, it influences letters a lot. I should Google Scholar is also hugely important - it is hard to tenure somebody with low citations.

But I think this paper actually supports Top-5s. If you remove survey papers (JEP, JEL and to some extent JEG) and finance papers (they are heavily cited by practitioners which is not a measure of research impact) then the Top-5 are clearly dominant.

So yes I realized many people (including myself and i suspect the authors of this) are upset at not having papers published in the top-5. They often also think this is unfair - it is natural to think one's own research is particularly important otherwise we would exit academia. But the current system seems better than any other alternative.  It reminds me of Churchill's quote on democracy "Democracy is the worst form of Government except for all those other forms that have been tried from time to time" - don't abandon a system until you have a better alternative.

Comments

A minor point: I suspect the authors are not all that upset at not having papers published in the top-5. One of the authors is a pre-doctoral fellow. The other is an editor of a top-5...and there's that Nobel prize thing he has. :)

Answer 2

I would like to start this by saying a lot of the time my papers have been rejected for completely valid reasons.  I'd like to think that when they've been published, its also been valid reasons.  

The fact of the matter is that publishing in general is part luck, part skill.  Getting the right editor matters, meaning one who is just excited about your question, (especially if the referees are split).   

Can we be too focused on top 5's....sure.  I know I have been when I get depressed at rejections I've received for papers I felt were great.  

In the long run, what matters is influence.  If I look at my most most influential papers so far, they have been those published in field journals (but also got a ton of press).  But measuring influence in 5 years is pretty trick sometimes.  

Is a lexicograph standard right?  I tend to think not.  Especially at departments ranked 30-60 who may or may not have a standard of at least 1 top 5.  Just because I know people who were on the bubble for tenure, and then that one final pub decided everything.  They've had fantastic careers since then.  But that last acceptance probably did not add much signal given everything else one could see about them.

Or another way to address this is to lengthen tenure clocks (say ten years), and have more promotions prior to tenure (so that getting tenure is more about research creativity and freedom than a salary bump and job guarantee).  Some places are experimenting with this.