Should regression tables use * for significant coefficients?

Question

EconSpark is dying, alas. It can't compete with Economics Job Market Rumors, @Econtwitter, and Twitter generally, despite being a differentiated product.  Still, I'll keep trying.


 A recent Tweet by @jhaushofer said,
 
"Retweet if the AER's new policy of not showing significance stars has changed little apart from a) slowing you down and b) making you a lot better at dividing by two."

Is the new policy efficient?  My own response was:

Well said, well phrased. What the AER should *really* do is italicize insignificant coefficients, normalfont the 5%, and boldface the 1%, except in contexts where even 1% is meaningless because the dataset is giant.

Answer 1

The deeper question is whether economists should abandon the statistical significance framework altogether, as advocated by, i.a., the comment in Nature earlier this year entitled "Retire Statistical Significance".  I'm on the yes side.  By all means, report p values or compare coefficient estimates to standard errors, but the dichotomous significant/insignificant determination is counterproductive -- so no need for asterisks.  Beneath this is the still deeper question of how we should evaluate the credibility of statistical evidence.  That's a big topic, but increasingly those who consider it are moving away from criteria internal to the statistical properties of a given result and more toward questions of measurement, model suitability, place in the broader empirical literature, etc.

I'm hardly an expert in this stuff, but I find it fascinating.  It will be interesting to see how the significance controversy develops.

Comments

Great read Amrhein, Greenland, and McShane (2019)! I think the two biggest ideas are: (1) "We must learn to embrace uncertainty." and (2) "that it is needed to make yes-or-no decisions.".

I usually don't like to report point estimates and prefer confidence intervals given it is a better picture of uncertainty and that the study can actually provide as estimates. My personal take is that it most be taken as what it is and not as what it isn't. If the question is whether the study provides evidence of statistically difference at some confidence interval, then consider the design, the data, and the estimates. For updating your beliefs (science is all about beliefs), take those results as what they are. All that said, it is quite useful to have rules for making decisions as for average bioequivalence determination which are based on a quasi-standard rule (is the confidence intervals at some level within x-margin).

I see value in reporting those measures, but the question is whether we can get to a first-best where those can be useful or a second-best as avoiding those in order to minimize the harm from misusage.

Answer 2

AEA has I gather gone to a system where you cannot use asterisks.  I think that's a mistake-- asterisks developed and are still widely used because there are many good reasons to compress continuous information into categorical bins, full stop.
I often have to go through and circle or mark the effects that are notable, which is just a little easier than putting back in asterisks. It could well be that better visualization would convey information effectively without asterisks (e.g. confidence intervals so one can easily see "distance from null=0", say, and the eye can essentially imagine the strength of effect in NHST which binned p-value is getting at.

From a design point of view, it's usually a bad idea to prohibit one kind of visual shortcut without suggesting or even requiring another.

There's also a recent paper analyzing a journal that *banned inferential statistics* (p-values, F-stats, confidence intervals).
An analysis of papers afterward concludes that
"We found multiple instances of authors overstating conclusions beyond what the data would support if statistical significance had been considered. Readers would be largely unable to recognize this because the necessary information to do so was not readily available."
https://www.tandfonline.com/doi/full/10.1080/00031305.2018.1537892

In other words, asterisks are just an author's way of making salient what they think a casual reader (with a normal brain that processes symbols more rapidly than language)  should take away. If you take away the asterisks you get more confusion, more hyperbolic language, or something.