In several earlier posts (A Mixed Bag, , More re Simmons et al, Part I, More re Simmons et al, Part II, More re Simons et al, Part III, Though Many Social Psychologists May Be Unaware, Multiple Testing Often Leads to Multiple Spurious Conclusions, Beyond the Buzz Part IV: Multiple Testing), I have commented on lack of accounting for multiple testing in literature in psychology. So I was pleased to receive notice of a paper “Hidden Multiplicity in the Multiway ANOVA: Prevalence, Consequences and Remedies” that has been submitted by Cramer et al to PLoS ONE and is currently available on the arXiv.
In the paper, the authors examined all articles published in 2010 in six widely read and cited psychology journals, identifying those articles using a multiway ANOVA. They found that close to half of the articles did indeed use a multiway ANOVA, but only about 1% of those papers used some correction for multiple testing (with percentages in individual journals ranging from 0 to 2.59).
They then randomly chose 10 papers from each journal that involved at least one multiway ANOVA, and used the Holm-Bonferroni procedure to adjust for multiple testing. They found that in 75% of the articles, at least one test that had been declared significant could no longer qualify as significant after the Holm-Bonferroni correction. Regrettably, this does not surprise me, based on the articles in psychology journals that I have read.
I hope this paper gets wide circulation and helps change practices in the field.