Discussant (8) in the Psychological Science editorial discussion mentioned in my January 23 post wrote,
“The [Simmons et al] paper seems to be based on the possibility of a direct link between the data and the truth of a theory. In my view, it is always the educated reader who needs to be persuaded using convincing methodology. Therefore, I am not interested in the autobiography of the researcher. That is, I do not care whether s/he has actually held the tested hypothesis before learning about the outcomes, I am not interested how many failed studies preceded the submitted paper and I do not want to know whether if the results would have been insignificant with a lower number of subjects.
Finally, I see no reason to report inconsistent findings that were assessed after the main DV …”
If a reader of papers employing frequentist hypothesis testing is indeed [well] educated, s/he should not find methodology convincing unless (among other things) the paper discusses which hypotheses were pre-planned and which were “data snooping;” the history of previous attempts to establish the result; and how multiple testing (including hypothesis testing done after testing the main dependent variable) was taken into account in establishing claims of statistical significance — in other words, exactly the things discussant (8) doesn’t care about/isn’t interested in/doesn’t want to know/sees no reason to report. (Take a look at xkcd, Significant, if you haven’t already seen it, to help drive the point home.)
Ironically, discussant (8) ends with
“Our decision should be less based on the cuteness of the findings and the headlines the[y] might cause in the popular press but more on the answers they provide concerning the underlying psychological process. Less ‘wow!’ and more ‘how?’ might be another guiding principle for the new editorship.”
I agree that editorial decisions should not be based on cuteness of findings nor on headlines they might cause. I agree that less “wow!” and more “how?” sounds like a good guiding principle — but the “how?” needs to include how the results were obtained, and whether or not the methodology is sound. The items that discussant (8) doesn’t care about/isn’t interested in/doesn’t want to know/sees no reason to report are indeed important to the “how?”.