Meunier, L., & Ohadi, S.
Is p < .05 a reasonable threshold? An experiment on the implicit p-value and its determinants
Significance levels of 1%, 5%, or 10% are routinely used in science. These levels were chosen out of intuition and convenience by the founders of modern statistics. Does their intuition correspond with the ones of the rest of the population? We investigate this question through a rigged coin-tossing game in two studies (N = 363 and N = 208). These studies highlight that a first doubt is framed at an average p-value of 7%. On average, subjects were sufficiently convinced the game was manipulated to stop it when the p-value reached 0.46%. Regarding explanatory variables for these implicit significance levels, we find that participants having a lower internal locus of control and scoring higher on the Cognitive Reflection Test (CRT) were more likely to realize the experiment was manipulated. We also find that participants with a higher score in trust and an external locus of control had lower implicit significance levels. Participants who did not realize the experiment was manipulated or who did not stop the experi-ment on their own had a lower propensity to have ever invested in the stock market.
Testing, Psychometrics, Methodology in Applied Psychology, 2022, Vol. 29, pp. 419-429, DOI: 10.4473/TPM29.4.2
