How and why alpha should depend on sample size: A Bayesian-frequentist compromise for significance testing

Jesper Wulff*, Luke Nicholas Taylor

*Corresponding author for this work

Research output: Contribution to journal/Conference contribution in journal/Contribution to newspaperJournal articleResearchpeer-review

5 Citations (Scopus)

Abstract

In management research, fixed alpha levels in statistical testing are ubiquitous. However, in highly powered studies, they can lead to Lindley’s paradox, a situation where the null hypothesis is rejected despite evidence in the test actually supporting it. We propose a sample-size-dependent alpha level that combines the benefits of both frequentist and Bayesian statistics, enabling strict hypothesis testing with known error rates while also quantifying the evidence for a hypothesis. We offer actionable guidelines of how to implement the sample-size-dependent alpha in practice and provide an R-package and web app to implement our method for regression models. By using this approach, researchers can avoid mindless defaults and instead justify alpha as a function of sample size, thus improving the reliability of statistical analysis in management research.

Original languageEnglish
JournalStrategic Organization
Volume22
Issue3
Pages (from-to)550-581
Number of pages32
ISSN1476-1270
DOIs
Publication statusPublished - Aug 2024

Keywords

  • Bayes factor
  • Bayesian-frequentist compromise
  • alpha level
  • hypothesis testing
  • significance testing

Fingerprint

Dive into the research topics of 'How and why alpha should depend on sample size: A Bayesian-frequentist compromise for significance testing'. Together they form a unique fingerprint.

Cite this