Dansk Center for Forskningsanalyse

Caveats for using statistical significance tests in research assessments

Publikation: Bidrag til tidsskrift/Konferencebidrag i tidsskrift /Bidrag til avisTidsskriftartikelForskningpeer review

Standard

Caveats for using statistical significance tests in research assessments. / Schneider, Jesper Wiborg.

I: Journal of Informetrics, Bind 7, Nr. 1, 01.2013, s. 50-62.

Publikation: Bidrag til tidsskrift/Konferencebidrag i tidsskrift /Bidrag til avisTidsskriftartikelForskningpeer review

Harvard

APA

CBE

MLA

Vancouver

Author

Schneider, Jesper Wiborg. / Caveats for using statistical significance tests in research assessments. I: Journal of Informetrics. 2013 ; Bind 7, Nr. 1. s. 50-62.

Bibtex

@article{31a1ed0fae284aefabb21dc0e1d6fc27,
title = "Caveats for using statistical significance tests in research assessments",
abstract = "This article raises concerns about the advantages of using statistical significance tests in research assessments as has recently been suggested in the debate about proper normalization procedures for citation indicators by Opthof and Leydesdorff (2010). Statistical significance tests are highly controversial and numerous criticisms have been leveled against their use. Based on examples from articles by proponents of the use statistical significance tests in research assessments, we address some of the numerous problems with such tests. The issues specifically discussed are the ritual practice of such tests, their dichotomous application in decision making, the difference between statistical and substantive significance, the implausibility of most null hypotheses, the crucial assumption of randomness, as well as the utility of standard errors and confidence intervals for inferential purposes. We argue that applying statistical significance tests and mechanically adhering to their results are highly problematic and detrimental to critical thinking. We claim that the use of such tests do not provide any advantages in relation to deciding whether differences between citation indicators are important or not. On the contrary their use may be harmful. Like many other critics, we generally believe that statistical significance tests are over- and misused in the empirical sciences including scientometrics and we encourage a reform on these matters.",
keywords = "statistical significance tests, scientometrics, research evaluation, controversy",
author = "Schneider, {Jesper Wiborg}",
year = "2013",
month = jan,
doi = "10.1016/j.joi.2012.08.005",
language = "English",
volume = "7",
pages = "50--62",
journal = "Journal of Informetrics",
issn = "1751-1577",
publisher = "Elsevier BV",
number = "1",

}

RIS

TY - JOUR

T1 - Caveats for using statistical significance tests in research assessments

AU - Schneider, Jesper Wiborg

PY - 2013/1

Y1 - 2013/1

N2 - This article raises concerns about the advantages of using statistical significance tests in research assessments as has recently been suggested in the debate about proper normalization procedures for citation indicators by Opthof and Leydesdorff (2010). Statistical significance tests are highly controversial and numerous criticisms have been leveled against their use. Based on examples from articles by proponents of the use statistical significance tests in research assessments, we address some of the numerous problems with such tests. The issues specifically discussed are the ritual practice of such tests, their dichotomous application in decision making, the difference between statistical and substantive significance, the implausibility of most null hypotheses, the crucial assumption of randomness, as well as the utility of standard errors and confidence intervals for inferential purposes. We argue that applying statistical significance tests and mechanically adhering to their results are highly problematic and detrimental to critical thinking. We claim that the use of such tests do not provide any advantages in relation to deciding whether differences between citation indicators are important or not. On the contrary their use may be harmful. Like many other critics, we generally believe that statistical significance tests are over- and misused in the empirical sciences including scientometrics and we encourage a reform on these matters.

AB - This article raises concerns about the advantages of using statistical significance tests in research assessments as has recently been suggested in the debate about proper normalization procedures for citation indicators by Opthof and Leydesdorff (2010). Statistical significance tests are highly controversial and numerous criticisms have been leveled against their use. Based on examples from articles by proponents of the use statistical significance tests in research assessments, we address some of the numerous problems with such tests. The issues specifically discussed are the ritual practice of such tests, their dichotomous application in decision making, the difference between statistical and substantive significance, the implausibility of most null hypotheses, the crucial assumption of randomness, as well as the utility of standard errors and confidence intervals for inferential purposes. We argue that applying statistical significance tests and mechanically adhering to their results are highly problematic and detrimental to critical thinking. We claim that the use of such tests do not provide any advantages in relation to deciding whether differences between citation indicators are important or not. On the contrary their use may be harmful. Like many other critics, we generally believe that statistical significance tests are over- and misused in the empirical sciences including scientometrics and we encourage a reform on these matters.

KW - statistical significance tests

KW - scientometrics

KW - research evaluation

KW - controversy

U2 - 10.1016/j.joi.2012.08.005

DO - 10.1016/j.joi.2012.08.005

M3 - Journal article

VL - 7

SP - 50

EP - 62

JO - Journal of Informetrics

JF - Journal of Informetrics

SN - 1751-1577

IS - 1

ER -