TY - GEN

T1 - Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations

AU - Schneider, Jesper Wiborg

N1 - Conference code: 17th

PY - 2012/9

Y1 - 2012/9

N2 - In this paper we discuss and question the use of statistical significance tests in relation to university rankings as recently suggested. We outline the assumptions behind and interpretations of statistical significance tests and relate this to examples from the recent SCImago Institutions Ranking. By use of statistical power analyses and demonstration of effect sizes, we emphasize that importance of empirical findings lies in “differences that make a difference” and not statistical significance tests per se. Finally we discuss the crucial assumption of randomness and question the presumption that randomness is present in the university ranking data. We conclude that the application of statistical significance tests in relation to university rankings, as recently advocated, is problematic and can be misleading.

AB - In this paper we discuss and question the use of statistical significance tests in relation to university rankings as recently suggested. We outline the assumptions behind and interpretations of statistical significance tests and relate this to examples from the recent SCImago Institutions Ranking. By use of statistical power analyses and demonstration of effect sizes, we emphasize that importance of empirical findings lies in “differences that make a difference” and not statistical significance tests per se. Finally we discuss the crucial assumption of randomness and question the presumption that randomness is present in the university ranking data. We conclude that the application of statistical significance tests in relation to university rankings, as recently advocated, is problematic and can be misleading.

KW - statistical significance tests

KW - controversy

KW - university rankings

KW - effect sizes

KW - statistcial power

KW - scientometrics

KW - indicators

M3 - Article in proceedings

SP - 719

EP - 732

BT - Proceedings of 17th International Conference on Science and Technology Indicators

A2 - Archambault, Éric

A2 - Gingras, Yves

A2 - Larivière , Vincent

T2 - International Conference on Science and Technology Indicators

Y2 - 5 September 2012 through 8 September 2012

ER -