The Danish Centre for Studies in Research and Research Policy

Jesper Wiborg Schneider

Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations

Research output: Contribution to book/anthology/report/proceedingArticle in proceedingsResearchpeer-review

In this paper we discuss and question the use of statistical significance tests in relation to university rankings as recently suggested. We outline the assumptions behind and interpretations of statistical significance tests and relate this to examples from the recent SCImago Institutions Ranking. By use of statistical power analyses and demonstration of effect sizes, we emphasize that importance of empirical findings lies in “differences that make a difference” and not statistical significance tests per se. Finally we discuss the crucial assumption of randomness and question the presumption that randomness is present in the university ranking data. We conclude that the application of statistical significance tests in relation to university rankings, as recently advocated, is problematic and can be misleading.
Original languageEnglish
Title of host publicationProceedings of 17th International Conference on Science and Technology Indicators
EditorsÉric Archambault, Yves Gingras, Vincent Larivière
Number of pages14
Publication yearSep 2012
Pages719-732
Publication statusPublished - Sep 2012
Event International Conference on Science and Technology Indicators - Université du Québec à Montréal , Montreal, Quebec, Canada
Duration: 5 Sep 20128 Sep 2012
Conference number: 17th

Conference

Conference International Conference on Science and Technology Indicators
Nummer17th
LocationUniversité du Québec à Montréal
LandCanada
ByMontreal, Quebec
Periode05/09/201208/09/2012

Bibliographical note

Paper presented at 17th International Conference on Science and Technology Indicators (STI), 5-8 September, 2012 in Montreal, Quebec, Canada.

    Research areas

  • statistical significance tests, controversy, university rankings, effect sizes, statistcial power, scientometrics, indicators

See relations at Aarhus University Citationformats

Download statistics

No data available

ID: 52188038