- About the Centre
- Research projects
- Publications
- Public sector consultancy
- PhD education at CFA
- News
- Staff
- Contact

Research output: Contribution to book/anthology/report/proceeding › Article in proceedings › Research › peer-review

**Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations.** / Schneider, Jesper Wiborg.

Research output: Contribution to book/anthology/report/proceeding › Article in proceedings › Research › peer-review

Schneider, JW 2012, Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations. in É Archambault, Y Gingras & V Larivière (eds), *Proceedings of 17th International Conference on Science and Technology Indicators.* pp. 719-732, International Conference on Science and Technology Indicators , Montreal, Quebec, Canada, 05/09/2012. <http://sticonference.org/Proceedings/vol2/Schneider_Testing_719.pdf>

Schneider, J. W. (2012). Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations. In É. Archambault, Y. Gingras, & V. Larivière (Eds.), *Proceedings of 17th International Conference on Science and Technology Indicators *(pp. 719-732) http://sticonference.org/Proceedings/vol2/Schneider_Testing_719.pdf

Schneider JW. 2012. Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations. Archambault É, Gingras Y, Larivière V, editors. In Proceedings of 17th International Conference on Science and Technology Indicators. pp. 719-732.

Schneider, Jesper Wiborg "Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations"., Archambault, Éric Gingras, Yves Larivière , Vincent (editors). *Proceedings of 17th International Conference on Science and Technology Indicators.* 2012, 719-732.

Schneider JW. Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations. In Archambault É, Gingras Y, Larivière V, editors, Proceedings of 17th International Conference on Science and Technology Indicators. 2012. p. 719-732

Schneider, Jesper Wiborg. / **Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations**. Proceedings of 17th International Conference on Science and Technology Indicators. editor / Éric Archambault ; Yves Gingras ; Vincent Larivière . 2012. pp. 719-732

@inproceedings{ea9624d021254376bb71d3f204c2d13e,

title = "Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations",

abstract = "In this paper we discuss and question the use of statistical significance tests in relation to university rankings as recently suggested. We outline the assumptions behind and interpretations of statistical significance tests and relate this to examples from the recent SCImago Institutions Ranking. By use of statistical power analyses and demonstration of effect sizes, we emphasize that importance of empirical findings lies in “differences that make a difference” and not statistical significance tests per se. Finally we discuss the crucial assumption of randomness and question the presumption that randomness is present in the university ranking data. We conclude that the application of statistical significance tests in relation to university rankings, as recently advocated, is problematic and can be misleading.",

keywords = "statistical significance tests, controversy, university rankings, effect sizes, statistcial power, scientometrics, indicators",

author = "Schneider, {Jesper Wiborg}",

note = "Paper presented at 17th International Conference on Science and Technology Indicators (STI), 5-8 September, 2012 in Montreal, Quebec, Canada.; International Conference on Science and Technology Indicators , STI ; Conference date: 05-09-2012 Through 08-09-2012",

year = "2012",

month = sep,

language = "English",

pages = "719--732",

editor = "{ Archambault}, {\'E}ric and Gingras, {Yves } and {Larivi{\`e}re }, {Vincent }",

booktitle = "Proceedings of 17th International Conference on Science and Technology Indicators",

}

TY - GEN

T1 - Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations

AU - Schneider, Jesper Wiborg

N1 - Conference code: 17th

PY - 2012/9

Y1 - 2012/9

N2 - In this paper we discuss and question the use of statistical significance tests in relation to university rankings as recently suggested. We outline the assumptions behind and interpretations of statistical significance tests and relate this to examples from the recent SCImago Institutions Ranking. By use of statistical power analyses and demonstration of effect sizes, we emphasize that importance of empirical findings lies in “differences that make a difference” and not statistical significance tests per se. Finally we discuss the crucial assumption of randomness and question the presumption that randomness is present in the university ranking data. We conclude that the application of statistical significance tests in relation to university rankings, as recently advocated, is problematic and can be misleading.

AB - In this paper we discuss and question the use of statistical significance tests in relation to university rankings as recently suggested. We outline the assumptions behind and interpretations of statistical significance tests and relate this to examples from the recent SCImago Institutions Ranking. By use of statistical power analyses and demonstration of effect sizes, we emphasize that importance of empirical findings lies in “differences that make a difference” and not statistical significance tests per se. Finally we discuss the crucial assumption of randomness and question the presumption that randomness is present in the university ranking data. We conclude that the application of statistical significance tests in relation to university rankings, as recently advocated, is problematic and can be misleading.

KW - statistical significance tests

KW - controversy

KW - university rankings

KW - effect sizes

KW - statistcial power

KW - scientometrics

KW - indicators

M3 - Article in proceedings

SP - 719

EP - 732

BT - Proceedings of 17th International Conference on Science and Technology Indicators

A2 - Archambault, Éric

A2 - Gingras, Yves

A2 - Larivière , Vincent

T2 - International Conference on Science and Technology Indicators

Y2 - 5 September 2012 through 8 September 2012

ER -

Department of Political Science

Aarhus BSS

Aarhus University

Bartholins Allé 7

DK–8000 Aarhus C

Denmark

Email: cfa@cfa.au.dk

Tel.: +45 8716 5238

See the location of the Centre (building 1331) on the map of Aarhus University.

CVR no: 31119103

P no: 1013137702

EAN no: 5798000419629

Budget code: 5321