Dansk Center for Forskningsanalyse

Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations

Publikation: Bidrag til bog/antologi/rapport/proceedingKonferencebidrag i proceedingsForskningpeer review

Standard

Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations. / Schneider, Jesper Wiborg.

Proceedings of 17th International Conference on Science and Technology Indicators. red. / Éric Archambault; Yves Gingras; Vincent Larivière . 2012. s. 719-732.

Publikation: Bidrag til bog/antologi/rapport/proceedingKonferencebidrag i proceedingsForskningpeer review

Harvard

Schneider, JW 2012, Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations. i É Archambault, Y Gingras & V Larivière (red), Proceedings of 17th International Conference on Science and Technology Indicators. s. 719-732, International Conference on Science and Technology Indicators , Montreal, Quebec, Canada, 05/09/2012. <http://sticonference.org/Proceedings/vol2/Schneider_Testing_719.pdf>

APA

CBE

Schneider JW. 2012. Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations. Archambault É, Gingras Y, Larivière V, red. I Proceedings of 17th International Conference on Science and Technology Indicators. s. 719-732.

MLA

Schneider, Jesper Wiborg "Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations"., Archambault, Éric Gingras, Yves Larivière , Vincent (red.). Proceedings of 17th International Conference on Science and Technology Indicators. 2012, 719-732.

Vancouver

Schneider JW. Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations. I Archambault É, Gingras Y, Larivière V, red., Proceedings of 17th International Conference on Science and Technology Indicators. 2012. s. 719-732

Author

Schneider, Jesper Wiborg. / Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations. Proceedings of 17th International Conference on Science and Technology Indicators. red. / Éric Archambault ; Yves Gingras ; Vincent Larivière . 2012. s. 719-732

Bibtex

@inproceedings{ea9624d021254376bb71d3f204c2d13e,
title = "Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations",
abstract = "In this paper we discuss and question the use of statistical significance tests in relation to university rankings as recently suggested. We outline the assumptions behind and interpretations of statistical significance tests and relate this to examples from the recent SCImago Institutions Ranking. By use of statistical power analyses and demonstration of effect sizes, we emphasize that importance of empirical findings lies in “differences that make a difference” and not statistical significance tests per se. Finally we discuss the crucial assumption of randomness and question the presumption that randomness is present in the university ranking data. We conclude that the application of statistical significance tests in relation to university rankings, as recently advocated, is problematic and can be misleading.",
keywords = "statistical significance tests, controversy, university rankings, effect sizes, statistcial power, scientometrics, indicators",
author = "Schneider, {Jesper Wiborg}",
note = "Paper presented at 17th International Conference on Science and Technology Indicators (STI), 5-8 September, 2012 in Montreal, Quebec, Canada.; International Conference on Science and Technology Indicators , STI ; Conference date: 05-09-2012 Through 08-09-2012",
year = "2012",
month = sep,
language = "English",
pages = "719--732",
editor = "{ Archambault}, {\'E}ric and Gingras, {Yves } and {Larivi{\`e}re }, {Vincent }",
booktitle = "Proceedings of 17th International Conference on Science and Technology Indicators",

}

RIS

TY - GEN

T1 - Testing University Rankings Statistically: Why this Perhaps is not such a Good Idea after All. Some Reflections on Statistical Power, Effect Size, Random Sampling and Imaginary Populations

AU - Schneider, Jesper Wiborg

N1 - Conference code: 17th

PY - 2012/9

Y1 - 2012/9

N2 - In this paper we discuss and question the use of statistical significance tests in relation to university rankings as recently suggested. We outline the assumptions behind and interpretations of statistical significance tests and relate this to examples from the recent SCImago Institutions Ranking. By use of statistical power analyses and demonstration of effect sizes, we emphasize that importance of empirical findings lies in “differences that make a difference” and not statistical significance tests per se. Finally we discuss the crucial assumption of randomness and question the presumption that randomness is present in the university ranking data. We conclude that the application of statistical significance tests in relation to university rankings, as recently advocated, is problematic and can be misleading.

AB - In this paper we discuss and question the use of statistical significance tests in relation to university rankings as recently suggested. We outline the assumptions behind and interpretations of statistical significance tests and relate this to examples from the recent SCImago Institutions Ranking. By use of statistical power analyses and demonstration of effect sizes, we emphasize that importance of empirical findings lies in “differences that make a difference” and not statistical significance tests per se. Finally we discuss the crucial assumption of randomness and question the presumption that randomness is present in the university ranking data. We conclude that the application of statistical significance tests in relation to university rankings, as recently advocated, is problematic and can be misleading.

KW - statistical significance tests

KW - controversy

KW - university rankings

KW - effect sizes

KW - statistcial power

KW - scientometrics

KW - indicators

M3 - Article in proceedings

SP - 719

EP - 732

BT - Proceedings of 17th International Conference on Science and Technology Indicators

A2 - Archambault, Éric

A2 - Gingras, Yves

A2 - Larivière , Vincent

T2 - International Conference on Science and Technology Indicators

Y2 - 5 September 2012 through 8 September 2012

ER -