Examining the generalizability of research findings from archival data

Andrew Delios; Elena Giulia Clemente; Tao Wu; Hongbin Tan; Yong Wang; Michael Gordon; Domenico Viganola; Zhaowei Chen; Anna Dreber; Magnus Johannesson; Thomas Pfeiffer; Generalizability Tests Forecasting Collaboration; Eric Luis Uhlmann

doi:10.1073/pnas.2120377119

Examining the generalizability of research findings from archival data

Andrew Delios^*, Elena Giulia Clemente, Tao Wu, Hongbin Tan, Yong Wang, Michael Gordon, Domenico Viganola, Zhaowei Chen, Anna Dreber, Magnus Johannesson, Thomas Pfeiffer, Generalizability Tests Forecasting Collaboration, Eric Luis Uhlmann^*

^*Corresponding author for this work

Department of Management

Research output: Contribution to journal/Conference contribution in journal/Contribution to newspaper › Journal article › Research › peer-review

21 Citations (Scopus)

169 Downloads (Pure)

Abstract

This initiative examined systematically the extent to which a large set of archival
research findings generalizes across contexts. We repeated the key analyses for 29 original
strategic management effects in the same context (direct reproduction) as well as in 52 novel
time periods and geographies. 45% of the reproductions returned results matching the
original reports, together with 55% of tests in different spans of years and 40% of tests in
novel geographies. Some original findings were associated with multiple new tests.
Reproducibility was the best predictor of generalizability — for the findings that proved
directly reproducible, 84% emerged in other available time periods and 57% in other
geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a
forecasting survey, independent scientists were able to anticipate which effects would find
support in tests in new samples.

Original language	English
Article number	e2120377119
Journal	Proceedings of the National Academy of Sciences (PNAS)
Volume	119
Issue	30
Number of pages	9
ISSN	0027-8424
DOIs	https://doi.org/10.1073/pnas.2120377119
Publication status	Published - 26 Jul 2022

Keywords

Research reliability, generalizability, archival data, reproducibility, context sensitivity

Access to Document

10.1073/pnas.2120377119

Generalizability Tests.Manuscript and Supplements

Cite this

Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., Pfeiffer, T., Generalizability Tests Forecasting Collaboration, & Uhlmann, E. L. (2022). Examining the generalizability of research findings from archival data. Proceedings of the National Academy of Sciences (PNAS), 119(30), Article e2120377119. https://doi.org/10.1073/pnas.2120377119

@article{2c78e0bc4cd14c978b64a4324e2e07f5,

title = "Examining the generalizability of research findings from archival data",

abstract = "This initiative examined systematically the extent to which a large set of archivalresearch findings generalizes across contexts. We repeated the key analyses for 29 originalstrategic management effects in the same context (direct reproduction) as well as in 52 noveltime periods and geographies. 45% of the reproductions returned results matching theoriginal reports, together with 55% of tests in different spans of years and 40% of tests innovel geographies. Some original findings were associated with multiple new tests.Reproducibility was the best predictor of generalizability — for the findings that proveddirectly reproducible, 84% emerged in other available time periods and 57% in othergeographies. Overall, only limited empirical evidence emerged for context sensitivity. In aforecasting survey, independent scientists were able to anticipate which effects would findsupport in tests in new samples.",

keywords = "Research reliability, generalizability, archival data, reproducibility, context sensitivity",

author = "Andrew Delios and Clemente, {Elena Giulia} and Tao Wu and Hongbin Tan and Yong Wang and Michael Gordon and Domenico Viganola and Zhaowei Chen and Anna Dreber and Magnus Johannesson and Thomas Pfeiffer and {Generalizability Tests Forecasting Collaboration} and Uhlmann, {Eric Luis} and Rahul Anand and Elb{\ae}k, {Christian Truelsen}",

year = "2022",

month = jul,

day = "26",

doi = "10.1073/pnas.2120377119",

language = "English",

volume = "119",

journal = "Proceedings of the National Academy of Sciences (PNAS)",

issn = "0027-8424",

publisher = "National Academy of Sciences",

number = "30",

}

Delios, A, Clemente, EG, Wu, T, Tan, H, Wang, Y, Gordon, M, Viganola, D, Chen, Z, Dreber, A, Johannesson, M, Pfeiffer, T, Generalizability Tests Forecasting Collaboration & Uhlmann, EL 2022, 'Examining the generalizability of research findings from archival data', Proceedings of the National Academy of Sciences (PNAS), vol. 119, no. 30, e2120377119. https://doi.org/10.1073/pnas.2120377119

Examining the generalizability of research findings from archival data. / Delios, Andrew; Clemente, Elena Giulia; Wu, Tao et al.
In: Proceedings of the National Academy of Sciences (PNAS), Vol. 119, No. 30, e2120377119, 26.07.2022.

Research output: Contribution to journal/Conference contribution in journal/Contribution to newspaper › Journal article › Research › peer-review

TY - JOUR

T1 - Examining the generalizability of research findings from archival data

AU - Delios, Andrew

AU - Clemente, Elena Giulia

AU - Wu, Tao

AU - Tan, Hongbin

AU - Wang, Yong

AU - Gordon, Michael

AU - Viganola, Domenico

AU - Chen, Zhaowei

AU - Dreber, Anna

AU - Johannesson, Magnus

AU - Pfeiffer, Thomas

AU - Generalizability Tests Forecasting Collaboration

AU - Uhlmann, Eric Luis

AU - Anand, Rahul

AU - Elbæk, Christian Truelsen

PY - 2022/7/26

Y1 - 2022/7/26

N2 - This initiative examined systematically the extent to which a large set of archivalresearch findings generalizes across contexts. We repeated the key analyses for 29 originalstrategic management effects in the same context (direct reproduction) as well as in 52 noveltime periods and geographies. 45% of the reproductions returned results matching theoriginal reports, together with 55% of tests in different spans of years and 40% of tests innovel geographies. Some original findings were associated with multiple new tests.Reproducibility was the best predictor of generalizability — for the findings that proveddirectly reproducible, 84% emerged in other available time periods and 57% in othergeographies. Overall, only limited empirical evidence emerged for context sensitivity. In aforecasting survey, independent scientists were able to anticipate which effects would findsupport in tests in new samples.

AB - This initiative examined systematically the extent to which a large set of archivalresearch findings generalizes across contexts. We repeated the key analyses for 29 originalstrategic management effects in the same context (direct reproduction) as well as in 52 noveltime periods and geographies. 45% of the reproductions returned results matching theoriginal reports, together with 55% of tests in different spans of years and 40% of tests innovel geographies. Some original findings were associated with multiple new tests.Reproducibility was the best predictor of generalizability — for the findings that proveddirectly reproducible, 84% emerged in other available time periods and 57% in othergeographies. Overall, only limited empirical evidence emerged for context sensitivity. In aforecasting survey, independent scientists were able to anticipate which effects would findsupport in tests in new samples.

KW - Research reliability, generalizability, archival data, reproducibility, context sensitivity

UR - http://www.scopus.com/inward/record.url?scp=85134491120&partnerID=8YFLogxK

U2 - 10.1073/pnas.2120377119

DO - 10.1073/pnas.2120377119

M3 - Journal article

C2 - 35858443

SN - 0027-8424

VL - 119

JO - Proceedings of the National Academy of Sciences (PNAS)

JF - Proceedings of the National Academy of Sciences (PNAS)

IS - 30

M1 - e2120377119

ER -

Examining the generalizability of research findings from archival data

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this