Is the Lack of Inter-referee Reliability a Threat to the Effectiveness of Peer Review in Science?

TY - JOUR

T1 - Is the Lack of Inter-referee Reliability a Threat to the Effectiveness of Peer Review in Science?

AU - Wray, K. Brad

PY - 2025/4/1

Y1 - 2025/4/1

N2 - Many have criticized the peer-review system employed in scientific publication because of the apparent lack of inter-referee reliability. It is often remarked that if referees cannot agree about the quality of a scientific manuscript, then the selection of manuscripts for publication is arbitrary. Some critics have even called for abolishing pre-publication peer-review. I argue that too much emphasis has been put on inter-referee reliability. As such, a lack of inter-referee reliability in the current peer-review system is not damning for the system. I argue that peer review is effective at finding mistakes.

AB - Many have criticized the peer-review system employed in scientific publication because of the apparent lack of inter-referee reliability. It is often remarked that if referees cannot agree about the quality of a scientific manuscript, then the selection of manuscripts for publication is arbitrary. Some critics have even called for abolishing pre-publication peer-review. I argue that too much emphasis has been put on inter-referee reliability. As such, a lack of inter-referee reliability in the current peer-review system is not damning for the system. I argue that peer review is effective at finding mistakes.

KW - peer review

KW - reliability

KW - fairness

KW - validity

KW - scientific publication

UR - https://direct.mit.edu/posc/article-abstract/doi/10.1162/posc_a_00639/127782/Is-the-Lack-of-Inter-referee-Reliability-a-Threat?redirectedFrom=fulltext

M3 - Journal article

SN - 1063-6145

VL - 33

SP - 202

EP - 224

JO - Perspectives on Science

JF - Perspectives on Science

IS - 2

M1 - https://doi.org/10.1162/posc_a_00639

ER -