Abstract
A growing number of studies evidence that quality notions and evaluative practices vary both between and within research fields, and co-exist with notions and practices emerging from research policy. The increasing availability and use of research metrics in recent decades across research fields, in research organisations, and in the policy-sphere has affected quality notions and evaluative practices. However, most studies on quality evaluation tend to focus on single evaluative practices or sites of evaluation, with few addressing how they are interrelated. This is important because it implies their aggregate effect on research practice may thus be different than that of single practices/sites considered in isolation.
In this paper, we address the question of how this full spectrum of multiple co-existing quality notions and evaluative devices are configured for a particular discipline. We explore this through a case study of three top ranked economics departments in three Scandinavian countries. Analytically, we draw on sociology of (e)valuation, and introduce the concept of a quality landscape, understood as the total configuration of quality notions, sites of evaluation and evaluative devices researchers’ face in a particular discipline. We assume that configuration, relative weight and heterogeneity of quality practices are structurally distinct, in discipline- or field-specific ways accessible by empirical investigation. The quality landscape may be experienced differently depending on the researcher’s vantage point (e.g. local context or career stage), and we may investigate the relative relevance of quality notions at different levels (e.g. research field, national policy), as well as tensions and ambivalence.
We investigate the quality landscape of economics, with an empirical case study where different university and policy contexts allow us to discern similarity and variations in the quality landscape experienced. These variations anchor around notions of economics as a traditional discipline, known for its high degree of hierarchy, disciplinary boundaries, and epistemic consensus around theoretical and methodological core. Additionally, we know that economics journal rankings are a strong evaluative device in comparison to other fields. This provides a good case for demonstrating that, even under such conditions, tension, ambivalence and dynamic selection of quality notions and evaluative devices can exist in a field, as different researchers differently navigate notions of what is quality and good research. This point may seem trivial in other, more plural fields or disciplines, but is important to note for economics, which has been portrayed as dominated by monolithic devices, such as the ’top five’ journals.
Our paper explores how a quality landscape appears for economists themselves. More specifically, we ask what notions of quality and evaluative devices are understood as important, by whom, at which sites of evaluation, and how researchers mobilise them, interrelated with social and epistemic conditions. Our data consists of 42 semi-structured interviews with researchers, and review of department-level background documents and publication data.
We find that the economics journal hierarchy plays a seemingly inescapably landmark role in how economists perceive their quality landscape, but not as an uncontested device. It is applied in varying situations, including selection of literature, project topic selection, career advancement considerations, and tenure evaluations. Yet we find ambivalence about it as a quality signal, as an imperfect proxy, and how it empowers a disciplinary elite. Beyond such publication metrics the landscape also involves different devices, like letters of recommendation, in certain instances. Beyond the journal hierarchy, other research metrics like citations or Journal Impact Factor play a perhaps surprisingly limited role.
We discuss how this quality landscape structure relates to social and epistemic conditions. It appears shaped by theoretical notions and social practices of market-like competition and resulting hierarchy, themselves grounded in economic theory. This may explain the journal hierarchy’s landmark place, in fact driven more by endogenous than exogenous pressures, as sometimes assumed. The quality landscape approach also allows us to capture and explore a spectrum of coexisting, disparate evaluative devices as driven by similar notions of competition and hierarchy. We close with ideas about studying quality landscapes in other settings, as a way to move beyond more narrow perspectives on research quality as experienced and practiced by researchers in other fields and disciplines.
In this paper, we address the question of how this full spectrum of multiple co-existing quality notions and evaluative devices are configured for a particular discipline. We explore this through a case study of three top ranked economics departments in three Scandinavian countries. Analytically, we draw on sociology of (e)valuation, and introduce the concept of a quality landscape, understood as the total configuration of quality notions, sites of evaluation and evaluative devices researchers’ face in a particular discipline. We assume that configuration, relative weight and heterogeneity of quality practices are structurally distinct, in discipline- or field-specific ways accessible by empirical investigation. The quality landscape may be experienced differently depending on the researcher’s vantage point (e.g. local context or career stage), and we may investigate the relative relevance of quality notions at different levels (e.g. research field, national policy), as well as tensions and ambivalence.
We investigate the quality landscape of economics, with an empirical case study where different university and policy contexts allow us to discern similarity and variations in the quality landscape experienced. These variations anchor around notions of economics as a traditional discipline, known for its high degree of hierarchy, disciplinary boundaries, and epistemic consensus around theoretical and methodological core. Additionally, we know that economics journal rankings are a strong evaluative device in comparison to other fields. This provides a good case for demonstrating that, even under such conditions, tension, ambivalence and dynamic selection of quality notions and evaluative devices can exist in a field, as different researchers differently navigate notions of what is quality and good research. This point may seem trivial in other, more plural fields or disciplines, but is important to note for economics, which has been portrayed as dominated by monolithic devices, such as the ’top five’ journals.
Our paper explores how a quality landscape appears for economists themselves. More specifically, we ask what notions of quality and evaluative devices are understood as important, by whom, at which sites of evaluation, and how researchers mobilise them, interrelated with social and epistemic conditions. Our data consists of 42 semi-structured interviews with researchers, and review of department-level background documents and publication data.
We find that the economics journal hierarchy plays a seemingly inescapably landmark role in how economists perceive their quality landscape, but not as an uncontested device. It is applied in varying situations, including selection of literature, project topic selection, career advancement considerations, and tenure evaluations. Yet we find ambivalence about it as a quality signal, as an imperfect proxy, and how it empowers a disciplinary elite. Beyond such publication metrics the landscape also involves different devices, like letters of recommendation, in certain instances. Beyond the journal hierarchy, other research metrics like citations or Journal Impact Factor play a perhaps surprisingly limited role.
We discuss how this quality landscape structure relates to social and epistemic conditions. It appears shaped by theoretical notions and social practices of market-like competition and resulting hierarchy, themselves grounded in economic theory. This may explain the journal hierarchy’s landmark place, in fact driven more by endogenous than exogenous pressures, as sometimes assumed. The quality landscape approach also allows us to capture and explore a spectrum of coexisting, disparate evaluative devices as driven by similar notions of competition and hierarchy. We close with ideas about studying quality landscapes in other settings, as a way to move beyond more narrow perspectives on research quality as experienced and practiced by researchers in other fields and disciplines.
Originalsprog | Engelsk |
---|---|
Publikationsdato | sep. 2022 |
Antal sider | 2 |
Status | Udgivet - sep. 2022 |
Begivenhed | Unsettling Research Quality - NIFU, Oslo, Norge Varighed: 21 sep. 2022 → 22 sep. 2022 |
Workshop
Workshop | Unsettling Research Quality |
---|---|
Lokation | NIFU |
Land/Område | Norge |
By | Oslo |
Periode | 21/09/2022 → 22/09/2022 |