The Danish Centre for Studies in Research and Research Policy

The quality landscape of economics: Beyond the Top Five and the journal hierarchy

Research output: Contribution to conferenceConference abstract for conferenceResearch

Standard

The quality landscape of economics: Beyond the Top Five and the journal hierarchy. / Hylmö , Anders; Steffy, Kody; Langfeldt, Liv et al.

2022. Abstract from Unsettling Research Quality, Oslo, Norway.

Research output: Contribution to conferenceConference abstract for conferenceResearch

Harvard

Hylmö , A, Steffy, K, Langfeldt, L & Thomas, DA 2022, 'The quality landscape of economics: Beyond the Top Five and the journal hierarchy', Unsettling Research Quality, Oslo, Norway, 21/09/2022 - 22/09/2022.

APA

Hylmö , A., Steffy, K., Langfeldt, L., & Thomas, D. A. (2022). The quality landscape of economics: Beyond the Top Five and the journal hierarchy. Abstract from Unsettling Research Quality, Oslo, Norway.

CBE

Hylmö A, Steffy K, Langfeldt L, Thomas DA. 2022. The quality landscape of economics: Beyond the Top Five and the journal hierarchy. Abstract from Unsettling Research Quality, Oslo, Norway.

MLA

Hylmö , Anders et al. The quality landscape of economics: Beyond the Top Five and the journal hierarchy. Unsettling Research Quality, 21 Sep 2022, Oslo, Norway, Conference abstract for conference, 2022. 2 p.

Vancouver

Hylmö A, Steffy K, Langfeldt L, Thomas DA. The quality landscape of economics: Beyond the Top Five and the journal hierarchy. 2022. Abstract from Unsettling Research Quality, Oslo, Norway.

Author

Hylmö , Anders ; Steffy, Kody ; Langfeldt, Liv et al. / The quality landscape of economics: Beyond the Top Five and the journal hierarchy. Abstract from Unsettling Research Quality, Oslo, Norway.2 p.

Bibtex

@conference{8d8e43a2d2ae40eba2903cda7688e1bb,
title = "The quality landscape of economics: Beyond the Top Five and the journal hierarchy",
abstract = "A growing number of studies evidence that quality notions and evaluative practices vary both between and within research fields, and co-exist with notions and practices emerging from research policy. The increasing availability and use of research metrics in recent decades across research fields, in research organisations, and in the policy-sphere has affected quality notions and evaluative practices. However, most studies on quality evaluation tend to focus on single evaluative practices or sites of evaluation, with few addressing how they are interrelated. This is important because it implies their aggregate effect on research practice may thus be different than that of single practices/sites considered in isolation.In this paper, we address the question of how this full spectrum of multiple co-existing quality notions and evaluative devices are configured for a particular discipline. We explore this through a case study of three top ranked economics departments in three Scandinavian countries. Analytically, we draw on sociology of (e)valuation, and introduce the concept of a quality landscape, understood as the total configuration of quality notions, sites of evaluation and evaluative devices researchers{\textquoteright} face in a particular discipline. We assume that configuration, relative weight and heterogeneity of quality practices are structurally distinct, in discipline- or field-specific ways accessible by empirical investigation. The quality landscape may be experienced differently depending on the researcher{\textquoteright}s vantage point (e.g. local context or career stage), and we may investigate the relative relevance of quality notions at different levels (e.g. research field, national policy), as well as tensions and ambivalence. We investigate the quality landscape of economics, with an empirical case study where different university and policy contexts allow us to discern similarity and variations in the quality landscape experienced. These variations anchor around notions of economics as a traditional discipline, known for its high degree of hierarchy, disciplinary boundaries, and epistemic consensus around theoretical and methodological core. Additionally, we know that economics journal rankings are a strong evaluative device in comparison to other fields. This provides a good case for demonstrating that, even under such conditions, tension, ambivalence and dynamic selection of quality notions and evaluative devices can exist in a field, as different researchers differently navigate notions of what is quality and good research. This point may seem trivial in other, more plural fields or disciplines, but is important to note for economics, which has been portrayed as dominated by monolithic devices, such as the {\textquoteright}top five{\textquoteright} journals.Our paper explores how a quality landscape appears for economists themselves. More specifically, we ask what notions of quality and evaluative devices are understood as important, by whom, at which sites of evaluation, and how researchers mobilise them, interrelated with social and epistemic conditions. Our data consists of 42 semi-structured interviews with researchers, and review of department-level background documents and publication data.We find that the economics journal hierarchy plays a seemingly inescapably landmark role in how economists perceive their quality landscape, but not as an uncontested device. It is applied in varying situations, including selection of literature, project topic selection, career advancement considerations, and tenure evaluations. Yet we find ambivalence about it as a quality signal, as an imperfect proxy, and how it empowers a disciplinary elite. Beyond such publication metrics the landscape also involves different devices, like letters of recommendation, in certain instances. Beyond the journal hierarchy, other research metrics like citations or Journal Impact Factor play a perhaps surprisingly limited role.We discuss how this quality landscape structure relates to social and epistemic conditions. It appears shaped by theoretical notions and social practices of market-like competition and resulting hierarchy, themselves grounded in economic theory. This may explain the journal hierarchy{\textquoteright}s landmark place, in fact driven more by endogenous than exogenous pressures, as sometimes assumed. The quality landscape approach also allows us to capture and explore a spectrum of coexisting, disparate evaluative devices as driven by similar notions of competition and hierarchy. We close with ideas about studying quality landscapes in other settings, as a way to move beyond more narrow perspectives on research quality as experienced and practiced by researchers in other fields and disciplines.",
author = "Anders Hylm{\"o} and Kody Steffy and Liv Langfeldt and Thomas, {Duncan Andrew}",
year = "2022",
month = may,
day = "6",
language = "English",
note = "Unsettling Research Quality ; Conference date: 21-09-2022 Through 22-09-2022",

}

RIS

TY - ABST

T1 - The quality landscape of economics: Beyond the Top Five and the journal hierarchy

AU - Hylmö , Anders

AU - Steffy, Kody

AU - Langfeldt, Liv

AU - Thomas, Duncan Andrew

PY - 2022/5/6

Y1 - 2022/5/6

N2 - A growing number of studies evidence that quality notions and evaluative practices vary both between and within research fields, and co-exist with notions and practices emerging from research policy. The increasing availability and use of research metrics in recent decades across research fields, in research organisations, and in the policy-sphere has affected quality notions and evaluative practices. However, most studies on quality evaluation tend to focus on single evaluative practices or sites of evaluation, with few addressing how they are interrelated. This is important because it implies their aggregate effect on research practice may thus be different than that of single practices/sites considered in isolation.In this paper, we address the question of how this full spectrum of multiple co-existing quality notions and evaluative devices are configured for a particular discipline. We explore this through a case study of three top ranked economics departments in three Scandinavian countries. Analytically, we draw on sociology of (e)valuation, and introduce the concept of a quality landscape, understood as the total configuration of quality notions, sites of evaluation and evaluative devices researchers’ face in a particular discipline. We assume that configuration, relative weight and heterogeneity of quality practices are structurally distinct, in discipline- or field-specific ways accessible by empirical investigation. The quality landscape may be experienced differently depending on the researcher’s vantage point (e.g. local context or career stage), and we may investigate the relative relevance of quality notions at different levels (e.g. research field, national policy), as well as tensions and ambivalence. We investigate the quality landscape of economics, with an empirical case study where different university and policy contexts allow us to discern similarity and variations in the quality landscape experienced. These variations anchor around notions of economics as a traditional discipline, known for its high degree of hierarchy, disciplinary boundaries, and epistemic consensus around theoretical and methodological core. Additionally, we know that economics journal rankings are a strong evaluative device in comparison to other fields. This provides a good case for demonstrating that, even under such conditions, tension, ambivalence and dynamic selection of quality notions and evaluative devices can exist in a field, as different researchers differently navigate notions of what is quality and good research. This point may seem trivial in other, more plural fields or disciplines, but is important to note for economics, which has been portrayed as dominated by monolithic devices, such as the ’top five’ journals.Our paper explores how a quality landscape appears for economists themselves. More specifically, we ask what notions of quality and evaluative devices are understood as important, by whom, at which sites of evaluation, and how researchers mobilise them, interrelated with social and epistemic conditions. Our data consists of 42 semi-structured interviews with researchers, and review of department-level background documents and publication data.We find that the economics journal hierarchy plays a seemingly inescapably landmark role in how economists perceive their quality landscape, but not as an uncontested device. It is applied in varying situations, including selection of literature, project topic selection, career advancement considerations, and tenure evaluations. Yet we find ambivalence about it as a quality signal, as an imperfect proxy, and how it empowers a disciplinary elite. Beyond such publication metrics the landscape also involves different devices, like letters of recommendation, in certain instances. Beyond the journal hierarchy, other research metrics like citations or Journal Impact Factor play a perhaps surprisingly limited role.We discuss how this quality landscape structure relates to social and epistemic conditions. It appears shaped by theoretical notions and social practices of market-like competition and resulting hierarchy, themselves grounded in economic theory. This may explain the journal hierarchy’s landmark place, in fact driven more by endogenous than exogenous pressures, as sometimes assumed. The quality landscape approach also allows us to capture and explore a spectrum of coexisting, disparate evaluative devices as driven by similar notions of competition and hierarchy. We close with ideas about studying quality landscapes in other settings, as a way to move beyond more narrow perspectives on research quality as experienced and practiced by researchers in other fields and disciplines.

AB - A growing number of studies evidence that quality notions and evaluative practices vary both between and within research fields, and co-exist with notions and practices emerging from research policy. The increasing availability and use of research metrics in recent decades across research fields, in research organisations, and in the policy-sphere has affected quality notions and evaluative practices. However, most studies on quality evaluation tend to focus on single evaluative practices or sites of evaluation, with few addressing how they are interrelated. This is important because it implies their aggregate effect on research practice may thus be different than that of single practices/sites considered in isolation.In this paper, we address the question of how this full spectrum of multiple co-existing quality notions and evaluative devices are configured for a particular discipline. We explore this through a case study of three top ranked economics departments in three Scandinavian countries. Analytically, we draw on sociology of (e)valuation, and introduce the concept of a quality landscape, understood as the total configuration of quality notions, sites of evaluation and evaluative devices researchers’ face in a particular discipline. We assume that configuration, relative weight and heterogeneity of quality practices are structurally distinct, in discipline- or field-specific ways accessible by empirical investigation. The quality landscape may be experienced differently depending on the researcher’s vantage point (e.g. local context or career stage), and we may investigate the relative relevance of quality notions at different levels (e.g. research field, national policy), as well as tensions and ambivalence. We investigate the quality landscape of economics, with an empirical case study where different university and policy contexts allow us to discern similarity and variations in the quality landscape experienced. These variations anchor around notions of economics as a traditional discipline, known for its high degree of hierarchy, disciplinary boundaries, and epistemic consensus around theoretical and methodological core. Additionally, we know that economics journal rankings are a strong evaluative device in comparison to other fields. This provides a good case for demonstrating that, even under such conditions, tension, ambivalence and dynamic selection of quality notions and evaluative devices can exist in a field, as different researchers differently navigate notions of what is quality and good research. This point may seem trivial in other, more plural fields or disciplines, but is important to note for economics, which has been portrayed as dominated by monolithic devices, such as the ’top five’ journals.Our paper explores how a quality landscape appears for economists themselves. More specifically, we ask what notions of quality and evaluative devices are understood as important, by whom, at which sites of evaluation, and how researchers mobilise them, interrelated with social and epistemic conditions. Our data consists of 42 semi-structured interviews with researchers, and review of department-level background documents and publication data.We find that the economics journal hierarchy plays a seemingly inescapably landmark role in how economists perceive their quality landscape, but not as an uncontested device. It is applied in varying situations, including selection of literature, project topic selection, career advancement considerations, and tenure evaluations. Yet we find ambivalence about it as a quality signal, as an imperfect proxy, and how it empowers a disciplinary elite. Beyond such publication metrics the landscape also involves different devices, like letters of recommendation, in certain instances. Beyond the journal hierarchy, other research metrics like citations or Journal Impact Factor play a perhaps surprisingly limited role.We discuss how this quality landscape structure relates to social and epistemic conditions. It appears shaped by theoretical notions and social practices of market-like competition and resulting hierarchy, themselves grounded in economic theory. This may explain the journal hierarchy’s landmark place, in fact driven more by endogenous than exogenous pressures, as sometimes assumed. The quality landscape approach also allows us to capture and explore a spectrum of coexisting, disparate evaluative devices as driven by similar notions of competition and hierarchy. We close with ideas about studying quality landscapes in other settings, as a way to move beyond more narrow perspectives on research quality as experienced and practiced by researchers in other fields and disciplines.

M3 - Conference abstract for conference

T2 - Unsettling Research Quality

Y2 - 21 September 2022 through 22 September 2022

ER -