Aarhus Universitets segl

No Golden Path: A Cautionary Tale of Quality and Biases

Publikation: KonferencebidragKonferenceabstrakt til konferenceForskningpeer review

Standard

No Golden Path : A Cautionary Tale of Quality and Biases. / Lassen, Ida Marie S.; Bizzoni, Yuri; Peura, Telma et al.

2022. Abstract fra DARIAH Annual Event 2022, Athen, Grækenland.

Publikation: KonferencebidragKonferenceabstrakt til konferenceForskningpeer review

Harvard

APA

CBE

Lassen IMS, Bizzoni Y, Peura T, Thomsen MR, Nielbo KL. 2022. No Golden Path: A Cautionary Tale of Quality and Biases. Abstract fra DARIAH Annual Event 2022, Athen, Grækenland.

MLA

Lassen, Ida Marie S. et al. No Golden Path: A Cautionary Tale of Quality and Biases. DARIAH Annual Event 2022, 31 maj 2022, Athen, Grækenland, Konferenceabstrakt til konference, 2022.

Vancouver

Lassen IMS, Bizzoni Y, Peura T, Thomsen MR, Nielbo KL. No Golden Path: A Cautionary Tale of Quality and Biases. 2022. Abstract fra DARIAH Annual Event 2022, Athen, Grækenland.

Author

Lassen, Ida Marie S. ; Bizzoni, Yuri ; Peura, Telma et al. / No Golden Path : A Cautionary Tale of Quality and Biases. Abstract fra DARIAH Annual Event 2022, Athen, Grækenland.

Bibtex

@conference{c1b98db653554158be97bbea153312d1,
title = "No Golden Path: A Cautionary Tale of Quality and Biases",
abstract = "Narrative organization of information ties together storytelling in its many modalities. One archetypal expression of narratives we find in literary fiction. In this paper, we approach the elements of a successful narrative, and by extension storytelling, from the perspective of computational narratology. We are specifically interested in how to identify {\textquoteright}a good story{\textquoteright} and navigate in two dimensions of literary success: {\textquoteleft}Extrinsic{\textquoteright} and {\textquoteleft}intrinsic{\textquoteright}. We investigate the association between the self-similarity of a sentiment story arc and review assessment of a literary work. In doing so, we direct our attention to known and unknown biases in the suggested dimensions of success to avoid reinforcing existing unwanted structures in our exploration of successful narratives.Quality assessment of literature is not a simple inquiry and is complicated by various factors. Literature is a complex linguistic phenomenon that conveys information indirectly, and readers have different aesthetic preferences, and, in general, there is a lack of robust scientific instruments for measuring literary quality. One noisy, but ecologically valid measure is quantitative {\textquoteright}reader reviews.{\textquoteright} On this account, a narrative is successful if readers rate it high. Such an {\textquoteright}extrinsic{\textquoteright} success criterion is tempting because it is relatively easy to access, reflects readers{\textquoteright} preferences in a natural setting, and its standardization appears trivial. A criterion that relies on reviews is however prone to several well-known biases, for instance, grading disparities between gender [1], ethnicity and race [2], which point to fairness challenges in classification of real-world data [3]. Instead of merely relying upon review annotation of the success of a story, we suggest paying attention to the inner structure of a story, the {\textquoteright}intrinsic success{\textquoteright}.A recent theoretical paper has suggested that the affective coherence of a story, that is, the self-similarity of a sentiment story arc, functions as an index of a narrative{\textquoteright}s intrinsic success [4]. A complementary empirical study has shown that affective coherence can detect canonical literature [5].While the use of computational narratology may seem compelling to minimize demographic disparities introduced by extrinsic success, it introduces less apparent and unknown biases. Genre, for instance, impacts a story arc and shows complex interactions with psychological propensities, aesthetic evaluation, and gender [1]. In this work, we examine the association between the internal sentiment structure of a work and known biases. Socio-cultural norms may also play an important role in introducing unknown biases even at the methodological level, such as representational biases of reader types.In sum, there is no golden path to identify successful storytelling, that is, no single path that optimizes both quality assessment and bias response. Instead of relying on single dimensions of success, either compelling computational approaches or accessible standardization, we suggest a deliberate combination of dimensions and approaches which includes choices about bias acceptance. We see a multitude of possible trajectories, each of which implies different choices of known and unknown biases.",
keywords = "computational narratology, bias analysis, literary quality assessment",
author = "Lassen, {Ida Marie S.} and Yuri Bizzoni and Telma Peura and Thomsen, {Mads Rosendahl} and Nielbo, {Kristoffer Laigaard}",
year = "2022",
language = "English",
note = "DARIAH Annual Event 2022 : Storytelling ; Conference date: 31-05-2022 Through 03-06-2022",
url = "https://annualevent.dariah.eu/",

}

RIS

TY - ABST

T1 - No Golden Path

T2 - DARIAH Annual Event 2022

AU - Lassen, Ida Marie S.

AU - Bizzoni, Yuri

AU - Peura, Telma

AU - Thomsen, Mads Rosendahl

AU - Nielbo, Kristoffer Laigaard

PY - 2022

Y1 - 2022

N2 - Narrative organization of information ties together storytelling in its many modalities. One archetypal expression of narratives we find in literary fiction. In this paper, we approach the elements of a successful narrative, and by extension storytelling, from the perspective of computational narratology. We are specifically interested in how to identify ’a good story’ and navigate in two dimensions of literary success: ‘Extrinsic’ and ‘intrinsic’. We investigate the association between the self-similarity of a sentiment story arc and review assessment of a literary work. In doing so, we direct our attention to known and unknown biases in the suggested dimensions of success to avoid reinforcing existing unwanted structures in our exploration of successful narratives.Quality assessment of literature is not a simple inquiry and is complicated by various factors. Literature is a complex linguistic phenomenon that conveys information indirectly, and readers have different aesthetic preferences, and, in general, there is a lack of robust scientific instruments for measuring literary quality. One noisy, but ecologically valid measure is quantitative ’reader reviews.’ On this account, a narrative is successful if readers rate it high. Such an ’extrinsic’ success criterion is tempting because it is relatively easy to access, reflects readers’ preferences in a natural setting, and its standardization appears trivial. A criterion that relies on reviews is however prone to several well-known biases, for instance, grading disparities between gender [1], ethnicity and race [2], which point to fairness challenges in classification of real-world data [3]. Instead of merely relying upon review annotation of the success of a story, we suggest paying attention to the inner structure of a story, the ’intrinsic success’.A recent theoretical paper has suggested that the affective coherence of a story, that is, the self-similarity of a sentiment story arc, functions as an index of a narrative’s intrinsic success [4]. A complementary empirical study has shown that affective coherence can detect canonical literature [5].While the use of computational narratology may seem compelling to minimize demographic disparities introduced by extrinsic success, it introduces less apparent and unknown biases. Genre, for instance, impacts a story arc and shows complex interactions with psychological propensities, aesthetic evaluation, and gender [1]. In this work, we examine the association between the internal sentiment structure of a work and known biases. Socio-cultural norms may also play an important role in introducing unknown biases even at the methodological level, such as representational biases of reader types.In sum, there is no golden path to identify successful storytelling, that is, no single path that optimizes both quality assessment and bias response. Instead of relying on single dimensions of success, either compelling computational approaches or accessible standardization, we suggest a deliberate combination of dimensions and approaches which includes choices about bias acceptance. We see a multitude of possible trajectories, each of which implies different choices of known and unknown biases.

AB - Narrative organization of information ties together storytelling in its many modalities. One archetypal expression of narratives we find in literary fiction. In this paper, we approach the elements of a successful narrative, and by extension storytelling, from the perspective of computational narratology. We are specifically interested in how to identify ’a good story’ and navigate in two dimensions of literary success: ‘Extrinsic’ and ‘intrinsic’. We investigate the association between the self-similarity of a sentiment story arc and review assessment of a literary work. In doing so, we direct our attention to known and unknown biases in the suggested dimensions of success to avoid reinforcing existing unwanted structures in our exploration of successful narratives.Quality assessment of literature is not a simple inquiry and is complicated by various factors. Literature is a complex linguistic phenomenon that conveys information indirectly, and readers have different aesthetic preferences, and, in general, there is a lack of robust scientific instruments for measuring literary quality. One noisy, but ecologically valid measure is quantitative ’reader reviews.’ On this account, a narrative is successful if readers rate it high. Such an ’extrinsic’ success criterion is tempting because it is relatively easy to access, reflects readers’ preferences in a natural setting, and its standardization appears trivial. A criterion that relies on reviews is however prone to several well-known biases, for instance, grading disparities between gender [1], ethnicity and race [2], which point to fairness challenges in classification of real-world data [3]. Instead of merely relying upon review annotation of the success of a story, we suggest paying attention to the inner structure of a story, the ’intrinsic success’.A recent theoretical paper has suggested that the affective coherence of a story, that is, the self-similarity of a sentiment story arc, functions as an index of a narrative’s intrinsic success [4]. A complementary empirical study has shown that affective coherence can detect canonical literature [5].While the use of computational narratology may seem compelling to minimize demographic disparities introduced by extrinsic success, it introduces less apparent and unknown biases. Genre, for instance, impacts a story arc and shows complex interactions with psychological propensities, aesthetic evaluation, and gender [1]. In this work, we examine the association between the internal sentiment structure of a work and known biases. Socio-cultural norms may also play an important role in introducing unknown biases even at the methodological level, such as representational biases of reader types.In sum, there is no golden path to identify successful storytelling, that is, no single path that optimizes both quality assessment and bias response. Instead of relying on single dimensions of success, either compelling computational approaches or accessible standardization, we suggest a deliberate combination of dimensions and approaches which includes choices about bias acceptance. We see a multitude of possible trajectories, each of which implies different choices of known and unknown biases.

KW - computational narratology

KW - bias analysis

KW - literary quality assessment

M3 - Conference abstract for conference

Y2 - 31 May 2022 through 3 June 2022

ER -