Automatic proficiency scoring for early-stage writing

TY - JOUR

T1 - Automatic proficiency scoring for early-stage writing

AU - Riis Andersen, Michael

AU - Kabel, Kristine

AU - Bremholm, Jesper

AU - Bundsgaard, Jeppe

AU - Hansen, Lars Kai

PY - 2023/10

Y1 - 2023/10

N2 - In this work, we study the feasibility of using machine learning and natural language processing methods for assessing writing proficiency in Danish with respect to text construction, sentence construction, and use of modifiers. Our work is based on the analytical framework for scoring early writing proposed by Kabel et al. (2022), where each text is first annotated by a human expert according to a predefined coding scheme and subsequently scored using statistical Rasch modelling (Rasch, 1960). We investigate two different strategies for estimating these scores automatically: 1) we propose a system for identifying the central linguistic features automatically mimicking the role of the human experts and 2) we train state-of-the-art discriminative machine learning models to predict the proficiency scores directly from the texts. We conduct a number of experiments to evaluate and compare the two approaches. Our results show strong and statistically significant correlations between the scores generated using the automatic system and scores based on human experts. We also estimate and report the reliability of the individual linguistic features in the automatic annotation system. Finally, we also propose and evaluate an extension of the statistical model, which allows the model to compensate for potential systematic errors in the automatic annotations. The article thereby contributes to the area of automated essay scoring (AES) and shows that it is possible to provide teachers with automated valid and reliable knowledge about the development of their students' writing competences, which they can use in their feedback to students.

AB - In this work, we study the feasibility of using machine learning and natural language processing methods for assessing writing proficiency in Danish with respect to text construction, sentence construction, and use of modifiers. Our work is based on the analytical framework for scoring early writing proposed by Kabel et al. (2022), where each text is first annotated by a human expert according to a predefined coding scheme and subsequently scored using statistical Rasch modelling (Rasch, 1960). We investigate two different strategies for estimating these scores automatically: 1) we propose a system for identifying the central linguistic features automatically mimicking the role of the human experts and 2) we train state-of-the-art discriminative machine learning models to predict the proficiency scores directly from the texts. We conduct a number of experiments to evaluate and compare the two approaches. Our results show strong and statistically significant correlations between the scores generated using the automatic system and scores based on human experts. We also estimate and report the reliability of the individual linguistic features in the automatic annotation system. Finally, we also propose and evaluate an extension of the statistical model, which allows the model to compensate for potential systematic errors in the automatic annotations. The article thereby contributes to the area of automated essay scoring (AES) and shows that it is possible to provide teachers with automated valid and reliable knowledge about the development of their students' writing competences, which they can use in their feedback to students.

KW - Early-stage literacy

KW - Machine learning

KW - Natural language processing

KW - Rasch models

KW - Writing proficiency

KW - Automatic scoring

KW - Danish

KW - Low-resource languages

KW - Automatic scoring

KW - Danish

KW - Early-stage literacy

KW - Low-resource languages

KW - Machine learning

KW - Natural language processing

KW - Rasch models

KW - Writing proficiency

UR - http://www.scopus.com/inward/record.url?scp=85173164760&partnerID=8YFLogxK

U2 - 10.1016/j.caeai.2023.100168

DO - 10.1016/j.caeai.2023.100168

M3 - Journal article

SN - 2666-920X

VL - 5

JO - Computers and Education: Artificial Intelligence

JF - Computers and Education: Artificial Intelligence

M1 - 100168

ER -