AI-prompted peer feedback: Can an AI feedback coach enhance peer feedback quality?

Research output: Contribution to conferencePosterResearchpeer-review

Abstract

Research shows that peer feedback can significantly impact higher education students’ performance. However, peer feedback often is of poor quality. Studies indicate that peer feedback quality can be enhanced by using prompts to guide students’ feedback composition. To date, interventions applied static prompts that do not interact with students’ peer feedback input. Hence, this study explores the impact of an AI-based feedback coach that dynamically prompts students during feedback provision. The following research question was addressed: Does an AI-based feedback coach influence the composition of written peer feedback?

To investigate this, a quasi-experimental study was conducted in a competency development course for Ph.D. students (N = 46). Participants were tasked with submitting a written lesson plan in a digital learning environment and providing peer feedback. Students were divided into an intervention group (IG, n = 20) and a control group (CG, n = 26). In the IG an AI feedback coach was enabled during the feedback process, in the CG not.
Peer feedback quality was assessed by analyzing the feedback comments concerning length, readability, and sentiment using Natural Language Processing. Using Prins et al.’s (2006) feedback quality index (e.g., use of examples and explanations), quantitative-qualitative content analysis of each feedback comment is currently being conducted.

Analyses showed that CG participants wrote 326 comments, while the AI-assisted group wrote 418 comments. Preliminary findings indicate that participants with AI coaching wrote significantly shorter comments (t-test, p < .05 d = 0.15). Readability scores and sentiment analysis did not yield significant findings. Results from the in-depth content analysis will be presented at the conference.

This study contributes to the field of feedback research by analyzing how AI applications can enhance peer feedback quality and the peer feedback process.
Original languageEnglish
Publication date21 Mar 2024
Number of pages3
Publication statusAccepted/In press - 21 Mar 2024
EventJURE 2024 - Faculty of Psychology of the University of Sevilla, C. Camilo José Cela, s/n, 41018 Sevilla., Sevilla, Spain
Duration: 24 Jun 202428 Jun 2024
Conference number: 28th
https://www.earli.org/events/jure2024

Conference

ConferenceJURE 2024
Number28th
LocationFaculty of Psychology of the University of Sevilla, C. Camilo José Cela, s/n, 41018 Sevilla.
Country/TerritorySpain
CitySevilla
Period24/06/202428/06/2024
Internet address

Fingerprint

Dive into the research topics of 'AI-prompted peer feedback: Can an AI feedback coach enhance peer feedback quality?'. Together they form a unique fingerprint.

Cite this