Chatbots Are Not Reliable Text Annotators

Ross Deans Kristensen-McLachlan; Miceal Canavan; Márton  Kardos; Mia Jacobsen; Lene Aarøe

doi:10.48550/arXiv.2311.05769

Chatbots Are Not Reliable Text Annotators

Ross Deans Kristensen-McLachlan^*, Miceal Canavan, Márton Kardos (Member of author collaboration), Mia Jacobsen (Member of author collaboration), Lene Aarøe

^*Corresponding author for this work

Research output: Working paper/Preprint › Preprint

Abstract

Recent research highlights the significant potential of ChatGPT for text annotation in social science research. However, ChatGPT is a closed-source product which has major drawbacks with regards to transparency, reproducibility, cost, and data protection. Recent advances in open-source (OS) large language models (LLMs) offer alternatives which remedy these challenges. This means that it is important to evaluate the performance of OS LLMs relative to ChatGPT and standard approaches to supervised machine learning classification. We conduct a systematic comparative evaluation of the performance of a range of OS LLM models alongside ChatGPT, using both zero- and few-shot learning as well as generic and custom prompts, with results compared to more traditional supervised classification models. Using a new dataset of Tweets from US news media, and focusing on simple binary text annotation tasks for standard social science concepts, we find significant variation in the performance of ChatGPT and OS models across the tasks, and that supervised classifiers consistently outperform both. Given the unreliable performance of ChatGPT and the significant challenges it poses to Open Science we advise against using ChatGPT for substantive text annotation tasks in social science research.

Original language	English
Publisher	arxiv.org
DOIs	https://doi.org/10.48550/arXiv.2311.05769
Publication status	Published - 2023

Access to Document

10.48550/arXiv.2311.05769Licence: CC BY-NC-SA

Cite this

@misc{dac18044af974c9fb049366a7ac01e07,

title = "Chatbots Are Not Reliable Text Annotators",

abstract = "Recent research highlights the significant potential of ChatGPT for text annotation in social science research. However, ChatGPT is a closed-source product which has major drawbacks with regards to transparency, reproducibility, cost, and data protection. Recent advances in open-source (OS) large language models (LLMs) offer alternatives which remedy these challenges. This means that it is important to evaluate the performance of OS LLMs relative to ChatGPT and standard approaches to supervised machine learning classification. We conduct a systematic comparative evaluation of the performance of a range of OS LLM models alongside ChatGPT, using both zero- and few-shot learning as well as generic and custom prompts, with results compared to more traditional supervised classification models. Using a new dataset of Tweets from US news media, and focusing on simple binary text annotation tasks for standard social science concepts, we find significant variation in the performance of ChatGPT and OS models across the tasks, and that supervised classifiers consistently outperform both. Given the unreliable performance of ChatGPT and the significant challenges it poses to Open Science we advise against using ChatGPT for substantive text annotation tasks in social science research.",

author = "Kristensen-McLachlan, {Ross Deans} and Miceal Canavan and M{\'a}rton Kardos and Mia Jacobsen and Lene Aar{\o}e",

year = "2023",

doi = "10.48550/arXiv.2311.05769",

language = "English",

publisher = "arxiv.org",

type = "WorkingPaper",

institution = "arxiv.org",

}

TY - UNPB

T1 - Chatbots Are Not Reliable Text Annotators

AU - Kristensen-McLachlan, Ross Deans

AU - Canavan, Miceal

AU - Kardos, Márton

AU - Jacobsen, Mia

AU - Aarøe, Lene

PY - 2023

Y1 - 2023

N2 - Recent research highlights the significant potential of ChatGPT for text annotation in social science research. However, ChatGPT is a closed-source product which has major drawbacks with regards to transparency, reproducibility, cost, and data protection. Recent advances in open-source (OS) large language models (LLMs) offer alternatives which remedy these challenges. This means that it is important to evaluate the performance of OS LLMs relative to ChatGPT and standard approaches to supervised machine learning classification. We conduct a systematic comparative evaluation of the performance of a range of OS LLM models alongside ChatGPT, using both zero- and few-shot learning as well as generic and custom prompts, with results compared to more traditional supervised classification models. Using a new dataset of Tweets from US news media, and focusing on simple binary text annotation tasks for standard social science concepts, we find significant variation in the performance of ChatGPT and OS models across the tasks, and that supervised classifiers consistently outperform both. Given the unreliable performance of ChatGPT and the significant challenges it poses to Open Science we advise against using ChatGPT for substantive text annotation tasks in social science research.

AB - Recent research highlights the significant potential of ChatGPT for text annotation in social science research. However, ChatGPT is a closed-source product which has major drawbacks with regards to transparency, reproducibility, cost, and data protection. Recent advances in open-source (OS) large language models (LLMs) offer alternatives which remedy these challenges. This means that it is important to evaluate the performance of OS LLMs relative to ChatGPT and standard approaches to supervised machine learning classification. We conduct a systematic comparative evaluation of the performance of a range of OS LLM models alongside ChatGPT, using both zero- and few-shot learning as well as generic and custom prompts, with results compared to more traditional supervised classification models. Using a new dataset of Tweets from US news media, and focusing on simple binary text annotation tasks for standard social science concepts, we find significant variation in the performance of ChatGPT and OS models across the tasks, and that supervised classifiers consistently outperform both. Given the unreliable performance of ChatGPT and the significant challenges it poses to Open Science we advise against using ChatGPT for substantive text annotation tasks in social science research.

U2 - 10.48550/arXiv.2311.05769

DO - 10.48550/arXiv.2311.05769

M3 - Preprint

BT - Chatbots Are Not Reliable Text Annotators

PB - arxiv.org

ER -

Chatbots Are Not Reliable Text Annotators

Abstract

Access to Document

Fingerprint

Cite this