A reproduction of Apple's bi-directional LSTM models for language identification in short strings

TY - UNPB

T1 - A reproduction of Apple's bi-directional LSTM models for language identification in short strings

AU - Toftrup, Mads

AU - Sørensen, Søren Asger

AU - Ciosici, Manuel R.

AU - Assent, Ira

N1 - Will be presented at EACL 2021 SRW

PY - 2021/2/11

Y1 - 2021/2/11

N2 - Language Identification is the task of identifying a document's language. For applications like automatic spell checker selection, language identification must use very short strings such as text message fragments. In this work, we reproduce a language identification architecture that Apple briefly sketched in a blog post. We confirm the bi-LSTM model's performance and find that it outperforms current open-source language identifiers. We further find that its language identification mistakes are due to confusion between related languages.

AB - Language Identification is the task of identifying a document's language. For applications like automatic spell checker selection, language identification must use very short strings such as text message fragments. In this work, we reproduce a language identification architecture that Apple briefly sketched in a blog post. We confirm the bi-LSTM model's performance and find that it outperforms current open-source language identifiers. We further find that its language identification mistakes are due to confusion between related languages.

KW - cs.CL

KW - cs.LG

M3 - Preprint

BT - A reproduction of Apple's bi-directional LSTM models for language identification in short strings

ER -