Can You Translate for Me? Code-Switched Machine Translation with Large Language Models

摘要

Large language models (LLMs) have shown remarkable performance on a variety of multilingual NLP tasks. Code-switching is one of the most convenient styles of communication in multilingual communities. It is known to present several challenges to the existing language models and task-specific models. In this paper, we evaluate the capability of multi-lingual LLMs for the code-switched machine translation (CSMT) task in traditional and novel settings and present our insights. We observe that ChatGPT outperforms other LLMs and shows competitive performance to the supervised fine-tuned models. Though promising, ChatGPT shows major limitations, such as high gender bias, stereotypes, and factual inconsistencies. It further demands a multi-dimensional large-scale evaluation of the multilingual LLMs for code-switched languages.