GeniL: A Multilingual Dataset on Generalizing Language

摘要

Generative language models are increasingly transforming our digital ecosystem, but they often inherit societal biases learned from their training data, for instance stereotypes associating certain attributes with specific identity groups. While whether and how these biases are mitigated may depend on the specific use cases, being able to effectively detect instances of stereotype perpetuation is a crucial first step. Current methods to assess presence of stereotypes in generated language rely on simple template or co-occurrence based measures, without accounting for the variety of sentential contexts they manifest in. We argue that the sentential context is crucial to determine if the co-occurrence of an identity term and an attribute is an instance of generalization. We distinguish two types of generalizations ---(1) where the language merely mentions the presence of a generalization (e.g., "people think the French are very rude"), and (2) where the language reinforces such a generalization (e.g., "as French they must be rude"---, from a non-generalizing context (e.g., "My French friends think I am rude"). For meaningful stereotype evaluations, we need scalable ways to reliably detect and distinguish such instances of generalizations. To address this gap, we introduce the new task of detecting generalization in language, and build GeniL, a multilingual dataset of over 50K sentences from 9 languages ---English, Arabic, Bengali, Spanish, French, Hindi, Indonesian, Malay, and Portuguese--- annotated for instances of generalizations and their types. We demonstrate that the likelihood of a co-occurrence being an instance of generalization is usually low, and varies across different languages, identity groups, and attributes, underscoring the inadequacy of simplistic co-occurrence based approaches. We also build classifiers that can detect generalization in language with an overall PR-AUC of 58.7, with varying degrees of performance across languages. Our research provides data and tools to enable a nuanced understanding of stereotype perpetuation, a crucial step towards more inclusive and responsible language technologies.