Improving Machine Reading Comprehension through A Simple Masked-Training Scheme

摘要

Extractive Question Answering (EQA) is a fundamental problem in Natural Language Understanding, aiming at answering given questions via extracting a contiguous sequence or span of words from a passage. Recent work on EQA has achieved promising performance with the help of pre-trained language models, for which Masked Language Modeling (MLM) is usually adopted as a pre-training task to predict masked tokens. This paper revisits MLM and proposes a simple yet effective method to improve the EQA performance, termed the [Mask]-for-Answering method (M4A). Specifically, three masking strategies are first introduced, which produce masked copies of the original passages. Instead of predicting masked tokens as in MLM, both original samples and masked copies are utilized simultaneously for training the EQA model. Importantly, a discrepancy loss is further incorporated to ensure that masked copies remain semantically close to the originals. As such, M4A is able to produce robust embeddings for both original and masked samples and infer correct answers even with masked context. Experimental study on several highly-competitive benchmarks consistently demonstrates the superiority of our proposed method over existing methods. M4A also achieves strong performance in low-resource settings and out-of-domain generalization. Limit Context: Portugal has the largest aquarium in Europe, the Lisbon Oceanarium, and the Portuguese have several other notable organizations focused on science-related exhibits and divulgation, like the state agency Ciência Viva, a programme of the Portuguese Ministry of Science and Technology to the promotion of a scientific and technological culture among the Portuguese population, the Science Museum of the University of Coimbra, the National Museum of Natural History at the University of Lisbon, and the Visionarium. Question: What is the name of the largest European aquarium? Answer: Lisbon Oceanarium Figure 1: An illustration case from SQuAD (1.1) (Fisch et al., 2019), using the proposed [Mask]-for-Answering algorithm (M4A) to infer answer(s) from samples with limit (unmasked) context. question-passage pair. Abundant evidences indicate that the strong encoding capability of PLMs has rapidly advanced the EQA performance, compared to traditional word embedding methods, such as GloVe and Word2Vec (Devlin et al., 2019; Joshi et al., 2020; Liu et al., 2019). Recently, how humans approaching reading comprehension becomes a major source of inspiration for enhancing EQA. There is a rich literature to incorporate the human-like reading comprehension strategy with PLMs and achieve remarkable success beyond the vanilla models, as later shown in Section 2. Notably, when experienced human readers perform question reasoning, they could infer the correct answer using only few sentences (even some parts of one sentence), instead of the entire