On a Benefit of Masked Language Model Pretraining: Robustness to Simplicity Bias

摘要

Despite the success of pretrained masked lan- 001 guage models (MLM), why MLM pretraining 002 is useful is still a question not fully answered. 003 In this work we theoretically and empirically 004 show that MLM pretraining makes models ro- 005 bust to lexicon-level spurious features, partly 006 answering the question. Our explanation is 007 that MLM pretraining may alleviate problems 008 brought by simplicity bias (Shah et al., 2020), 009 which refers to the phenomenon that a deep 010 model tends to rely excessively on simple fea- 011 tures. In NLP tasks, those simple features 012 could be token-level features whose spurious 013 association with the label can be learned easily. 014 We show that MLM pretraining makes learn- 015 ing from the context easier. Thus, pretrained 016 models are less likely to rely excessively on 017 a single token. We also explore the theoreti- 018 cal explanations of MLM’s efficacy in causal 019 settings. Compared with Wei et al. (2021), we 020 achieve similar results with milder assumption. 021 Finally, we close the gap between our theories 022 and real-world practices by conducting experi- 023 ments on real-world tasks. 024