PRiSM: Enhancing Low-Resource Document-Level Relation Extraction with Relation-Aware Score Calibration

摘要

Document-level relation extraction (DocRE) aims to extract relations of all entity pairs in a document. A key challenge in DocRE is the cost of annotating such data which requires intensive human effort. Thus, we investigate the case of DocRE in a low-resource setting, and we find that existing models trained on low data overestimate the NA (“no relation”) label, causing limited performance. In this work, we approach the problem from a calibration perspective and propose PRiSM, which learns to adapt logits based on relation semantic information. We evaluate our method on three DocRE datasets and demonstrate that integrating existing models with PRiSM improves performance by as much as 26.38 F1 score, while the calibration error drops as much as 36 times when trained with about 3% of data. The code is publicly available at https: //github.com/brightjade/PRiSM. 1 Figure 1: An overview of our proposed method. Top represents the original DocRE framework. PRiSM (bottom) leverages relation descriptions to compute scores for each relation triple. These scores are then used to reweight the prediction logits. triples. Under the low-resource setting, chances to observe infrequent relations become much harder. Second, DocRE models predict the NA (“no relation”) label if an entity pair does not express any relation. In DocRED, about 97% of all entity pairs have the NA label. With limited data, there is a much less signal for ground-truth (GT) labels during training, resulting in models overpredicting the NA label instead. High confidence in common relations and the NA label and low confidence in rare relations suggest that models may be miscalibrated. We hypothesize that lowering the former and raising the latter would improve the overall RE performance. At a high level, we wish to penalize logits of frequent labels (including NA) and supplement logits of infrequent labels such that models are able to predict them without seeing them much during training. To implement such behavior, we leverage relation semantic information, which has proved to be effective in low-resource sentence-level RE (Yang et al., 2020; Dong et al., 2021; Zhang and Lu, 2022). In this work, we propose the Pair-Relation Similarity Module (PRiSM) that learns to adapt logits by exploiting semantic information from label descriptions, as depicted in Figure 1. Specifically, we compute a similarity function for each entity pair embedding, constructed from two entities of interest, with relation embeddings, built from corresponding label descriptions. PRiSM then learns re-