Predicting Terms in IS-A Relations with Pre-trained Transformers

摘要

In this paper, we explore the ability of the generative transformers to predict objects in IS-A (hypo-hypernym) relations. We solve the task for both directions of the relations: we learn to predict hypernyms given the input word and hyponyms, given the input concept and its neighbourhood from the taxonomy. To the best of our knowledge, this is the first paper which provides a comprehensive analysis of transformerbased models for the task of hypernymy extraction. Apart from the standard finetuning of various generative models, we experiment with different input formats and prefixes, zeroand few-shot learning strategies, and generation parameters. Results show that higher performance on both subtasks can be achieved by generative transformers with no additional data (like definitions or lemma names). Such models have phenomenally high abilities at the task given a little training and proper prompts in comparison to specialized rule-based and statistical methods as well as encoder-based transformer models. toy_dog.n.01 ym pern hy king charles spaniel n ictio pred hyp ony toy_spaniel.n.01 mp english toy spaniel red icti on papillon Figure 1: Two formulations of taxonomy enrichment task: attaching new candidates to the existing nodes (red) and generating new candidates at the specified place in the taxonomy (blue). et al., 2022), hypernymy (Ravichander et al., 2020), negation (Ettinger, 2020), etc. Applying modern transformers for identifying IS-A relations would allow not only to check the capacities of large language models (LLMs) in linguistics, but also to perform automatic extension of the lexical taxonomic structures with such kinds of