PhraseSumm: Abstractive Short Phrase Summarization

摘要

Utterance Intent Label/Short phrase Summary Dataset: CLINC150 give me a hand finding my mobile find phone device how to keep my credit score from improve credit score going down i lost my chase bank card and report lost card want it labeled as lost Dataset: SNIPS Give me Steve‘s address in Man- Get Place Details hattan Get me a table at a restaurant Book Restaurant near Emily’s place for tomorrow 9pm What is the cheapest restau- Compare Places rant between Balthazar and Lombardi’s? Prior work in the field of text summarization mostly focuses on generating summaries that are a sentence or two long. In this work, we introduce the task of abstractive short-phrase summarization (PhraseSumm), which aims at capturing the central theme of a document through a generated short phrase. We explore BART & T5-based neural summarization models, and measure their effectiveness for the task using both standard summarization metrics as well as human evaluation. Our work showcases the benefits of pre-training the summarization models using tasks such as phrasal paraphrase alignment and NLI before fine-tuning on the task itself, both of which help the model with abstraction and thereby yield improvements over the baselines. Human evaluation reveals that model generated summaries are often judged better than or equal to reference summaries, demonstrating that ROUGE scores underestimate true performance. Finally, we create and release a dataset for this task to enable further research in the area. 1 Table 1: Examples of utterances & intent labels from CLINC150 & SNIPS datasets. The intent labels are used as short phrase summaries for our task. Most prior work in both these areas focus on summarizing long documents such as news articles (Eyal et al., 2019; Nenkova et al., 2011; Ahuja et al., 2022), novel chapters (Ladhak et al., 2020; Wu et al., 2021), movie scripts (Chen et al., 2022; Gorinski and Lapata, 2015) etc. to generate summaries that are at least a few sentences in length. However, in this work, we introduce PhraseSumm, a new task of abstractive short-phrase summarization that aims at generating abstractive summaries consisting of a short phrase (2-3 words) from documents that are relatively shorter in length (in our case, mostly consisting of 5-12 words in a single sentence). The short phrase should aim to capture the main theme within the document. To the best of our knowledge, our work is unique in its focus on summarization using short phrases. Our task involves summarizing an input document using a short phrase which is often not present within the document, and can not, therefore, be obtained using extractive techniques alone. Thus, approaches based on topic modeling (Card