The Glass Ceiling of Automatic Evaluation in Natural Language Generation

Abstract

Figure 1: Correlation with humans over time considering all existing metrics combined. On the x-axis: evaluation metrics ordered by their release time; y-axis: utterance-level Kendall’s τ with human when training a model to fit human judgments with all metrics available at the time (5-Fold cross-validation with XGBoost regressor). The dotted lines represent different human annotations and datasets. Different variants of the same metrics (like ROUGE-1 and ROUGE-2) are averaged. The datasets and metrics are described in Sec. 2. judgments globally. We do not make metric recommendation but reflect upon the global progress in the field of automatic evaluation. Our work is motivated by the findings of Fig. 1. It depicts the