Variable-Length Word Encodings for Neural Translation Models

摘要

Recent work in neural machine translation has shown promising performance, but the most eective architectures do not scale naturally to large vocabulary sizes. We propose and compare three variable-length encoding schemes that represent a large vocabulary corpus using a much smaller vocabulary with no loss in information. Common words are unaected by our encoding, but rare words are encoded using a sequence of two pseudo-words. Our method is simple and eective: it requires no complete dictionaries, learning procedures, increased training time, changes to the model, or new parameters. Compared to a baseline that replaces all rare words with an unknown word symbol, our best variable-length encoding strategy improves WMT English-French translation performance by up to 1.7 BLEU.