Character-based Neural Machine TranslationCharacter-based Neural Machine TranslationCosta-Jussà, Marta R. and Fonollosa, José A. R.2016
Paper summaryaleju * Most neural machine translation models currently operate on word vectors or one hot vectors of words.
* They instead generate the vector of each word on a character-level.
* Thereby, the model can spot character-similarities between words and treat them in a similar way.
* They do that only for the source language, not for the target language.
* They treat each word of the source text on its own.
* To each word they then apply the model from [Character-aware neural language models](https://arxiv.org/abs/1508.06615), i.e. they do per word:
* Embed each character into a 620-dimensional space.
* Stack these vectors next to each other, resulting in a 2d-tensor in which each column is one of the vectors (i.e. shape `620xN` for `N` characters).
* Apply convolutions of size `620xW` to that tensor, where a few different values are used for `W` (i.e. some convolutions cover few characters, some cover many characters).
* Apply a tanh after these convolutions.
* Apply a max-over-time to the results of the convolutions, i.e. for each convolution use only the maximum value.
* Reshape to 1d-vector.
* Apply two highway-layers.
* They get 1024-dimensional vectors (one per word).
* Visualization of their steps:
* ![Architecture](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/Character-based_Neural_Machine_Translation__architecture.jpg?raw=true "Architecture")
* Afterwards they apply the model from [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473) to these vectors, yielding a translation to a target language.
* Whenever that translation yields an unknown target-language-word ("UNK"), they replace it with the respective (untranslated) word from the source text.
* They the German-English [WMT](http://www.statmt.org/wmt15/translation-task.html) dataset.
* BLEU improvemements (compared to neural translation without character-level words):
* German-English improves by about 1.5 points.
* English-German improves by about 3 points.
* Reduction in the number of unknown target-language-words (same baseline again):
* German-English goes down from about 1500 to about 1250.
* English-German goes down from about 3150 to about 2650.
* Translation examples (Phrase = phrase-based/non-neural translation, NN = non-character-based neural translation, CHAR = theirs):
* ![Examples](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/Character-based_Neural_Machine_Translation__examples.jpg?raw=true "Examples")