TLDR; The authors train a Hierarchical Recurrent Encoder-Decoder (HRED) network for dialog generation. The "lower" level encodes a sequence of words into a though vector, and the higher-level encoder uses these thought vectors to build a representation of the context. The authors evaluate their model on the *MoviesTriples* dataset using perplexity measures and achieve results better than plain RNNs and the DCGM model. Pre-training with a large Question-Answer corpus significantly reduces perplexity. #### Key Points - Three RNNs: Utterance encoder, context encoder, and decoder. GRU hidden units, ~300d hidden state spaces. - 10k vocabulary. Preprocessing: Remove entities and numbers using NLTK - The context in the experiments is only a single utterance - MovieTriples is a small dataset, about 200k training triples. Pretraining corpus has 5M Q-A pairs, 90M tokens. - Perplexity is used as an evaluation metric. Not perfect, but reasonable. - Pre-training has a much more significant impact than the choice of the model architecture. It reduces perplexity ~10 points, while model architecture makes a tiny difference (~1 point). - Authors suggest exploring architectures that separate semantic from syntactic structure - Realization: Most good predictions are generic. Evaluation metrics like BLEU will favor pronouns and punctuation marks that dominate during training and are therefore bad metrics. #### Notes/Questions - Does using a larger dataset eliminate the need for pre-training? - What about the more challenging task for longer contexts?