[vicaya]

This is obviously data dependent. I suspect that the advantage of human is much higher in colloquial content vs written (esp. news) content. "Universal Adversarial Perturbations"[1] last year showed that you can easily generate reasonable (to human) perturbations to completely fool the state of the art DNNs for images. I suspect that the same is true for the current batch of NMTs as well. As a simple demo, I change the example Chinese a little (就要下班了。下班了再说吧。Only aux characters changes with the same meaning) and all NMTs failed spectacularly in different ways.

  Google: It is necessary to get off work. To say it again.
  OpenNMT: it's going to work. Go back to work again.
  Baidu: It's going to work. After work.

[1] https://arxiv.org/abs/1610.08401

[srush]

Yeah this is a nice connection. Note however that there has been much less success in using perturbation in language. The fact that inputs are discrete makes it harder to apply some of the tricks in the adverserial image work.