[srush]

It's not there yet, but the improvement has been quite significant in aggregate. Key table from the GoogleNMT paper, empirically showing a 60% relative improvement on this task:

                  PBMT  GNMT  Human Relative Improvement

English → Spanish 4.885 5.428 5.504 87%

English → French 4.932 5.295 5.496 64%

English → Chinese 4.035 4.594 4.987 58%

Spanish → English 4.872 5.187 5.372 63%

French → English 5.046 5.343 5.404 83%

Chinese → English 3.694 4.263 4.636 60%

[vicaya]

This is obviously data dependent. I suspect that the advantage of human is much higher in colloquial content vs written (esp. news) content. "Universal Adversarial Perturbations"[1] last year showed that you can easily generate reasonable (to human) perturbations to completely fool the state of the art DNNs for images. I suspect that the same is true for the current batch of NMTs as well. As a simple demo, I change the example Chinese a little (就要下班了。下班了再说吧。Only aux characters changes with the same meaning) and all NMTs failed spectacularly in different ways.

  Google: It is necessary to get off work. To say it again.
  OpenNMT: it's going to work. Go back to work again.
  Baidu: It's going to work. After work.

[1] https://arxiv.org/abs/1610.08401

[srush]

Yeah this is a nice connection. Note however that there has been much less success in using perturbation in language. The fact that inputs are discrete makes it harder to apply some of the tricks in the adverserial image work.