[rtkaratekid]

Late reply, but it was a bug using XLA GPUs to add concurrency to the training process. Maybe someone figured it out or fixed it, but I've moved on already.

[p1esk]

You shouldn’t need XLA for multi GPU training. Have you tried training without it?