[rfoo]

For FlashMLA? No. The code here runs on one GPU only and do not have a builtin communication part.

[pk-protect-ai]

But for the training it does. You need to communicate gradient changes between GPUs.