[Nouser76]

Is there any framework/system that distributes the work across multiple GPUs on different computers over a network (LAN or WAN)? I'm not concerned much about latency or generation time, but would love to train or load up huge models and send jobs to run overnight.

[jph00]

Yes, you can run FSDP/QLoRA over multi-node. There's a slurm script in the repo showing how to do it.