[nsomaru]

Curious; is it better to train locally on something like a 2080ti 11G or go for colab and offload checkpoints to S3?

Asking because it seems V100 performance (or the other colab paid GPU) is worth the occasional instability if you’ve set up checkpoints.

[mdda]

Look under "FP16 16-bit (Half Precision) Floating Point Calculations" on https://www.microway.com/knowledge-center-articles/compariso...

These raw numbers don't tell the whole story, of course. But IMHO, the convenience of a local 2080Ti outweighs the speed benefits of an _somewhat flaky_ V100 via Colab for day-to-day use (unless memory size is an issue, which you can't really get around).

OTOH, for just trying out stuff / one-offs, Colab is perfect - and bonus points if you score a V100.