[jaxbot]

I'm waiting for GCP to support preemptible GPU instances. Would love to be able to spend ~50% less on GPU instances when running a batch job that can be stopped and reloaded.

[doh]

Not sure that will happen anytime soon. It's the same with Local SSD. The virtualization of this environment is very challenging, maybe beyond a point where it makes sense for them to do so.

[williamstein]

But GCE already has local SSD’s for preempt instances, and they even cost less.

[doh]

Wasn't aware they added the support. Well, maybe they can figure out GPUs too.

[jeffmcjunkin]

I'd be surprised if it'd be that much additional work, if the local SSD options are NVMe -- which is also PCI-e like GPU's.

[0x0]

Wait what, would they expose the raw SSD pci device to your vm? What's stopping you from scraping all the leftover data from the previous customer?

[tylerjd]

Probably something along the lines of secure erase. Most modern SSDs/NVMe drives are encrypted by default in firmware. All the firmware needs to do is throw away the old keys and generate new ones. It's better than zeroing the drive as there is no wear to the write cycles and guarantees that the slack space in the SSD is also cleared, which DD'ing to /dev/nvme0 wont be certain of. The nvme-format tool can be used for this: http://manpages.ubuntu.com/manpages/zesty/man1/nvme-format.1...

[wmf]

I've seen some hints that GCE uses NVMe emulation not PCI passthrough. This would allow the hypervisor to implement features like live migration.

[sdevoid]

Your data is encrypted as soon as it exits the VM and before it's written to storage.

https://cloud.google.com/compute/docs/disks/#ssd_encryption