How do you handle spot instances being shut down mid-build? Can you recover any built artifacts before the system goes down? IIRC you get a 2 minute warning.
Right now if a spot instance terminates mid-build it doesn't attempt to save any progress. I do have an open item to at least alert users that this has happened though (https://github.com/dan-v/rattlesnakeos-stack/issues/41). Although if Chromium is built it will checkpoint that by saving the built artifact to S3 and would then skip the Chromium build next time. I've found it's much cheaper to do full builds on AWS each time rather than trying to store the source tree for AOSP and Chromium as they are just so large.