[throwaway439893]

> This all started when I was looking into why instances in an auto scaling group were sometimes failing to bootstrap correctly.

Please use Packer to build your images; don't do it on ASG instance deploy.

If somebody in the company tells you you're not allowed to build your own images, tell them to go fuck themselves, and write an e-mail to that guy's boss's boss explaining how much engineering time you're wasting (and how likely the products are to fail due to ASGs trying to bootstrap systems on the fly) because they won't let you cut an AMI.

[deathanatos]

> Please use Packer to build your images; don't do it on ASG instance deploy.

So you start using Packer to build your images. Your Packer script does "apt get install" and fails because something is holding the apt lock, and the author ends up writing more or less the same article.

Additionally, I work in Azure, and VM images in Azure are a world of absolute nightmarish pain there: a normal user literally cannot make the API call to bring up a VM with a custom image if the image is not in the same tenant. There is a way to do it with SP, but it is so completely and thoroughly undocumented as to be black magic. (Yes, if you Google, you'll be able to find Azure documentation on this exact subject. No, the instructions do not work.)

Yeah, I agree at the end of the day, it's the right way to do things, but it is an absolute, utter, PITA.

But then again, if you choose to not bring up a VM with a custom image, you get an unpinned image: "Ubuntu 20.04 LTS" is a moving target, and we once got one with a kernel that would BUG after ~5 minutes. Azure needed us to tell them what kernel we got from them.

[electroly]

In the Packer build you can have your provisioning script wait for the cloud-init to finish because you're not delaying a production scaleout; you've got all the time in the world. It's true that this article still gets written, but I don't think OP was suggesting otherwise. Building fully baked images rather than installing things on production startup avoids all sorts of flakiness.

Incidentally the author's suggestions are not that good, and not what I would suggest. He should be waiting for cloud-init to finish, not just apt:

    cloud-init status --wait

Before I knew about that, I used the following, which is still better than what the author suggests:

    while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done

Other stuff happens in cloud-init besides locking apt for awhile, and these will wait for all of it to finish.

[deathanatos]

Oooh, I did not know about that cloud-init command. I've always done something like your second one.

That just leaves unattended-upgrades, which is harder than the undead to kill, and Azure's WAAgent…