Ask HN: Would you know a “Python idempotent incremental update manager”?

[ebreton]

I maintain a fleet of edge devices, including their embedded software written in python. There is a release every two weeks or so, which usually consists on pulling a new set of docker images.

My current pain comes from the fact that, sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.

I am currently writing a (python) script and a documented process for every update, which is gently becoming a pain in the xxx for the units which are not available so often, and for which I have a few updates late.

I would love to have this pile of scripts ran through automatically with a higher level command, and create the skeleton of a new migration script on demand.

Hence the question: how do you manage such remote incremental update manager ? I am looking for something as simple as alembic, but obviously to run scripts not DB migrations.

[ebreton]

My feedback for those who helped me or are interested in following up :)

Pyinfra was the perfect match.

I have used their "operations" to pile up the actions that come with the next release:

- A new package (apt)

- an updtated pip dependency

- checking a file

- and docker-compose actions to update all containers

I was able define my "inventory" easily, which allows me to update my whole fleet with one single command. One caveat here, the timeout is not properly configurable with a ProxyJump. Discussion opened on stackoverflow.

Pyinfra idempotency and dry-run approaches allow me to run the update as many times as I want.

For the next release, would I need to change anything different, I will simply add a new operation. The beauty of it is that all operations are strictly executed in the order I define them, which mean that any devices that would not have been updated with the first release, would be still updated appropriately with the second release.

Cheers !

[Someone]

You wrote “sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.”

and “I am looking for something as simple as alembic, but obviously to run scripts not DB migrations.”

I don’t see how having to run scripts follows from the first. Can’t you put the stuff that lives outside docker images in a git repo and pull changes from that?

Initial install would be a git clone and updates a git pull (maybe with a reboot to make sure nothing is running that’s using outdated stuff)

[ebreton]

The repo will provide a way to pull new scripts for new updates.

But I would still need the framework to iterate through all of them

[Someone]

I think you either didn’t mention some requirement, or are too focused on “I need to run scripts to do these updates” because that’s what you use now.

To do “sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.”, I don’t see what else than git pull you would need, if you put docker-compose.yml and the libraries in the repo.

rsync or other sync programs also should work.

[fermigier]

Not sure I understand the problem precisely, a couple of thoughts from what I understand:

1) Aren't (Docker/OCI) containers supposed to be immutable (at least, as a best practice) ?

2) Have you looked at Pyinfra and would it help solve your problem ?

[ebreton]

1) yes, docker containers are immutable. The pain comes more from updates that are around them 2) Trying pyinfra. Thanks for pointing it to me ! Looks promising indeed :)

[yuppie_scum]

Dependabot?

[ebreton]

Thanks for the idea ! I will try pyinfra first, which provides a way to connect to the fleet and then to execute a whole set of scripts. Whereas Dependabot's perimeter looks more contained on dependencies update.