Hacker News new | ask | show | jobs
by 100011_100001 1887 days ago
My team is responsible for about 200 microservices being deployed, some of them have 10 or more pods. We don't do continuous delivery. Instead it's done by 5 different groups deploying once every week or two.

Our production deployment jobs are in Jenkins and isolated. It's easy to check what was deployed when. We also have a script written that can run an environment report to see what versions and which microservices have been deployed. Along with their CPU/memory allocations, number of pods etc.

Release management tracks which JIRA stories are in which release, they do it mainly by looking at master merges between prod deployments.
2 comments
JamesSwift 1887 days ago
That seems overwhelming to me. Do you feel this is a tenable strategy? Or is the number of deployments becoming an issue?
link
twh270 1887 days ago
This is similar to what $myclient is doing.

Parent comment doesn't mention whether identification of versions is done manually or whether they just grab master. If the latter, it's probably reasonable. At $myclient, every release to stage and prod requires teams to manually identify each version of each microservice as well as the stories (JIRA tickets) that are being deployed. This is extremely painful, time-consuming, and error-prone. Avoid at all cost; as the number of services grows, the pain/time/error cost appears to increase geometrically.

link
100011_100001 1886 days ago
Our master versions automatically create new images with unique ids and JIRA stories are associated with them. You can see that from JIRA to Gitlab and vice versa.

Having said that there are people that do some manual correlation. Mainly to be able to determine that the fixes actually got in.

link
100011_100001 1886 days ago
If you automate enough you almost don't need to know the specifics. We no longer monitor actively during production deployments. They just work, once every 5 months there is a deployment that needs to be escalated to us.
link
Crazyontap 1887 days ago
Any plans for showing user's birthday on the settings page?

Sorry, I'm just kidding but that's the only thing I could think off when I heard the number 200!

link