[johnmarcus]

aaaalllllllll the way down at the bottom is this gem: >Some core Roblox services are using Consul’s KV store directly as a convenient place to store data, even though we have other storage systems that are likely more appropriate.

Yeah, don't use consul as redis, they are not the same.

[stuff4ben]

But you can... which is what some engineers were thinking. In my experience they do this because:

A) they're afraid to ask for permission and would rather ask for forgiveness

B) management refused to provision extra infra to support the engineers need, but they needed to do this "one thing" anyways

C) security was lax and permissions were wide open so people just decided to take advantage of it to test a thing that then became a feature and so they kept it but "put it on the backlog" to refactor to something better later

[aprdm]

Yes, this and having such a big consul cluster where the recommendation is to have more smaller clusters.

That said, could've happened to anyone and it was a great write up.