|
We use a giant pile of Clojure (almost everything, with specific measured exceptions, is). As a side effect, we have a lot of data. Not necessarily a lot in the sense of "big S3 bill", but definitely a lot in the sense of "you might not expect this being in a machine-readable format". Things like: what Lambdas existed in a customer AWS account 6 months ago in us-east-2 that had access to a specific SQS queue" (because we learned later that one of the consumers of that queue would actually consume Python pickles if you asked nicely, and hence get you RCE). As a side effect, we do a lot of data diffing: just mostly on more vanilla Clojure structures rather than data sets in the Datasette/CSV/... sense. For example, we have tooling (e.g. [recidiffist], which we also have wired up to Terraform + S3, so if you write some files to S3, you can get the structured diffs right next to it for free). It's one of those things that's simple and works ridiculously well. Well, if you do it consistently anyway. "Let's look at what resources aren't managed by IAC, and how that has changed historically" is an interesting question to me, but I couldn't do it without having that data available, encoded in a way that didn't pre-suppose how it would be used. Even though we mean "explanation" differently, this really is critical for enabling explanation and exploration. While many of these pieces of data (e.g. "historical complete snapshots of AWS environments") aren't big compared to a lot of corporate data lakes or log management systems, they're way bigger than what fits in my tiny human brain at once. They're still amenable to fast local analysis. We use [clerk] for amazing notebooks to aid in that. Where this all comes together is in the encoded, and machine-evaluatable, expressions. The way you do a Cloud audit is you write a Clojure program: you develop invariants about the environment that are or at least should be true, and then you talk about the places that didn't work out. [recidiffist]: https://github.com/latacora/recidiffist
[clerk]: https://github.com/nextjournal/clerk |