| Start by limiting access to secrets within various pipelines. There's no reason why a merge request on a small tool repository needs access to the AWS keys for production for example. Splitting instances into different needs. The CI/CD pipeline used for deployment shouldn't be the same one that runs tests for developers. Especially if that CI/CD pipeline allows for the docker socket to be mounted inside of the CI/CD pipeline so that it can spin up more docker containers (as an example). It's all about limiting scope. Far too many of these systems are not appropriately configured setup, or are exposed with far too many secrets as environment variables and the like. Where possible also provide time limited scoped tokens. So a deployment takes ~10 minutes? The AWS creds for that deployment are scoped to 15 minutes. This way if an attacker were able to get those creds in logs or they were copied and pasted into a public paste bin, they were no longer valid by the time someone came across them. It's all about reducing scope. The other thing I've noticed is that the team that sets up the CI/CD tooling usually is someone that set it up in their spare time to complete a goal and suddenly it is infrastructure that is weight bearing. You want to make sure that it is maintained like the infrastructure it actually is. That means dedicated system administrators, looping security in, understanding the flows, understanding who has access and how, and what ACL's exist, how execution happens, understanding what kind of access it has. The lack of dedicated team/updates after it has been configured is killer. Old Jenkins versions are vulnerable to various issues that allow remote code execution in the context of Jenkins itself, so even if your pipelines only have time limited tokens, generally Jenkins itself has access to the keys to the kingdom (or runs on EC2 and now I've got access to the IAM instance profile keys). Regularly updating means that pipelines will likely break, someone is relying on that old feature that is now gone in the new version. It requires dedicated engineering time to feed the CI/CD system. That is ignored far too often. Here's ways that the team I was on broke in using CI/CD: - Pipeline logs were sent to an S3 bucket, which was public and we found a URL an engineer pasted into an open source ticket (the team had helpfully base64 encoded all the secrets in a debugging build, which was also in the open bucket) - CI/CD service was outdated and we could bypass the authentication it required giving us access to update pipelines - We could create a new pull request on an open source project that was using an internal CI/CD tool, and steal all the secrets - Found credentials for CI/CD in the source code to trigger a build on a downstream project (creds were not scoped, and had full access) - Figured out CI/CD tooling was not tied to corporate AD, so guessed user accounts based upon devs commenting on open source and one of them re-used a password found in a previous breach - Found creds for a Docker registry, uploaded our own build containers that exfiltrated all of the secrets based upon their own container so they didn't even notice - CI/CD re-used the same host for executing privileged builds as un-privileged builds, and branches were automatically built. Found an abandoned branch, pushed to it, and broke out of the container due it being privileged, and then waited for the privileged build to start to steal the environment variables for that container CI/CD literally is remote code execution as a service. Treat it as such. |
That sums it up. The code in question is just doing the job it claims to do. CI/CD services literally have forms that allow you to run direct shell commands. If a highly paid engineer doesn't understand the code they are running, then it's time to fire them and get someone in place who does.