Thank you for the detailed writeup. This is a topic which I think is not discussed much.
> We will split public-facing CI from release infrastructure and internal CI infrastructure. (teleport#8268)
Did you also consider some form of out-of-band approval mechanism for production environment access? (via a chatbot / push notification etc). I think something like that might work technically, but scalability might be a challenge. It might be easier to manage in comparison to a self-managed complete second CI system though. I have been pondering over it for some time to be able to utilize Gitlab CD without providing Gitlab all keys to the kingdom.
> Did you also consider some form of out-of-band approval mechanism for production environment access?
No, not before your comment at least. Vendor CI tools (be it GitLab, Drone, etc) often make it difficult to use this workflow. Their typical model is long lived static creds, and gating authn/authz around job kick off. I'm not aware of any that would work with delegated/approved credentials, at least without writing a custom secrets plugin. If anyone knows of such capabilities, give a me a holler.
Furthermore, there is still the risk of any service available to external contributors being compromised (as we saw in the this vulnerability). I'd just as soon have "no prod secrets touch a system that does external CI" as a security invariant -- no matter how trustworthy that external CI system is.
In a bittersweet irony, out-of-band approvals are in our product:
but we're not there with CI yet. :/ It would be fantastic if we could have short lived credentials issued only for the duration of the job, after approval (or better: after delegation) from a trusted party. Something like AWS's `CalledVia`.
> We will split public-facing CI from release infrastructure and internal CI infrastructure. (teleport#8268)
Did you also consider some form of out-of-band approval mechanism for production environment access? (via a chatbot / push notification etc). I think something like that might work technically, but scalability might be a challenge. It might be easier to manage in comparison to a self-managed complete second CI system though. I have been pondering over it for some time to be able to utilize Gitlab CD without providing Gitlab all keys to the kingdom.