[kureikain]

It's weird that these salt master are reach-able from internet and they can sleep well with it.

Even with zero-trust network or beyondcorp idea, I still found one extra layer of protection a VPC give are so great. Few years ago, it has an issue with K8S API Server, and updating k8s isn't a walk in the park. I felt relax back then because we have everything inside VPC.

You can use SSH or VPN to access service inside VPC. But any of tools that had permission to manage your infrastructure should never expose to the internet.

Same thing with Jenkins, if you are using Jenkins to manage Terraform or trigger Ansible/Salt/Chef run, make sure Jenkins is not reachable from internet. Using different method to route webhook into it.

[trabant00]

I never understood the current trent to say VPN is a thing of the past. Redundancy in security layers is how you dont't get affected by every CVE out there.

Imo this is THE lesson to learn from this story.

Seondary: salt and ansible are not very mature yet.

[dijit]

Salt is definitely immature (been using it for 5 years and the situation has actually gotten worse in that time) but Ansible is a weird thing to group.

What issues do you have with Ansible?

[darkwater]

Yeah, I completely agree and really don't see the point of having a Configuration Management server facing Internet and basically having all your servers connect to it through the Internet! One thing is BeyondCorp idea to eliminate the roadwarrior concept and another is having your infra management exposed to CVEs in the wild!

For Jenkins it's a bit more complicated because GitHub webhooks although they do publish their IPs in a programmatic form so you can whitelist them.

[kureikain]

For Jenkins, what I do is:

1. Configured webhook override in Jenkins. So Jenkins will register sth like https://ci-webhook.domain.com to github webhook.

2. This ci-webhook is a simple webapp that validate webhook and if it's valid(sign by correct key), write the payload to SQS queue

3. A small daemon, run on same Jenkins master, that pulls SQS queue, and replay it to local jenkins

I used to rely on Github IP whitelist but one day i realized anyone can hit my Jenkins use Github.

[darkwater]

> I used to rely on Github IP whitelist but one day i realized anyone can hit my Jenkins use Github.

That's a really good point but I guess you are talking about Actions egress right? Webhook in theory have dedicated IP ranges [1] and I think they are not shared with Actions egress, although TBH I haven't tested it.

[1] https://api.github.com/meta

[tilolebo]

Hooks have a dedicated IP range.

With Terraform and AWS it's pretty simple to create a security group on an Application Load Balancer and whitelist these IPs using https://www.terraform.io/docs/providers/github/d/ip_ranges.h...