[mag00]

Hi, I wrote this!

To continue a discussion:

  - How does your engineering team track new "debt" after releasing code? (if at all, and why not)
  - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful?
  - Do you feel like your company is good at managing access when hiring / firing people?

Otherwise thanks for any feedback, I enjoy writing these!

[haser_au]

Can only speak about my corner of a very large organisation;

- Technical debt of custom coded solutions is a known issue across our organisation. New strategy is to move to market solutions, therefore outsourcing the risk to organisations with (hopefully) better code management than we have. For my corner, we don't have technical debt measured accurately enough for my liking.

- Yes, we pay for an use centralised logging. We've actually been through two solutions, and are now moving to a third due to various factors (cost, integrations, speed, out-of-the-box metrics). Integration into the centralised logging system is part of our Request for Tender marking criteria.

- Relatively good at disabling access after someone leaves. We integrate as much as possible to a central repository. It's just the outliers that tend to last beyond someone in the organisation. Critical systems are absolutely shutdown within 24 hours of a leaver departing (usually immediately if they're a bad leaver).

Edit: Formatting

[BuuQu9hu]

> (hopefully)

I hope you are auditing the code of those external orgs.

[haser_au]

When you use SaaS products, auditing the code is not a service they offer. You have to rely on certifications from independent certifying organisations, etc.

[riprowan]

Part of the goal is to establish an arms length relationship to lower legal liability.

[tptacek]

Which logging systems did you like/not like?

[haser_au]

Alientvault: Ok...we probably didn't get full potential here HP ArcSight: Extremely powerful, especially the normalizing of logs across similar system. Requires a team to manage though. Splunk: Our business isn't ready for cloud based hosting of centralised logs. Otherwise, we'd be on this already. From my perspective, purely from a reduction in complexity to pull useful information (not just Security).

[ThePhysicist]

Thanks for writing this, really insightful! A question: What's your advice on how to store secrets on the server-side?

Currently, I mainly use a seperate "secrets.yml" file that gets deployed via Ansible and is stored there encrypted using Ansible-Vault with a strong password. Is that a reasonable approach? What is your opinion about storing secrets in environment variables? It seems that some people advise this over storing them in files, but I have seen some cases where environment variables can be exposed to the web client as well.

[mag00]

I don't like the idea of keeping secrets in ENV and limiting it to config, though it's the kind of thing I'd ask other folks about myself to understand any tradeoffs. I see Kubernetes and other things supporting secrets in env variables so unsure how common it is.

The big win is simply keeping secrets out of source code, out of an general engineer's copy/paste buffer, and with errors not going to a logging platform with single factor access. Your likelihood of a short term incident decreases dramatically. Especially if those secrets have well segmented access, (IE, not a single AWS key with `AdministratorAccess` everywhere).

[tptacek]

If your code adopts a convention of reading secrets from the environment, you get a lot of flexibility in how they're actually stored; you can put them in protected files and export the contents of the file before running the service, or you can have a tool that works like "env" that populates from a secret store. Your secret storage system can get more sophisticated without your code having to change.

I wouldn't recommend putting them in /etc/environment or /etc/profile or /home/service/.profile where you'll forget about them, though.

Just as a strategy for passing secrets to code, I like the environment a lot.

[andrewflnr]

Article and discussion re storing secrets in environment variables: https://news.ycombinator.com/item?id=8826024

The gist seems to be that it's easy to accidentally leak environment variables (which is why I think the top comment is off-base). tptacek, do you think this risk is overblown?

[tptacek]

It's good to be aware of the fact that environments are inherited by child processes (as are file descriptors), but I don't think that's a good reason to avoid using the environment.

[mzzter]

Have you heard of torus.sh keyrings? I don't know how well it works for an organization, but integrating torus into my side projects has been painless.

[spydum]

Great article, shared it on with my coworkers.

- poorly, really.

- for network, and security stuff, absolutely: splunk is the bees knees. For apps, each team tends to run their own mix (graylog2/elk/custom). Have pushed for more security type events from apps into splunk for correlation, but it just costs too damm much.

- depends on the region. I find US / UK do okay, but the more emerging/growth markets where we have employees, the worse it gets.

[jasonlotito]

You said this: "Rarely do I see a team eliminate all of their debt, but the organizations _that least respect_ their debt never get so far behind that they can no longer be helped in a breach."

Do you mean instead "that _at_ least respect"?

I ask only because they two have different meanings.

[mag00]

Yep, fixing

[dsacco]

Thank you for writing these. These blog posts are my go-to resources when my client companies want to learn more about what they can do to improve their security posture long term. It's a really great series.