[skissane]

> Instead, read secrets from a vault or from a file-system from _inside_ your process.

I’ve never liked making secrets available on the filesystem. Lots of security vulnerabilities have turned up over the years that let an attacker read an arbitrary file. If retrieving secrets is a completely different API from normal file IO (e.g. inject a Unix domain socket into each container, and the software running on that container sends a request to that socket to get secrets), that is much less likely to happen.

[andrewmcwatters]

God this is such a prime example of how we just don't do security well enough industry wide, and then you end up with weird stupid stuff like encryption being an enterprise paid feature.

Secrets have to be somewhere. Environment variables are not a good place for them, but if you can't trust your filesystem to be secure, you're already screwed. There's no where else to go. The only remaining place is memory, and it's the same story.

If you can't trust memory isolation, you're screwed.

As a counterintuitive example from a former insider: virtually no one is storing secrets for financial software on an HSM. Almost no one does it, period.

[skissane]

> Secrets have to be somewhere. Environment variables are not a good place for them, but if you can't trust your filesystem to be secure, you're already screwed. There's no where else to go. The only remaining place is memory, and it's the same story.

There’s a whole class of security vulnerabilities that let you read from arbitrary files on the filesystem. So if you end up having of those vulnerabilities, and your secret is in a file, then the vulnerability lets the attacker read the secret. And on Linux, if you have such a vulnerability, you can use it to read /proc/PID/environ and get the environment variables, hence getting secrets in environment variables too.

However, the same isn’t necessarily true for memory. /proc/PID/mem isn’t an ordinary file, and naive approaches to reading it fail. You normally read a file starting at position 0; reading /proc/PID/mem requires first seeking to a mapped address (which you can get from /proc/PID/maps); if you just open the file and start reading it from the start, you’ll be trying to read the unmapped zero page, and you’ll get an IO error. Many (I suspect the majority) of arbitrary-file read vulnerabilities only let you read from the start of the file and won’t let you seek past the initial unreadable portion, so they won’t let you read /proc/PID/mem.

Additionally, there are hardening features to lock down access to /proc/PID/mem, such as kernel.yama.ptrace_scope, or prctl(PR_SET_DUMPABLE)-that kind of hardening can interfere with debugging, but one option is to leave it on most of the time and only temporarily disable it when you have an issue to diagnose

Also, memfd_secret supports allocating extra-special memory for secret storage, which the kernel can’t read, so it shouldn’t be accessible via /proc/PID/mem

[delusional]

>There’s a whole class of security vulnerabilities that let you read from arbitrary files on the filesystem.

This is maybe putting the cart before the horse a little bit. The reason there's a class of vulnerabilities that allow arbitrary read is that we've, as an industry, decided that we classify file access as a vulnerability. It's not that file access is somehow materially different or easier from any other security issue, it's just that we set that as one of the goals of an attack.

If you decide that an attack is successful when it reads a file, then you'll obviously get a clustering of successful attacks that read files.

[skissane]

It isn’t just about preventing vulnerabilities, it is also about limiting the damage they can cause. Suppose you have a web app, with customer data in a remote relational database. An arbitrary file read vulnerability, in itself, might not actually help an attacker in stealing your customer data, since it is in a remote DB not the web app’s filesystem. But if that vulnerability enables them to exfiltrate database credentials, that gets them one step closer to actually stealing your customer data, which can be an enormously costly legal and PR headache. (By itself, those credentials won’t be that useful, since hopefully your firewall will block direct public access to the DB - but a lot of successful attacks involve chaining multiple vulnerabilities/weaknesses - e.g. they compromise some employee laptop that lets them talk to the DB but they don’t have credentials, and now they have the credentials too.)

Whereas, if all they manage to steal using a file read vulnerability is the code (possibly even just the binaries if you are using a compiled language like Go or Java) of your web app - that’s not good either, but it is a lot smaller headache. You’d much rather be having to tell the CEO “attackers stole the binaries of our app” than “attackers stole all the PII of our customers”. Both are bad but the second is a lot worse. The first kind of attack you possibly won’t be obliged to disclose, the second you legally will be

[TristanBall]

Did they say why?

It strikes me that those envs might be particularly prone to corporate inertia, ieg "the current way passed security audit, don't change it or we need to requalify"

It's possibly also harder to rely on a HSM when your software is in a container? ( I'm guessing here tho )

[nprateem]

It's a useless, unproveable generalisation from a supposedly omniscient "insider". I know of at least one finance organisation using HSM as you'd expect.

[skissane]

And I know non-finance orgs using HSM to protect encryption keys used to encrypt PII

[andrewmcwatters]

Yeah, you don't have to trust me, there are plenty of software engineers working in finance who can tell you the same. Or they're using outdated ciphers, or they're storing information in plaintext or in logs, or they have no security playbooks.

It's irrelevant to me whether you believe it, it's happening today, and it happens with some of the top financial institutions and their subsidiaries and it's the same bureaucratic nonsense to move those teams to do something about it like it is anywhere else.

[itchyouch]

I used to store secrets in the FS, and was told the best practice was env vars. Now it's not env vars. What is it then?

[qweqwe14]

There isn't a right answer. It's just that people don't understand that one doesn't provide any meaningful benefit over the other (in the context of storing secrets), but the security "experts" are always eager to claim "X is insecure, do Y instead, it's best practice btw"

Unless I'm missing something, there are three scenarios where this comes up:

1. You are using a .env file to store secrets that will then be passed to the program through env vars. There's literally no difference in this case, you end up storing secrets in the FS anyway.

2. You are manually setting an env var with the secret when launching a program, e.g. SECRET=foo ./bar. The secret can still be easily obtained by inspecting /proc/PID/environ. It can't be read by other users, but so are the files in your user's directory (.env/secrets.json/whatever)

3. A program obtains the secret via some other means (network, user input, etc). You can still access /proc/PID/mem and extract the secret from process memory.

So I'm assuming that what people really want is passing the secret to a program and having that secret not be readable by anything other than that program. The proper way to do this is using some OS-provided mechanism, like memfd_secret in Linux. The program can ask for the secret on startup via stdin, then store that secret in the special memory region designed for storing secrets.

[jampekka]

The main security benefit of byzantine paranoid security best practices is that they massively hinder productivity. If you can't make a system, the system will have no vulnerabilities.

[qweqwe14]

lmao so true

[delusional]

You don't even need to roll your own solution with memfd. Linux already has keyrings[1] as a kernel concept.

[1]: https://man7.org/linux/man-pages/man7/keyrings.7.html

[macNchz]

I’d wager that–in the context of web apps–over time there have been many more (or more readily exploitable) arbitrary file read/directory traversal/file inclusion vulnerabilities than remote code execution ones, so the preference for having secrets in memory as env vars may stem from that. You’re also probably not reading from /proc/self/mem without code execution either.

[qweqwe14]

Well, if there's an arbitrary file read, shouldn't the attacker be able to just read /proc/PID/environ anyway? It behaves like a regular file in that regard, unlike /proc/PID/mem, which requires seek operations to read data.

[merb]

Also for 3. you just added another supply chain issue and another thing to maintain and keeping up to date, which is probably worse

[skissane]

Well, I’d be the first to admit that we have a gap here, the solution that I personally would consider ideal doesn’t seem to actually exist, at least on the server-side.

If we are running under something like K8S or Docker, then I think there should be some component that runs on the host, that provides access to secrets over a Unix domain secret, and then we mount that socket into each container. (The reason I say a Unix domain socket, is so then the component can use SCM_CREDENTIALS/SO_PEERCRED/etc to authenticate the containers). I’d also suggest not using HTTP, to reduce the potential impact of any SSRF vulnerabilities (although maybe that’s less of a risk given many HTTP clients don’t work with Unix domain sockets-or at least not without special config). (Can we pass memfd_secret using SCM_RIGHTS?)

For desktop and native mobile, I think the best practice is to use the platform secret store (Keychain on macOS/iOS, Freedesktop Secret Service for desktop Linux, Android Keystore, Windows Credential Manager API, etc). But for server-side apps, those APIs generally aren’t available (Windows excepted). Server-side Linux often lacks desktop Linux components such as Freedesktop APIs (and even when they’re present, they aren’t the best fit for server-side use cases)

[graemep]

The problem with .env files is that you are doing both.

You have a .env file that is in the same directory as your code and you just copy to to env vars at some point. This does not even meet the security principles that dotenv is supposed to implement!

I think people are blindly following the advice "put secrets in env vars" without understanding that the point of it is to keep secrets outside files your app can read - because if you do a vulnerability or misconfiguration that lets people read those files leaks the secrets.

What you can do is have environment vars set outside your code, preferably by another user. You do it in your init system or process supervisor. Someone mentioned passing them in from outside a docker container in another comment.

[skissane]

> people are blindly following the advice "put secrets in env vars" without understanding that the point of it is to keep secrets outside files your app can read - because if you do a vulnerability or misconfiguration that lets people read those files leaks the secrets.

The problem with this is that, on Linux, the environment is a file, /proc/self/environ

And yes, as has been mentioned in some other comments, the process memory is also a file /proc/self/mem - but it is a special file that can only be read using special procedures, whereas /proc/self/environ behaves much more like a normal file, so a lot of vulnerabilities that enable reading /proc/self/environ wouldn’t enable reading /proc/self/mem

Technically one workaround on Linux is to not mount /proc (or at least not in your app’s container) - but doing that breaks a lot of things

[anentropic]

Yes... surely the point of .env file is as a shortcut to provide env vars for local development

.env file shouldn't be used in production, env vars should be injected directly

[graemep]

I agree that is what should happen. That is not what dotenv does though: https://www.npmjs.com/package/dotenv#-documentation

[anentropic]

I think dotenv would be fine as long as it doesn't raise exceptions if no .env file is found, i.e. if it works just as a helper for local dev and as a no-op for production

I notice the dotenv docs are recommending dotenvx now in places: https://www.npmjs.com/package/dotenv#-manage-multiple-enviro...

[graemep]

I have seen many .env files in production. SOme junior devs told me it was what they had been taught to do.

[csomar]

Not using env vars is security through obscurity. If someone has ssh access to your container, it doesn't matter whether the secrets are on a file or on memory. The attacker has as much access as the app itself.

On the other hand, using .env vars can leak in different ways like a developer mistakenly committing secrets to git or making this file available to the world wide web.

[andrewmcwatters]

The filesystem is fine, but we really shouldn't be using .env files that get loaded into environment variables due to them leaking in a few different ways.

[justinclift]

Pull them from a remote host over ssh? :D