[sl1ck731]

I've been thinking about this in regard to AWS. The encryption at rest for most things is completely transparent so the only thing your really protected against is someone walking into a data center and grabbing your drive somehow. Or improperly disposed drives. Maybe some kind of hypervisor or SAN exploit but I don't know much about that.

AWS seems to have turned part of the cloud operating model they are supposed to be responsible for back onto the user and no one questions it.

[gtsteve]

There are other use-cases. For example, you can crypto-shred an unlimited amount of data just by deleting the key associated with it.

You can also set up workflows such as your client owning the encryption key that encrypts data held by you and they can revoke it at any time. Slack has a similar system and I was asked by a large financial institution about the same. I expect to see this more in future.

[aj3]

> For example, you can crypto-shred an unlimited amount of data just by deleting the key associated with it.

Sounds like a great target for ransomware crews!

[gtsteve]

Yes indeed! But you mark the key for deletion and there's a minimum time of 7 days before it is deleted and during that time you can't use it. You've got quite a while to realise there's a problem and fix it.

[toomuchtodo]

How many critical domains and TLS certs expire with many months of notice? In a proper shop, someone will be alerted. Most places are not on the ball, and that alert is going to go entirely unnoticed.

[gtsteve]

You'll probably notice your disks not mounting and your vms not starting up and your users not being able to take data etc. Or not, depends on how frequently accessed it is.

We'd definitely notice.

[aj3]

This sounds reasonable.

[acdha]

AWS encryption at rest comes in two flavors: their managed keys protect against the threat you managed (which is still important for some compliance targets) but if you use customer-managed keys you can go further and protect against compromises in your accounts - server A only has access to encrypt, server B can only decrypt, role C in account D can encrypt data before transferring it, even the account root user can’t update the policy to break that, etc. It’s considerably more work but also more benefit.

[aj3]

Not sure what benefits are there in scheme you described. Which threat exactly is mitigated by using this tech?

[acdha]

Simple example: say I use S3 with the Amazon-managed key. Anyone who has a policy granting access to a bucket — and lots of people love to write bucket policies with Resource:* — can read or write objects in that bucket using that key.

If you use a CMK, you can write custom policies which are both far more restrictive and shouldn't change as much as other resource types. That means that I could, say, have a policy which says the key-admin role/group/user is the only principal in the account which can update the key settings at all (even administrator can't do that then), the writer role is the only one with kms:Encrypt, and the reader role is the only one with kms:Decrypt. No matter what S3 access you have, if you aren't one of those roles you won't be able to use the encrypted data. This is probably also in a scenario something like “central group A provisions the KMS, devops group B creates lots of other resources using that key”.

You can add conditions, too — “anyone in our account can encrypt, decrypt requests can only come from this IP address or VPC”, “only requests from this AWS service are accepted” (i.e. that compromised EC2 instance can't use it), “access to data encrypted with this key can only be done in the two regions we approve of that”, “this particular encryption value must be used on all requests”, etc.

https://docs.aws.amazon.com/kms/latest/developerguide/key-po...

https://docs.aws.amazon.com/kms/latest/developerguide/policy...

That adds an extra layer of defense: if I compromise a user, even one with some level of administrative access, who doesn't have the CMK access all I can get are errors or encrypted data rather than the raw data. If you're careful you can architect environments where a person can deploy code without direct access to secrets or a system can stream data through to an encrypted store (if you are storing PII, this can be a huge difference between “all of our users” and “only the ones who used the system during this time period”). In some cases these can be bypassed (i.e. a CMK might not allow Administrator to access it directly but they could possibly issue credentials for a user who does have access) but you're preventing generic attacks which just scrape up everything that compromised credentials have access to, and hopefully increasing both the level access required and the likelihood of producing an audit alert.

[WrtCdEvrydy]

Sorry, here's your cloud fees back.

[ape4]

Self hosting (remember that) seems best for really confidential things.

[DCKing]

Self hosting is only best if you want maximum control. But most people can't handle that imagined level of control in reality, at least not on the same level as a major cloud platform. AWS (or GCP or Azure) infrastructure is overwhelmingly more likely to be able to do security better from a maintenance, intrusion handling or physical security perspective than any single organization can do, especially organizations smaller than AWS. The tradeoff is that despite how much AWS can claim to not touch your data, and all the feature, contract and compliance documentation they have to show for it, you can never be sure they're not touching it (deliberately or by accident).

Self hosting is not about confidentiality. For nearly all categories of "confidential data", I would much much rather have it in a major cloud platform than running in some closet somewhere or in some random colocation center somewhere, all other circumstances being equal.

Self hosting is about how much you want to be in control, regardless of your capabilities to actually be in control.

[znpy]

> Self hosting is only best if you want maximum control.

Not necessarily.

Beyond a certain scale you can go build your own datacenter (or smaller: rent a whole rack cabinet in a datacenter) and start exploiting economics of scale.

A lot of people don't realize that nowadays you can pack tens of cores and literally terabytes of ram in a 2u server.

[gzu]

That's what I don't quite understand about the current state of cloud computing. We're seeing huge advances in hardware/network technologies this decade but there's an ever increasing push to centralize hosting with cloud providers. Will this ever swing the other way?

[acdha]

Look at what's driving the shift: data centers are a major capital investment up-front plus a significant amount of staffing to operate and secure them. If you have enough proven need to justify that, you can easily beat a cloud provider — especially if you can simplify the problem in some ways that a generic service cannot.

For most organizations, however, it's hard to justify investing millions of dollars up-front in the hope that at some point you'll be saving enough to make that pay off. If that's not your core business it's often easier and safer to outsource it so, for example, you don't end up with a data center full of 50% utilized hardware which you bought to have capacity for growth which wasn't quite what you expected — or a big crunch when you have more demand than capacity and now need to double that investment to handle [currently] 10% of your usage.

[ape4]

Yeah especially if your company has more than one location - for redunancy

[duhast]

Dell PowerEdge r6525 - 1U server

- CPU: 128 cores, 256 threads (2 sockets)

- RAM: Up to 2TB RDIMM or 4TB LRDIMM (16 channels)

- Avg. power at 100% load: 750W

Standard rack size is 45U:

- CPU: 5760 cores, 11520 threads

- RAM: 180 TB

- Power: 33kW

You might need to sacrifice 1U or 2U for switches.

[roynasser]

Current generation is so "cloud-happy" they dont appreciate the cost of being cloud-based... (10x? 100x? more?)

Large companies have been announcing HUGE savings, small companies would be able to save a LOT too... such a pity, all the cloud abstraction creates lazy teams IMO, and lazy companies... (again IMO, I know this wont be a popular view, because this audience is exactly the cloud-happy audiencem but if you achieve self criticism, self-hosting / colo etc is probably a better fit for 99% of cases)

[DCKing]

It seems you're talking more about running costs, and not about any of the security aspects I was talking about?

I'll happily accept that you can pay less money for the same amount of power, but security isn't free. You don't only outsource a considerable amount performance and reliability engineering to $MAJOR_CLOUD_PROVIDER, but also a lot of security engineering. Doing a lot less of that is cheaper for sure, but is that worth the cost? I'd argue that for most (not all: most), it isn't.

At ever growing scales the equation will eventually tip in your favor, but you have to either be working at a very substantial scale for that, or you simply must not care for an important portion of the tasks that the major cloud provider picks up for you. That is fine by the way, but you have to be sure that that is actually a concious decision and you're not simply forgetting to actually do that work or doing it poorly.

[ZWoz]

I hope I don't sound combative, but "most people can't handle that imagined level of control in reality" don't seem very fact based. All companies used to self host, because there wasn't cloud. Many organisations still self host, like shared hosting providers, governmental institutions, banks. I have two counter arguments against "big cloud can do better":

1) There is often assumption, that people behind cloud services are smarter than anyone else and don't make mistakes. In reality, they still are humans. Big names attract some bright people, but not anyone is genius with good working in team skills.

2) Cloud companies have much harder problem to solve. They have two hostile fronts, outside world and clients. They need protect themselves from malicious clients and keep clients separated. They offer generic services for everyone, so there is unused functionality for your use-case. You can disable/uninstall things in your self hosted setup (infra as whole, not meaning inside your virtual machine), filter aggressively in network perimeter and so on.

Self host don't only mean "running in some closet somewhere or in some random colocation center". I can't say, how things are done in US, but in my country government has several DC-s/server rooms for governmental agencies. There is on premises hosting too, sometimes with very good physical security.

[ocdtrekkie]

Bear in mind that while cloud providers can be more technically sophisticated with their security, they are inherently less secure from the get-go than a box on premise: Because by default, the cloud provider must be configurable across the Internet from anywhere in the world, and my self-hosted box, by default, can only be configured by a mouse, keyboard, and monitor physically plugged into it.

From that point, yes, you open up your self-hosting to the world in a (hopefully) limited fashion and restrict access to your cloud management (hopefully) to a much narrower scope. But by default, a box in your building starts completely secure, and your AWS box starts accessible to anyone on the planet with your AWS password.

[DCKing]

While it's true what you say, the underlying assumption is the general approach of "my network is secure and keeping track of what goes in and out is something I control". I won't comment on how it applies to your situation, but I do think this is assumption is outdated castle-and-moat security thinking. This assumption also falls apart on closer inspection in 99% of the situations and in my experience especially so in cases where people bring up this assumption.

[ocdtrekkie]

Large companies with very decentralized infrastructure (who also profit off selling clouds in many cases) promote zero-trust infrastructure models. This is predominantly based on what works for them (having hundreds of offices or large amounts of remote staff), and of course, a must to sell people on if you want to sell cloud services.

Zero-trust is not without merit, by any means. It is good to not assume there are no cracks in your walls, and you should indeed use as much internal security as possible wherever you can.

But you know what's really quite silly? Deciding to fill your moat in with dirt and knock over your castle wall because you think it's possible for someone to get in anyways.

You had better believe I'm going to use the latest authentication and encryption tools between machines that I can to ensure nobody can listen in from a stray network connection... and that I'm also going to put all of it behind a firewall.

Yeah, lock your doors inside your castle, but for heaven's sake, the moat and the castle walls still help. Defense-in-depth is a concept I swear everyone forgot when clouds became a thing.

[ENOTTY]

Self hosting is no longer realistic because all the data you actually want to compute on is now in the cloud. The egress costs of that data into your self hosted environment would destroy you.

Imagine a world where the cloud produces more data at a rate that exceeds the pipes leaving the cloud. You are quite literally locked into computing within that same cloud.

[roynasser]

Its very realistic! Its being done at this moment! (amazing, isnt it?)

lfmao... but jokes aside, the "cloud" is just series of datacenters with less choice of a brand, you do realize this? There is no such "cloud" as you say... Internet allows exactly for decentralized data (among cloud-1a-b1-c2 to cloud-2b-5x-3h is not all "in the cloud", its the same as datacenterA to datacenterB.... proximity still has an effect, as do all other network conditions...)

Dont let the marketing sandcastle trap you in!

[londons_explore]

Except Google has a much much bigger budget for security engineers than you have...

[thu2111]

In this threat model that's a downside, not an upside.

SEV is not, to me, a convincing security model. It was tried a long time ago, it doesn't work. SGX uses small enclaves for a reason. Hacking your average Linux box is quite easy already, which is why compromised passwords flow like water.

With the SEV threat model the cloud provider is no longer on your side. They are no longer defending you against threats, they are a threat. That's why you want encrypted RAM. But you're being threatened by one of the world's most advanced security organisations, a company that literally pays a large team to do nothing but locate zero day exploits all day, every day. And they swap zero days with other major tech firms too. So they have access to bugs in your OS before you do, and this is structural, there's no fix for it.

Worse, they recommend you use Google-controlled, 'hardened' OS images. But that makes no sense because you're trying to defend yourself against Google.

Finally, it's very unclear to me that the Linux kernel is going to accept bug reports of the form "the hardware behaved in arbitrary incorrect ways because it was redefined by a malicious hypervisor on the fly". The kernel isn't designed to deal with a malicious hypervisor. It's not going to be checking the results of things that might be hypercalls to check the hypervisor didn't hand back invalid results. In the past this has caused big problems with userspace apps trying to run on malicious kernels: the kernel was able to break in to the encrypted memory space immediately because the kernel could manipulate syscall returns in ways the app didn't expect.

But let's put OS hacking to one side.

SEV has a very poor track record of security. There were dozens of bugs in their firmware in past revisions, ordinary C type buffer overflows. Then there were basic crypto bugs: you could send the firmware an invalid point on the elliptic curve and it wasn't checking for that, things like this. Perhaps Google has audited AMD's firmware now and have knocked out all these problems. Perhaps not. Who knows - they mention working with AMD on performance but not security.

Moreover, SEV doesn't have anything to say on the topic of side channel attacks. And unlike with SGX, because the whole point of SEV is you use existing software and operating systems, there's also no fix for this problem. Normal kernels and apps aren't designed to resist a compromised hypervisor. SGX enclaves are designed for-purpose so you can argue that they're the smallest piece of app logic that needs to process your data and you can go to town on securing it, whilst the bulk of the app handling resource management, connections, scheduling, etc, is blinded by cryptography. Enclaves expect the host to attack them because they were written that way. They can do things in a less efficient but more side-channel proof way. With SEV this is theoretically an option (could run some specially written hardened OS), but, it's not how it's being advertised so nobody will do it.

[theevilsharpie]

> But that makes no sense because you're trying to defend yourself against Google.

If you're using a cloud provider, you ultimately have to trust that provider is doing what they claim. After all, you have to use their management control plane to configure SEV, and that control plane could always be lying about whether SEV is actually working.

So SEV isn't intended as a defense against Google as an organization. What it can do, is provide a layer of defense against rogue hardware administrators, as well as other tenants that might be sharing the physical machine.

[thu2111]

Actually SEV is meant to protect you against Google as an organisation (which would be the same thing as rogue administrators in this case). They don't mention it in the announcement but SEV is meant to be used with a little client side tool that does a remote attestation with the remote hardware. It handshakes with the firmware and you get back a hash as part of the VM boot process. You check that against an OS image you trust, and that's how you know what booted.

If you don't do this then it provides no protection. The host can break in by just telling you SEV is in use when it's really not.

[saagarjha]

You are aware that SGX is also extremely broken, right?

[thu2111]

Kinda when compared against perfection. Not when compared against SEV.

Consider that most of the side channel attacks being exploited against SGX also work across process and across VM domains. They tend to get advertised as SGX specific because that's the juiciest, newest and coolest target to hack. But they can break arbitrary CPU enforced protection domains.

Some of these side channel attacks are Intel specific. Many of them aren't: they're to do with how CPUs are designed, which is why Spectre et al affect AMD as well.

Whilst SGX gets a lot of focused attention from researchers exploring side channels, it has turned out to be pretty robust against the more ordinary kinds of attacks that felled a lot of prior systems, including multiple generations of SEV. Nobody has ever found basic cryptography or C programming bugs in the system enclaves, for example. I thought that would happen at least once - never did. All the bugs have been people reverse engineering CPU internals to a much greater degree than ever done before.

One reason they do this is because SGX is patchable in the field. A remote client can tell if the CPU microcode and SGX stack were updated to close vulnerabilities. Intel call this TCB recovery. So, it's kinda 'ethical' to research SGX bugs because you aren't breaking anyone's equipment.

AMD SEV has sadly not had a working equivalent of TCB recovery in prior versions. There was an attempt at such a mechanism but it can't stop downgrade attacks, so doesn't really work. Researchers have managed to totally break SEV such that the CPU generation itself had to be discarded and replaced, not just once, but multiple times. That's the worst case scenario for hardware roots of trust. Hopefully the new gen chips won't suffer any similar fate.

Given the fact that SGX has always been renewable/patchable, that all bugs found in it were really hard-core low level CPU design bugs of the type that AMD have also had, and that it has a stronger security posture to begin with (less code in enclaves than a whole OS), I'd say overall it's doing well. Now SEV is playing in the major leagues I expect to see more research on AMD chips: it'll be interesting to see what they come up with.