Self hosting is only best if you want maximum control. But most people can't handle that imagined level of control in reality, at least not on the same level as a major cloud platform. AWS (or GCP or Azure) infrastructure is overwhelmingly more likely to be able to do security better from a maintenance, intrusion handling or physical security perspective than any single organization can do, especially organizations smaller than AWS. The tradeoff is that despite how much AWS can claim to not touch your data, and all the feature, contract and compliance documentation they have to show for it, you can never be sure they're not touching it (deliberately or by accident).
Self hosting is not about confidentiality. For nearly all categories of "confidential data", I would much much rather have it in a major cloud platform than running in some closet somewhere or in some random colocation center somewhere, all other circumstances being equal.
Self hosting is about how much you want to be in control, regardless of your capabilities to actually be in control.
> Self hosting is only best if you want maximum control.
Not necessarily.
Beyond a certain scale you can go build your own datacenter (or smaller: rent a whole rack cabinet in a datacenter) and start exploiting economics of scale.
A lot of people don't realize that nowadays you can pack tens of cores and literally terabytes of ram in a 2u server.
That's what I don't quite understand about the current state of cloud computing. We're seeing huge advances in hardware/network technologies this decade but there's an ever increasing push to centralize hosting with cloud providers. Will this ever swing the other way?
Look at what's driving the shift: data centers are a major capital investment up-front plus a significant amount of staffing to operate and secure them. If you have enough proven need to justify that, you can easily beat a cloud provider — especially if you can simplify the problem in some ways that a generic service cannot.
For most organizations, however, it's hard to justify investing millions of dollars up-front in the hope that at some point you'll be saving enough to make that pay off. If that's not your core business it's often easier and safer to outsource it so, for example, you don't end up with a data center full of 50% utilized hardware which you bought to have capacity for growth which wasn't quite what you expected — or a big crunch when you have more demand than capacity and now need to double that investment to handle [currently] 10% of your usage.
> For most organizations, however, it's hard to justify investing millions of dollars up-front in the hope that at some point you'll be saving enough to make that pay off.
Well if you have your bills and a prospect of how much building and operating a datacenter would be, it could be very easy to do the calculation.
Btw one should not dismiss so easily the work of datacenter companies. They often have very high security standards and practices.
And this means that you don't necessarily have to build a datacenter from the ground up. You can start saving by just renting one or two rack cabinets and start putting your own hardware in there.
Current generation is so "cloud-happy" they dont appreciate the cost of being cloud-based... (10x? 100x? more?)
Large companies have been announcing HUGE savings, small companies would be able to save a LOT too... such a pity, all the cloud abstraction creates lazy teams IMO, and lazy companies... (again IMO, I know this wont be a popular view, because this audience is exactly the cloud-happy audiencem but if you achieve self criticism, self-hosting / colo etc is probably a better fit for 99% of cases)
It seems you're talking more about running costs, and not about any of the security aspects I was talking about?
I'll happily accept that you can pay less money for the same amount of power, but security isn't free. You don't only outsource a considerable amount performance and reliability engineering to $MAJOR_CLOUD_PROVIDER, but also a lot of security engineering. Doing a lot less of that is cheaper for sure, but is that worth the cost? I'd argue that for most (not all: most), it isn't.
At ever growing scales the equation will eventually tip in your favor, but you have to either be working at a very substantial scale for that, or you simply must not care for an important portion of the tasks that the major cloud provider picks up for you. That is fine by the way, but you have to be sure that that is actually a concious decision and you're not simply forgetting to actually do that work or doing it poorly.
I hope I don't sound combative, but "most people can't handle that imagined level of control in reality" don't seem very fact based. All companies used to self host, because there wasn't cloud. Many organisations still self host, like shared hosting providers, governmental institutions, banks.
I have two counter arguments against "big cloud can do better":
1) There is often assumption, that people behind cloud services are smarter than anyone else and don't make mistakes.
In reality, they still are humans. Big names attract some bright people, but not anyone is genius with good working in team skills.
2) Cloud companies have much harder problem to solve. They have two hostile fronts, outside world and clients. They need protect themselves from malicious clients and keep clients separated.
They offer generic services for everyone, so there is unused functionality for your use-case. You can disable/uninstall things in your self hosted setup (infra as whole, not meaning inside your virtual machine), filter aggressively in network perimeter and so on.
Self host don't only mean "running in some closet somewhere or in some random colocation center". I can't say, how things are done in US, but in my country government has several DC-s/server rooms for governmental agencies. There is on premises hosting too, sometimes with very good physical security.
Bear in mind that while cloud providers can be more technically sophisticated with their security, they are inherently less secure from the get-go than a box on premise: Because by default, the cloud provider must be configurable across the Internet from anywhere in the world, and my self-hosted box, by default, can only be configured by a mouse, keyboard, and monitor physically plugged into it.
From that point, yes, you open up your self-hosting to the world in a (hopefully) limited fashion and restrict access to your cloud management (hopefully) to a much narrower scope. But by default, a box in your building starts completely secure, and your AWS box starts accessible to anyone on the planet with your AWS password.
While it's true what you say, the underlying assumption is the general approach of "my network is secure and keeping track of what goes in and out is something I control". I won't comment on how it applies to your situation, but I do think this is assumption is outdated castle-and-moat security thinking. This assumption also falls apart on closer inspection in 99% of the situations and in my experience especially so in cases where people bring up this assumption.
Large companies with very decentralized infrastructure (who also profit off selling clouds in many cases) promote zero-trust infrastructure models. This is predominantly based on what works for them (having hundreds of offices or large amounts of remote staff), and of course, a must to sell people on if you want to sell cloud services.
Zero-trust is not without merit, by any means. It is good to not assume there are no cracks in your walls, and you should indeed use as much internal security as possible wherever you can.
But you know what's really quite silly? Deciding to fill your moat in with dirt and knock over your castle wall because you think it's possible for someone to get in anyways.
You had better believe I'm going to use the latest authentication and encryption tools between machines that I can to ensure nobody can listen in from a stray network connection... and that I'm also going to put all of it behind a firewall.
Yeah, lock your doors inside your castle, but for heaven's sake, the moat and the castle walls still help. Defense-in-depth is a concept I swear everyone forgot when clouds became a thing.
Ah yes, but now you're describing requiring you to set up engineering regiments within your castle, some portion of which you could outsource to your cloud provider in the alternative situation. My point is that this security boundary that you posit as an advantage in your original post is simply not very interesting (and often harmful) on its own.
Self hosting is no longer realistic because all the data you actually want to compute on is now in the cloud. The egress costs of that data into your self hosted environment would destroy you.
Imagine a world where the cloud produces more data at a rate that exceeds the pipes leaving the cloud. You are quite literally locked into computing within that same cloud.
Its very realistic! Its being done at this moment! (amazing, isnt it?)
lfmao... but jokes aside, the "cloud" is just series of datacenters with less choice of a brand, you do realize this? There is no such "cloud" as you say... Internet allows exactly for decentralized data (among cloud-1a-b1-c2 to cloud-2b-5x-3h is not all "in the cloud", its the same as datacenterA to datacenterB.... proximity still has an effect, as do all other network conditions...)
In this threat model that's a downside, not an upside.
SEV is not, to me, a convincing security model. It was tried a long time ago, it doesn't work. SGX uses small enclaves for a reason. Hacking your average Linux box is quite easy already, which is why compromised passwords flow like water.
With the SEV threat model the cloud provider is no longer on your side. They are no longer defending you against threats, they are a threat. That's why you want encrypted RAM. But you're being threatened by one of the world's most advanced security organisations, a company that literally pays a large team to do nothing but locate zero day exploits all day, every day. And they swap zero days with other major tech firms too. So they have access to bugs in your OS before you do, and this is structural, there's no fix for it.
Worse, they recommend you use Google-controlled, 'hardened' OS images. But that makes no sense because you're trying to defend yourself against Google.
Finally, it's very unclear to me that the Linux kernel is going to accept bug reports of the form "the hardware behaved in arbitrary incorrect ways because it was redefined by a malicious hypervisor on the fly". The kernel isn't designed to deal with a malicious hypervisor. It's not going to be checking the results of things that might be hypercalls to check the hypervisor didn't hand back invalid results. In the past this has caused big problems with userspace apps trying to run on malicious kernels: the kernel was able to break in to the encrypted memory space immediately because the kernel could manipulate syscall returns in ways the app didn't expect.
But let's put OS hacking to one side.
SEV has a very poor track record of security. There were dozens of bugs in their firmware in past revisions, ordinary C type buffer overflows. Then there were basic crypto bugs: you could send the firmware an invalid point on the elliptic curve and it wasn't checking for that, things like this. Perhaps Google has audited AMD's firmware now and have knocked out all these problems. Perhaps not. Who knows - they mention working with AMD on performance but not security.
Moreover, SEV doesn't have anything to say on the topic of side channel attacks. And unlike with SGX, because the whole point of SEV is you use existing software and operating systems, there's also no fix for this problem. Normal kernels and apps aren't designed to resist a compromised hypervisor. SGX enclaves are designed for-purpose so you can argue that they're the smallest piece of app logic that needs to process your data and you can go to town on securing it, whilst the bulk of the app handling resource management, connections, scheduling, etc, is blinded by cryptography. Enclaves expect the host to attack them because they were written that way. They can do things in a less efficient but more side-channel proof way. With SEV this is theoretically an option (could run some specially written hardened OS), but, it's not how it's being advertised so nobody will do it.
> But that makes no sense because you're trying to defend yourself against Google.
If you're using a cloud provider, you ultimately have to trust that provider is doing what they claim. After all, you have to use their management control plane to configure SEV, and that control plane could always be lying about whether SEV is actually working.
So SEV isn't intended as a defense against Google as an organization. What it can do, is provide a layer of defense against rogue hardware administrators, as well as other tenants that might be sharing the physical machine.
Actually SEV is meant to protect you against Google as an organisation (which would be the same thing as rogue administrators in this case). They don't mention it in the announcement but SEV is meant to be used with a little client side tool that does a remote attestation with the remote hardware. It handshakes with the firmware and you get back a hash as part of the VM boot process. You check that against an OS image you trust, and that's how you know what booted.
If you don't do this then it provides no protection. The host can break in by just telling you SEV is in use when it's really not.
Kinda when compared against perfection. Not when compared against SEV.
Consider that most of the side channel attacks being exploited against SGX also work across process and across VM domains. They tend to get advertised as SGX specific because that's the juiciest, newest and coolest target to hack. But they can break arbitrary CPU enforced protection domains.
Some of these side channel attacks are Intel specific. Many of them aren't: they're to do with how CPUs are designed, which is why Spectre et al affect AMD as well.
Whilst SGX gets a lot of focused attention from researchers exploring side channels, it has turned out to be pretty robust against the more ordinary kinds of attacks that felled a lot of prior systems, including multiple generations of SEV. Nobody has ever found basic cryptography or C programming bugs in the system enclaves, for example. I thought that would happen at least once - never did. All the bugs have been people reverse engineering CPU internals to a much greater degree than ever done before.
One reason they do this is because SGX is patchable in the field. A remote client can tell if the CPU microcode and SGX stack were updated to close vulnerabilities. Intel call this TCB recovery. So, it's kinda 'ethical' to research SGX bugs because you aren't breaking anyone's equipment.
AMD SEV has sadly not had a working equivalent of TCB recovery in prior versions. There was an attempt at such a mechanism but it can't stop downgrade attacks, so doesn't really work. Researchers have managed to totally break SEV such that the CPU generation itself had to be discarded and replaced, not just once, but multiple times. That's the worst case scenario for hardware roots of trust. Hopefully the new gen chips won't suffer any similar fate.
Given the fact that SGX has always been renewable/patchable, that all bugs found in it were really hard-core low level CPU design bugs of the type that AMD have also had, and that it has a stronger security posture to begin with (less code in enclaves than a whole OS), I'd say overall it's doing well. Now SEV is playing in the major leagues I expect to see more research on AMD chips: it'll be interesting to see what they come up with.