[SOLAR_FIELDS]

You know what's still stupid? That if you have an S3 bucket in the same region as your VPC that you will get billed on your NAT Gateway to send data out to the public internet and right back in to the same datacenter. There is simply no reason to not default that behavior to opt out vs opt in (via a VPC endpoint) beyond AWS profiting off of people's lack of knowledge in this realm. The amount of people who would want the current opt-in behavior is... if not zero, infinitesimally small.

[solatic]

It's a design that is secure by default. If you have no NAT gateway and no VPC Gateway Endpoint for S3 (and no other means of Internet egress) then workloads cannot access S3. Networking should be closed by default, and it is. If the user sets up things they don't understand (like NAT gateways), that's on them. Managed NAT gateways are not the only option for Internet egress and users are responsible for the networks they build on top of AWS's primitives (and yes, it is indeed important to remember that they are primitives, this is an IaaS, not a PaaS).

[bilalq]

Fine for when you have no NAT gateway and have a subnet with truly no egress allowed. But if you're adding a NAT gateway, it's crazy that you need to setup the gateway endpoint for S3/DDB separately. And even crazier that you have to pay for private links per AWS service endpoint.

[solatic]

There's very real differences between NAT gateways and VPC Gateway Endpoints.

NAT gateways are not purely hands-off, you can attach additional IP addresses to NAT gateways to help them scale to supporting more instances behind the NAT gateway, which is a fundamental part of how NAT gateways work in network architectures, because of the limit on the number of ports that can be opened through a single IP address. When you use a VPC Gateway Endpoint then it doesn't use up ports or IP addresses attached to a NAT gateway at all. And what about metering? If you pay per GB for traffic passing through the NAT gateway, but I guess not for traffic to an implicit built-in S3 gateway, so do you expect AWS to show you different meters for billed and not-billed traffic, but performance still depends on the sum total of the traffic (S3 and Internet egress) passing through it? How is that not confusing?

It's also besides the point that not all NAT gateways are used for Internet egress, indeed there are many enterprise networks where there are nested layers of private networks where NAT gateways help deal with overlapping private IP CIDR ranges. In such cases, having some kind of implicit built-in S3 gateway violates assumptions about how network traffic is controlled and routed, since the assumption is for the traffic to be completely private. So even if it was supported, it would need to be disabled by default (for secure defaults), and you're right back at the equivalent situation you have today, where the VPC Gateway Endpoint is a separate resource to be configured.

Not to mention that VPC Gateway Endpoints allow you to define policy on the gateway describing what may pass through, e.g. permitting read-only traffic through the endpoint but not writes. Not sure how you expect that to work with NAT gateways. This is something that AWS and Azure have very similar implementatoons for that work really well, whereas GCP only permits configuring such controls at the Organization level (!)

They are just completely different networking tools for completely different purposes. I expect closed-by-default secure defaults. I expect AWS to expose the power of different networking implements to me because these are low-level building blocks. Because they are low-level building blocks, I expect for there to be footguns and for the user to be held responsible for correct configuration.

[bilalq]

My objections here are in terms of how this manifests in billing. Especially when you consider the highway robbery rates for internet egress.

[solatic]

Again, you are dealing with low-level primitives. You can provision an EC2 VM with multiple GPUs at high cost and use it to host nginx. That is not a correct configuration. There are much cheaper ways available to you. It's ridiculous to imply that AWS shouldn't send you a higher bill because you didn't use the GPUs or that AWS shouldn't offer instances with GPUs because they are more expensive. You, the user, are responsible for building a correct configuration with the low-level primitives that have been made available to you! If it's too much then feel free to move up the stack and host your workloads on a PaaS instead.

[Dylan16807]

It being low level is not an excuse for systems that lead people down the wrong path.

And the traffic never even reaches the public internet. There's a mismatch between what the billing is supposedly for and what it's actually applied to.

> do you expect AWS to show you different meters for billed and not-billed traffic, but performance still depends on the sum total of the traffic (S3 and Internet egress) passing through it?

Yes.

> How is that not confusing?

That's how network ports work. They only go so fast, and you can be charged based on destination. I don't see the issue.

> It's also besides the point that not all NAT gateways are used for Internet egress

Okay, if two NAT gateways talk to each other it also should not have egress fees.

> some kind of implicit built-in S3 gateway violates assumptions

So don't do that. Checking if the traffic will leave the datacenter doesn't need such a thing.

[otterley]

This is the intended use case for S3 VPC Gateway Endpoints, which are free of charge.

https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpo...

(Disclaimer: I work for AWS, opinions are my own.)

[darkwater]

I think they know it. They are complaining it's not enabled by default (and so do I).

[otterley]

AWS VPCs are secure by default, which means no traffic traverses their boundaries unless you intentionally enable it.

There are many IaC libraries, including the standard CloudFormation VPC template and CDK VPC class, that can create them automatically if you so choose. I suspect the same is also true of commonly-used Terraform templates.

[hylaride]

As others have pointed out, this is by design. If VPCs have access to AWS resources (such as S3, DynamoDB, etc), an otherwise locked down VPC can still have data leaks to those services, including to other AWS accounts.

It's a convenience VS security argument, though the documentation could be better (including via AWS recommended settings if it sees you using S3).

[conradludgate]

I've been testing our PrivateLink connectivity at work in the past few weeks. This means I've been creating and destroying a bunch of VPCs to test the functionality. The flow in the AWS console when you select the "VPC and more" wizard does have an S3 Gateway enabled by default

[SOLAR_FIELDS]

The problem is that the default behavior for this is opt-in, rather than opt-out. No one prefers opt-in. So why is it opt-in?

[icedchai]

If it were opt-out someone would accidentally leave it on and eventually realize that entire systems had been accidentally "backed up" and exfiltrated to S3.

[SOLAR_FIELDS]

What? The same is possible whether it's opt-in or opt-out. It's just that if you have the gateway as opt-out you wouldn't also have this problem AND a massive AWS bill. You would just have this problem.

[Spivak]

The bad situation is if you created a VPC with no internet access but the hypothetical automatic VPC endpoint still let instances access S3. Then a compromised instance has a vector for data exfiltration.

[icedchai]

No, with opt-in the VPC subnet is secure by default. Someone has to explicitly allow access to S3 (or anything else.)

[otterley]

AWS VPCs are secure by default, which means no traffic traverses their boundaries unless you intentionally enable it.

[SOLAR_FIELDS]

"The door is locked, so instead of suggesting to the end user that they should unlock the door with this key that we know how to give the end user deterministically, we instead tell them to drive across town and back on our toll roads and collect money from it"

This has been a common gotcha for over a decade now: https://www.lastweekinaws.com/blog/the-aws-managed-nat-gatew...

[otterley]

Speaking solely on my own behalf: I don't know a single person at AWS (and I know a lot of them) who wants to mislead customers into spending more money than they need to. I remember a time before Gateway Endpoints existed, and customers (including me at the time) were spending tons of money passing traffic through pricey NAT Gateways to S3. S3 Gateway Endpoints saved them money.

[SOLAR_FIELDS]

Clearly you guys are aware of the problem though. I mean, every time this thing happens there's probably a ticket. I've personally filed one myself years ago when it happened to me. So why has the behavior not changed? You don't have to give up security to remove this footgun, it's possible to remove it and still make it an opt-in action for security purposes.

[hinkley]

Your job depends upon you misunderstanding the problem.

[afandian]

Having experienced the joy of setting up VPC, subnets and PrivateLink endpoints the whole thing just seems absurd.

They spent the effort of branding private VPC endpoints "PrivateLink". Maybe it took some engineering effort on their part, but it should be the default out of the box, and an entirely unremarkable feature.

In fact, I think if you have private subnets, the only way to use S3 etc is Private Link (correct me if I'm wrong).

It's just baffling.

[time0ut]

You can provision gateway endpoints for S3 and DynamoDB. They are free and considered best practice. They are opt-in though, but easy to enable.

[mdaniel]

And ECR, which I would guess impacts more folks than DynamoDB https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-e...

And, as as added benefit, they distinguish between "just pull" and "pull and push" which is nice

[afandian]

True, I forgot that. But depending on services you still have to have some Gateway and some Interface endpoints.

[dmart]

VPC endpoints in general should be free and enabled by default. That you need to pay extra to reach AWS' own API endpoints from your VPC feels egregious.

[otterley]

Gateway endpoints are free. Network endpoints (which are basically AWS-managed ENIs that can tunnel through VPC boundaries) are not free.

S3 can use either, and we recommend establishing VPC Gateway endpoints by default whenever you need S3 access.

(Disclaimer: I work for AWS, opinions are my own.)

[Hikikomori]

Why don't you have gateway endpoints for all your APIs?

[donavanm]

The original private endpoints implementation required meaningful work from the service teams (ec2 networking, s3, & ddb). It also changed how the "front end" API servers handled requests and how their infrastructure was deployed (at the time?). The newer LB/ENI style privatelink abstracts away _most_ of that "per service" implementation effort at the cost of more per-request/connection work fromthe virtual network. Hence why theres more support from other services, and it includes a cost.

[count]

The service teams don’t talk to each other…

[mdaniel]

I think that is by design https://konghq.com/blog/enterprise/api-mandate

[tux3]

That is price segmentation. People who are price insensitive will not invest the time to fix it

People who are probably shouldn't be on aws - but they usually have to for unrelated reasons, and they will work to reduce their bill.

[nbngeorcjhe]

> People who are price insensitive will not invest the time to fix it

This just sounds like a polite way of saying "we're taking peoples' money in exchange for nothing of value, and we can get away with it because they don't know any better".

[robertlagrant]

It's more like: we made loads of stuff super cheap but here's where we make some money because it scales with use.

[amenghra]

Price segmentation happens all the time in pretty much every industry.

[hinkley]

There’s an entire Pandora’s box of shitty things that happen in pretty much every industry. I don’t think you want to use that defense.

[happytoexplain]

Must be un-criticizable then?

[happytoexplain]

>People who are price insensitive will not invest the time to fix it

Hideous.

[kbolino]

The problem is that VPC endpoints aren't free.

They should be, of course, at least when the destination is an AWS service in the same region.

[edit: I'm speaking about interface endpoints, but S3 and DynamoDB can use gateway endpoints, which are free to the same region]

[otterley]

Gateway endpoints are free. Network endpoints (which are basically AWS-managed ENIs that can tunnel through VPC boundaries) are not free.

S3 can use either, and we recommend establishing VPC Gateway endpoints by default whenever you need S3 access.

(Disclaimer: I work for AWS, opinions are my own.)

[JoshTriplett]

That's fascinating! I hadn't found that in the documentation; everything seems to steer people towards PrivateLink, not gateway endpoints.

Would you recommend using VPC Gateway even on a public VPC that has an Internet gateway (note: not a NAT gateway)? Or only on a private VPC or one with a NAT gateway?

[otterley]

I recommend S3 Gateways for all VPCs that need to access S3, even those that already have routes to the Internet. Plus they eliminate the need for NAT Gateway traversal for requests that originate from private subnets.

[JoshTriplett]

> I recommend S3 Gateways for all VPCs that need to access S3, even those that already have routes to the Internet.

Fascinating. What's the advantage of doing that?

[donavanm]

It's a much more direct/efficient connection from the EC2 instance to the S3 storage servers through the virtual network layer. It reduces the network path/length through the AWS network _and_ removes the number of virtual network functions/servers (ala "LB") that your connections will traverse.

[paulddraper]

> everything seems to steer people towards PrivateLink, not gateway endpoints

Gateway endpoints only work for some things.

[Hikikomori]

Privatelink endpoints can be of type gateway or interface. Only gateway is free and only S3 and dynamodb supports it.

[kbolino]

Fair point, and valid for S3 (the topic at hand) and DynamoDB.

Other AWS services, though, don't support gateway endpoints.

[mdaniel]

https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-e...

~~I get the impression there are several others, too, but that one is of especial interest to me~~ Wowzers, they really are much better now:

  aws --region us-east-1 ec2 describe-vpc-endpoint-services | jq '.ServiceNames|length'
  459

If you're saying "other services should offer VPC Endpoints," I am 100% on-board. One should never have to traverse the Internet to contact any AWS control plane

[watermelon0]

Those are VPC endpoints, not gateway endpoints.

[kbolino]

Both interface endpoints and gateway endpoints are also called VPC endpoints. The former get distinct IP addresses in your VPC subnets while the latter get distinct entries in your VPC routing tables. They are even created with the same API call: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_C...

[paulddraper]

Well yeah that's the point....why route through the public internet.

[kbolino]

I doubt the traffic ever actually leaves AWS. Assuming it does make it all the way out to their edge routers, the destination ASN will still be one of their own. Not that the pricing will reflect this, of course.

The other problem with (interface) VPC endpoints is that they eat up IP addresses. Every service/region permutation needs a separate IP address drawn from your subnets. Immaterial if you're using IPv6, but can be quite limiting if you're using IPv4.

[immibis]

Sounds like a good reason to use IPv6.

[kbolino]

There were still a couple of services/features that choked on IPv6 last time I looked (1.5-2 years ago) but it works with most things and they do seem to be making progress on the others.

[paulddraper]

I don’t know any software that doesn’t work with IPv6 as of 2021.

Just some internet services that haven’t upgraded. (But fixed by NAT.)

[torginus]

If you had an ALB inside the VPC that routed the requests to something that lives inside the VPC, which called the AWS PutObject api on the bucket, would that still be the case?

[immibis]

A company making revenue is not stupid.