[rjst01]

Let me give you an alternative perspective.

My startup pays Docker for their registry hosting services, for our private registry. However, some of our production machines are not set up to authenticate towards our account, because they are only running public containers.

Because of this change, we now need to either make sure that every machine is authenticated, or take the risk of a production outage in case we do too many pulls at once.

If we had instead simply mirrored everything into a registry at a big cloud provider, we would never have paid docker a cent for the privilege of having unplanned work foisted upon us.

[hkwerf]

I get why this is annoying.

However, if you are using docker's registry without authentication and you don't want to go through the effort of adding the credentials you already have, you are essentially relying on a free service for production already, which may be pulled any time without prior notice. You are already taking the risk of a production outage. Now it's just formalized that your limit is 10 pulls per IP per hour. I don't really get how this can shift your evaluation from using (and paying for) docker's registry to paying for your own registry. It seems orthogonal to the evaluation itself.

[hedora]

The big problem is that the docker client makes it nearly impossible to audit a large deployment to make sure it’s not accidentally talking to docker hub.

This is by design, according to docker.

I’ve never encountered anyone at any of my employers that wanted to use docker hub for anything other than a one-time download of a base image like Ubuntu or Alpine.

I’ve also never seen a CD deployment that doesn’t repeatedly accidentally pull in a docker hub dependency, and then occasionally have outages because of it.

It’s also a massive security hole.

Fork it.

[donmcronald]

> This is by design, according to docker.

I have a vague memory of reading something to that effect on their bug tracker, but I always thought the reasoning was ok. IIRC it was something to the effect that the goal was to keep things simple for first time users. I think that's disservice to users, because you end up with many refusing to learn how things actually work, but I get the sentiment.

> I’ve also never seen a CD deployment that doesn’t repeatedly accidentally pull in a docker hub dependency, and then occasionally have outages because of it.

There's a point where developers need to take responsibility for some of those issues. The core systems don't prevent anyone from setting up durable build pipelines. Structure the build like this [1]. Set up a local container registry for any images that are required by the build and pull/push those images into a hosted repo. Use a pull through cache so you aren't pulling the same image over the internet 1000 times.

Basically, gate all registry access through something like Nexus. Don't set up the pull through cache as a mirror on local clients. Use a dedicated host name. I use 'xxcr.io' for my local Nexus and set up subdomains for different pull-through upstreams; 'hub.xxcr.io/ubuntu', 'ghcr.xxcr.io/group/project', etc..

Beyond having control over all the build infrastructure, it's also something that would have been considered good netiquette, at least 15-20 years ago. I'm always surprised to see people shocked that free services disappear when the stats quo seems to be to ignore efficiency as long as the cost of inefficiency is externalized to a free service somewhere.

1. https://phauer.com/2019/no-fat-jar-in-docker-image/

[naikrovek]

> I'm always surprised to see people shocked that free services disappear when the stats quo seems to be to ignore efficiency as long as the cost of inefficiency is externalized to a free service somewhere.

Same. The “I don’t pay for it, why do I care” attitude is abundant, and it drives me nuts. Don’t bite the hand that feeds you, and make sure, regularly, that you’re not doing that by mistake. Else, you might find the hand biting you back.

[leoqa]

Block the DNS if you don’t want dockerhub images. Rewrite it to your artifactory.

This is really not complicated and your not entitled to unlimited anonymous usage of any service.

[a022311]

That will most likely fail, since the daemon tries to connect to the registry with SSL and your registry will not have the same SSL certificate as Docker Hub. I don't know if a proxy could solve this.

[flubbergusto]

This is supported in the client/daemon. You configure your client to use a self-hosted registry mirror (e.g. docker.io/distribution or zot) with your own TLS cert (or insecure without if you must) as pull-through cache (that's your search key word). This way it works "automagically" with existing docker.io/ image references now being proxied and cached via your mirror.

You would put this as a separate registry and storage from your actual self-hosted registry of explicitly pushed example.com/ images.

It's an extremely common use-case and well-documented if you try to RTFM instead of just throwing your hands in the air before speculating and posting about how hard or impossible this supposedly is.

You could fall back to DNS rewrite and front with your own trusted CA but I don't think that particular approach is generally advisable given how straightforward a pull-through cache is to set up and operate.

[amluto]

This is ridiculous.

All the large objects in the OCI world are identified by their cryptographic hash. When you’re pulling things when building a Dockerfile or preparing to run a container, you are doing one of two things:

a) resolving a name (like ubuntu:latest or whatever)

b) downloading an object, possibly a quite large object, by hash

Part b may recurse in the sense that an object can reference other objects by hash.

In a sensible universe, we would describe the things we want to pull by name, pin hashes via a lock file, and download the objects. And the only part that requires any sort of authentication of the server is the resolution of a name that is not in the lockfile to the corresponding hash.

Of course, the tooling doesn’t work like this, there usually aren’t lockfiles, and there is no effort made AFAICT to allow pulling an object with a known hash without dealing with the almost entirely pointless authentication of the source server.

[ndriscoll]

Right but then you notice the failing CI job and fix it to correctly pull from your artifact repository. It's definitely doable. We require using an internal repo at my work where we run things like vulnerability scanners.

[hkwerf]

> since the daemon tries to connect to the registry with SSL

If you rewrite DNS, you should of course also have a custom CA trusted by your container engine as well as appropriate certificates and host configurations for your registry.

You'll always need to take these steps if you want to go the rewrite-DNS path for isolation from external services because some proprietary tool forces you to use those services.

[martinsnow]

You don't have to run docker. Containerd is available.

[mgiampapa]

It's trivial to audit a large deployment, you look at dns logs.

[btown]

This is Infamous Dropbox Comment https://news.ycombinator.com/item?id=9224 energy

[Dylan16807]

They didn't say it's easy to fix, just detect.

[kazinator]

Is there no way to operate a caching proxy for docker hub?!

[johntash]

There are quite a few docker registries you can self-host. A lot of them also have a pull-through cache.

Artifactory and Nexus are the two I've used for work. Harbor is also popular.

I can't think of the name right now, but there are some cool projects doing a p2p/distributed type of cache on the nodes directly too.

[themgt]

I don't really get how this can shift your evaluation from using (and paying for) docker's registry to paying for your own registry

Announcing a new limitation that requires rolling out changes to prod with 1 week notice should absolutely shift your evaluation of whether you should pay for this company's services.

[tomnipotent]

Here's an announcement from September 2024.

https://www.docker.com/blog/november-2024-updated-plans-anno...

[themgt]

You're right, that is "an announcement":

At Docker, our mission is to empower development teams by providing the tools they need to ship secure, high-quality apps — FAST. Over the past few years, we’ve continually added value for our customers, responding to the evolving needs of individual developers and organizations alike. Today, we’re excited to announce significant updates to our Docker subscription plans that will deliver even more value, flexibility, and power to your development workflows.

We’ve listened closely to our community, and the message is clear: Developers want tools that meet their current needs and evolve with new capabilities to meet their future needs.

That’s why we’ve revamped our plans to include access to ALL the tools our most successful customers are leveraging — Docker Desktop, Docker Hub, Docker Build Cloud, Docker Scout, and Testcontainers Cloud. Our new unified suite makes it easier for development teams to access everything they need under one subscription with included consumption for each new product and the ability to add more as they need it. This gives every paid user full access, including consumption-based options, allowing developers to scale resources as their needs evolve. Whether customers are individual developers, members of small teams, or work in large enterprises, the refreshed Docker Personal, Docker Pro, Docker Team, and Docker Business plans ensure developers have the right tools at their fingertips.

These changes increase access to Docker Hub across the board, bring more value into Docker Desktop, and grant access to the additional value and new capabilities we’ve delivered to development teams over the past few years. From Docker Scout’s advanced security and software supply chain insights to Docker Build Cloud’s productivity-generating cloud build capabilities, Docker provides developers with the tools to build, deploy, and verify applications faster and more efficiently.

Sorry, where in this hyped up marketingspeak walloftext does it say "WARNING we are rugging your pulls per IPv4"?

[sbarre]

That's some cherry-picking right there. That is a small part of the announcement.

Right at the top of the page it says:

> consumption limits are coming March 1st, 2025.

Then further in the article it says:

> We’re introducing image pull and storage limits for Docker Hub.

Then at the bottom in the summary it says again:

> The Docker Hub plan limits will take effect on March 1, 2025

I think like everyone else is saying here, if you rely on a service for your production environments it is your responsibility to stay up to date on upcoming changes and plan for them appropriately.

If I were using a critical service, paid or otherwise, that said "limits are coming on this date" and it wasn't clear to me what those limits were, I certainly would not sit around waiting to find out. I would proactively investigate and plan for it.

[noname120]

The whole article is PR bs that makes it sound like they are introducing new features in the commercial plans and hiking up their prices accordingly to make up for the additional value of the plans.

I mean just starting with the title:

> Announcing Upgraded Docker Plans: Simpler, More Value, Better Development and Productivity

Wow great it's simpler, more value, better development and productivity!

Then somewhere in the middle of the 1500-word (!) PR fluff there is a paragraph with bullet points:

> With the rollout of our unified suites, we’re also updating our pricing to reflect the additional value. Here’s what’s changing at a high level:

> • Docker Business pricing stays the same but gains the additional value and features announced today.

> • Docker Personal remains — and will always remain — free. This plan will continue to be improved upon as we work to grant access to a container-first approach to software development for all developers.

> • Docker Pro will increase from $5/month to $9/month and Docker Team prices will increase from $9/user/month to $15/user/mo (annual discounts). Docker Business pricing remains the same.

And at that point if you're still reading this bullet point is coming:

> We’re introducing image pull and storage limits for Docker Hub. This will impact less than 3% of accounts, the highest commercial consumers.

Ah cool I guess we'll need to be careful how much storage we use for images pushed to our private registry on Docker Hub and how much we pull them.

Well it's an utter and complete lie because even non-commercial users are affected.

————

This super long article (1500 words) intentionally buries the lede because they are afraid of a backlash. But you can't reasonably say “I told u so” when you only mentioned in a bullet point somewhere in a PR article that there will be limits that impact the top 3% of commercial users, then 4 months later give a one week notice that images pulls will be capped to 10 pulls per hour LOL.

The least they could do is to introduce random pull failures with an increasing probability rate over time until it finally entirely fails. That's what everyone does with deprecated APIs. Some people are in for a big surprise when a production incident will cause all their images to be pulled again which will cascade in an even bigger failure.

[tomnipotent]

Documentation on usage and limits from December 2024.

https://web.archive.org/web/20241213195423/https://docs.dock...

Here's the January 21st 2025 copy that includes the 10/HR limit.

https://web.archive.org/web/20250122190034/https://docs.dock...

The Pricing FAQ goes back further to December 12th 2024 and includes the 10/HR limit.

https://web.archive.org/web/20241212102929/https://www.docke...

I haven't gone through my emails, but I assume there was email communication somewhere along the way. It's safe to assume there's been a good 2-3 months of communication, though it may not have been as granular or targeted as some would have liked.

[rad_gruchalski]

Hey, can I have your services for free because I also feel entitled?

[guhidalg]

I mean, there has never not been some issue with Docker Desktop that I have to remember to work around. We're all just collectively cargo culting that Docker containers are "the way" and putting up with these troubles is the price to pay.

[gcapu]

If you offer a service, you have some responsibility towards your users. One of those responsibilities is to give enough notice about changes. IMO, this change doesn't provide enough notice. Why not making it a year, or at least a couple of months? Probably because they don't want people to have enough notice to force their hand.

[Aurornis]

> Why not making it a year, or at least a couple of months?

Announced in September 2024: https://www.docker.com/blog/november-2024-updated-plans-anno...

At least 6 months of notice.

[jmb99]

I use docker a few times a week and didn’t see that. Nor would I have seen this if I hadn’t opened HN today. Not exactly great notice.

[rad_gruchalski]

If you had an account you’d receive an email back in September 2024. I have received one…

[SkiFire13]

This is not announcing the 10 (or 40) pull/hr/ip

[lukeschlather]

Didn't they institute more modest limits some time ago? Doesn't really seem like this is out of nowhere.

[hirako2000]

Yes they have. They are reducing the quota further.

[cratermoon]

They altered the deal. Pray they don't alter it any further.

[rad_gruchalski]

s/Pray/Pay/

[grayhatter]

What principal are you using to suggest that responsibility comes from?

I have a blog, do I have to give my readers notice before I turn off the service because I can't afford the next hosting charge?

Isn't this almost exclusively going to effect engineers? Isn't it more of the engineer's responsibility not to allow their mission critical software to have such a fragile signal point of failure?

> Probably because they don't want people to have enough notice to force their hand.

He says without evidence, assuming bad faith.

[4ndrewl]

You don't. You have responsibility towards your owners/shareholders. You only have to worry about your customers if they are going to leave. Non-paying users not so much - you're just cutting costs now zirp isn't a thing.

[gcapu]

If this was a public company I would put my tin foil hat and believe that it's a quick buck scheme to boost CEO pay. A short sighted action that is not in the shareholders interest. But I guess that's not the case? Who knows...

[4ndrewl]

At this stage of the product lifecycle, free users are unlikely to ever give you money without some further "incentives". This shouldnt be news by now, especially on HN.

If you're production service is relying on a free-tier someone else provides, you must have some business continuity built in. These are not philanthropic organisations.

[popalchemist]

It's bait and switch that has the stakes of "adopt our new policy, that makes us money, that you never signed up for, or your business fails." That's a gun to the head.

Not an acceptable interaction. This will be the end of Docker Hub if they don't walk back.

[hirako2000]

To think of malice is mistaken. It's incompetence.

Docker doesn't know how to monetize.

[withinboredom]

Yes. But they are paying for this bandwidth, authenticated or not. This is just busy work, and I highly doubt it will make much of a difference. They should probably just charge more.

[rad_gruchalski]

They charge more.

[londons_explore]

> take the risk of a production outage in case we do too many pulls at once.

And the exact time you have some production emergency is probably the exact time you have a lot of containers being pulled as every node rolls forward/back rapidly...

And then docker.io rate limits you and suddenly your 10 minute outage becomes a 1 hour outage whilst someone plays a wild goose chase trying to track down every docker hub reference and point it at some local mirror/cache.

[wat10000]

I mean, don’t build your production environment to rely on some other company’s free tier, and then act surprised when they throttle high usage.

And yes, you’re still using the free tier even if you pay them, if your usage doesn’t have any connection to your paid account.

[prepend]

Companies that change their free tiers also change their paid tiers.

I just don’t build my environment to rely on unstable companies.

That’s sort of the comedy of second order effects as by reducing the amount of free stuff, I think Docker will end up reducing their paid customers.

[rad_gruchalski]

> If we had instead simply mirrored everything into a registry at a big cloud provider, we would never have paid docker a cent for the privilege of having unplanned work foisted upon us.

Indeed, you’d be paying the big cloud provider instead, most likely more than you pay today. Go figure.

[zmgsabst]

I’d you’re using popular images, they're probably free.

https://gallery.ecr.aws/docker/?page=1

[rad_gruchalski]

Please reread the comment I replied to.

[orochimaaru]

They should have provided more notice. Your case is simply prioritizing work that you would have wanted to complete anyway. As a paying customer you could check if your unauthenticated requests can go via specific outbound IP addresses that they can then whitelist? I’m not sure but they may be inclined to provide exceptions for paying customers - hopefully.

[rjst01]

> Your case is simply prioritizing work that you would have wanted to complete anyway

It's busy-work that provides no business benefit, but-for our supplier's problems.

> specific outbound IP addresses that they can then whitelist

And then we have an on-going burden of making sure the list is kept up to date. Too risky, IMO.

[troutwine]

> It's busy-work that provides no business benefit, but-for our supplier's problems.

I dunno, if I were paying for a particular quality-of-service I'd want my requests authenticated so I can make claims if that QoS is breached. Relying on public pulls negates that.

Making sure you can hold your suppliers to contract terms is basic due diligence.

[rjst01]

It is a trade-off. For many services I would absolutely agree with you, but for hosting public open-source binaries, well, that really should just work, and there's value in keeping our infrastructure simpler.

[cpuguy83]

This was announced last year.

[icehawk]

This sounds like its only talking about authenticated pulls:

> We’re introducing image pull and storage limits for Docker Hub. This will impact less than 3% of accounts, the highest commercial consumers. For many of our Docker Team and Docker Business customers with Service Accounts, the new higher image pull limits will eliminate previously incurred fees.

[cpuguy83]

Go look at the wayback for the exact page the OP is linking to.

[fennecbutt]

So it goes. You're a business, pay to make the changes. It's a business expense. Docker ain't doing anything that their agreements/licenses say they can't do.

It's not fair, people shout. Neither are second homes when people don't even have their first but that doesn't seem to be a popular opinion on here.

[jdhendrickson]

Devsec/ops guy here, the fact that you were pulling public images at all ever is the thing that is insane to me.

[rjst01]

Why? We are running the exact same images that we would be mirroring into and pulling from our private registry if we were doing that, pinned to the sha256sum.

[cyanydeez]

You can setup your own registry. You're complaining about now having to do your own IT.

this isn't a counterpoint is rewrapping the same point: free services for commercial enterprise is a counterproductive business plan

[vv_]

How can you make Docker pull debian:latest from your own registry instead of the official Docker registry, without explicitly specifying <my_registry>/debian:latest?

[cyanydeez]

Uh.

https://stackoverflow.com/questions/33054369/how-to-change-t...

[josteink]

> If we had instead simply mirrored everything into a registry at a big cloud provider

You would have had to authenticate to access that repo as well.

[rjst01]

Amazon ECR for instance provides the option to host a public registry.

[wiether]

> Data transferred out from public repositories is limited by source IP when an AWS account is not used.

https://aws.amazon.com/ecr/pricing/?nc1=h_ls

> For unauthenticated customers, Amazon ECR Public supports up to 500GB of data per month. https://docs.aws.amazon.com/AmazonECR/latest/public/public-s...

I don't see how it's better.

[a022311]

`mirror.gcr.io` works fine for many popular images on Docker Hub.

[lowercased]

Wouldn't they get a choice as to what type of authentication they want to use then? I'd assume they could limit access in multiple ways, vs just the dockerhub way.

[jjfanboy]

I just cannot imagine going into public and saying, roughly the equivalent of I want free unlimited bandwidth because I'm too lazy to do the very basics of managing my own infra.

> If we had instead simply mirrored everything into a registry at a big cloud provider, we would never have paid docker a cent for the privilege of having unplanned work foisted upon us.

I mean, if one is unwilling to bother to login to docker on their boxes, is this really even an actual option? Hm.

[SSLy]

> mirrored everything into a registry at a big cloud provider

https://cloud.google.com/artifact-registry/docs/pull-cached-...

[dbalatero]

You might try complaining and see if they give you an extension.