[taeric]

It is hard not to build this impression of containers. Worse, it seems that the docker image format is massive compared to what most war files were like. This is annoying not just in terms of raw bytes to move around, but in taking stock of what is getting moved around.

My team originally pitched how docker solved much of the dependency upgrade management by having layers for each major set of dependencies. That ignored the fact that upgrading a layer is not really something you do.

So, then you can go around the path of coordinating many containers communicating with each other. That works, but in that world, things really don't seem any easier than the earlier alternatives. Harder, in many ways.

Don't get me wrong, the momentum and raw money being put into containers certainly paints it as the future. It just feels like lying to say that they have even come close to parity with what we were capable of not that long ago.

[dnomad]

Exactly. Docker files are worse than war files in pretty much every way.

The economics are interesting though. Computing resources are super cheap and getting more cheap everyday. (Well except for RAM!) The waste produced by containers means says, yeah, give each application its own app server. Its own web server, its own JDBC driver, even its own JDK. We'll throw it all on the cloud and run it for a few cents each hour.

The problem is that testing isn't cheap. This is where microservices and containers and the all this craziness will fall down. Now you've got dozens and dozens of applications all running with their own application servers, JDBC drivers, and their own JDK. This is a combinatoric explosion in your testing surface.

The true value of the application server approach is that it forced applications to conform to a clear contract. Once your application conformed to this contract it could be thrown over the wall into a full-time ops organization that could transparently deploy, monitor and manage the app.

The container madness will come to an end. People will realize that giving developers the keys to the kingdom and total freedom over their application is a terrible idea. (And smart developers don't want total freedom.) What will emerge will be something much more interesting: a hybrid approach where applications can be bundled into into artifacts that fully express their dependencies and containers can be annotated in a way that fully express their capabilities.

Containers provide isolation and immutable infrastructure and this is good. Appservers provide standardization, specialization and separation of responsibilities and this is also good. There's no reason why we can't enjoy both.

[tyldum]

We use containers to reduce the testing surface and enforce standards. And if course to ease testing as we now transition the same build through dev, staging and prod.

The devs just have a single base container to work off from and we use CICD to further enforce whatever we need to.

It also makes it easier for devs to propose changes, as we know we can consistently apply them.

[haskellandchill]

You can use them to do that but is it working? That was the point, I also agree that no, it is not working and has rather introduced several layers with painful and hard to debug failure modes.

[taeric]

A problem comes down to answering questions such as "exactly what versions of X are installed in our beta stack?" Usually trivial if X is part of your application. Much more complicated if X is some lib that is on the system/image.

[mmt]

> People will realize that giving developers the keys to the kingdom and total freedom over their application is a terrible idea. (And smart developers don't want total freedom.)

Sadly, I don't think it's actually developers that were pushing for this "freedom" (with which comes responsibility, hence it makes sense that it would be unwanted), as such.

Rather, I place the blame with managers and with (fellow) sysadmins for not embracing, empowering, or forcing (whatever it takes) the aspects of the original DevOps cultural movement to have Ops bring the equivalent [1] of a production environment to every developer.

> Containers provide isolation and immutable infrastructure

I thought they weren't even (necessarily) immutable. If not, then maybe they're merely immutable-through-practice, but that's as unsatisfying as configuration management systems creating reproducible deployments that lack traceability [2].

[1] In terms of tooling for things like builds, deployment, and management (especially of dependencies), not necessarily full hardware capacity, although at one place I did provide each developer with a copy of yesterday's production database on their own, personal bare-metal database server, less "beefy" than the production hardware.

[2] is that the right word? something like chain-of-custody for code and/or configuration.

[kinghajj]

Nothing prevents one from having a container with many WAR files, so that there's only a single JVM for multiple applications.

[dnomad]

The entire exercise is about expressing dependencies. (The real third hard problem in computer science. Only idiots fall for off-by-1 errors.)

Containers are one way to force an application to explicitly declare all of its dependencies. The problem is the application then also bundles those dependencies. There's not much room for control or intermediation. Developers have total freedom to do all sorts of wackiness. (And oh the wackiness I've seen in containers. When the cat is away...) What's needed is a richer (and much more compact) mechanism for applications to express all their dependencies.

Another way to understand the issue to understand the underlying economics at work here. Container economics proposes that computing resources are so cheap that, sure, let the developers go crazy and do whatever they want. But those same developers must be on the hook for all operational concerns hence "DevOps." And this will scale until one day you look around and you realize that anarchy never works, not even a little bit.

Now there's a lot of room to fall here. RedHat is in the business of selling compute and storage resources. It's not clear that even if they could make their products more efficient they would. But as long as compute resources are cheap and getting cheaper you can let people go wild and RedHat will make a lot of money and everybody will be happy. For an organization the true costs will emerge in a very subtle manner: the productivity of the developers (already hard to measure) will ultimately fall. That's because instead of actually solving actual business problems their devs are deploying dozens and dozens of polyglot microservices and trying to figure out why on earth microservice #28 that Bob, who left the company a year ago, decided to write in a version of lisp that he invented crashes every day at noon.

But hey, at least Kubernetes makes it easy to deploy everything!

[parasubvert]

> There's not much room for control or intermediation.

FWIW there are container-based app platforms that do allow you to swap out filesystem layers to update dependencies and to remove control from developers by having a standardized containerizer that has extension hooks but can't be mucked with at the lowest levels. This is how Cloud Foundry works for example, or Heroku.

[smarterclayton]

For example, OpenShift (Kubernetes + stuff also from Red Hat) exposes this pattern like both heroku and cloud foundry by

A) focusing on having standard base images controlled by ops

B) encouraging combining source code / built artifacts as a layer on those base images

C) giving controls to ops so that the only images users could build or run must be built with A/B above.

In that mode containers are less wasteful because you can share the base image across every host (or rebuild everything centrally), and all that gets downloaded to a host is the source code top layer. Which is roughly indistinguishable from the lambda runtime and how it accesses the code to execute.

[taeric]

Don't most virtualized hosts already solve the "less wasteful" point? I get that hypervisors and friends aren't bulletproof, either; but I have fewer ways to shoot yourself with those than I have with containers.

More, you have to worry about the container host anyway, so you haven't removed the need to maintain a host. You've just added to it the need to maintain the rest of the container infrastructure as well as your application stuff.

For places that have not solved the "virtual hosts should be virtually free" problem, containers are quite welcome. You can get going with them quite quickly. If you have already solved that problem, they can look an awful lot like just more work.

[taeric]

To be fair, it is more likely that random microservice #28 is running fine, but folks don't like it and are uncomfortable modifying it to add a variation to one of its rules. Thus giving birth to microservice #28'. :) (Please read this tongue in cheek.)

[parasubvert]

> Exactly. Docker files are worse than war files in pretty much every way.

They focus at far too low level a problem.

> The problem is that testing isn't cheap. This is where microservices and containers and the all this craziness will fall down. Now you've got dozens and dozens of applications all running with their own application servers, JDBC drivers, and their own JDK. This is a combinatoric explosion in your testing surface.

I fail to see how this more difficult from 95% of existing enterprise systems that run various JDK versions, various WebSphere versions, JDBC drivers etc.

The difference with containers is that vendoring your dependencies makes all that explicit. Whether you choose to stay on old versions is a mistake you can still make (and will make if you leave container creation and patching in the hands of dev teams).

That said, since when does any organization build horizontal testing libraries, like, "test X version of JDBC", so they can standardize on a single driver across all projects? It's never done. Projects are all over the place with their dependencies unless there is institutionalized forcing through Maven repos or what not to block the download of old/insecure versions.

There is no combinatorial problem here, each test suite needs to unit test the individual service and then test the API contract of the service. If you let your dependencies decay you're accepting major security and maintenance risk, but you were in the WAR file world too.

> The true value of the application server approach is that it forced applications to conform to a clear contract. Once your application conformed to this contract it could be thrown over the wall into a full-time ops organization that could transparently deploy, monitor and manage the app.

LOL, I really think you're exaggerating. I've worked with WebLogic, WebSphere, IIS, JBoss, Tomcat, Django, Rails, Node, you name it over the years, and while some orgs got close in the Java world, this was mostly a pipe dream that never happened in any cost effective manner as a standard practice.

> The container madness will come to an end. People will realize that giving developers the keys to the kingdom and total freedom over their application is a terrible idea. (And smart developers don't want total freedom.)

This I agree with. It's all far too low level.

This really was Docker, Inc.'s fault IMO, their marketing message was to empower developers to get ops out of the way by building these containers that would magically just be run as opaque lego blocks. That was the hype everyone on HN was drooling over in 2013. Turns out it's not quite that simple.

[dionian]

So just reuse the common layers and only have your WAR file layer changing

[taeric]

The problem is when there is a vulnerability in some part of an earlier layer. You have to rebuild the entire stack.

I honestly don't think this is necessarily a terrible thing. But, the idea that your common layers are stable is a dangerously bad assumption.

[cpuguy83]

What are you defining as stable? Never changing? That's not something that should even hinted at.

The fact that you can login to a VM and update a package does not make the system stable either (and of course you can actually do this with running containers as well). Add to that you still need to restart running applications to take advantage of the package update (assuming the packages is a shared lib).

Meanwhile you can push new base images, automatically trigger rebuilds and roll out the update when ready.

[taeric]

I suspect we mostly agree with each other. My experience with folks in containers so far has proven to be that this is often hoped, though.

Specifically, we had devs talking about how we wouldn't have to worry about system patching anymore, because the containers would take care of that. With no answer for how we trace versions and patches through our systems.

If you are already tooled enough such that you can completely redeploy a full stack easily without worrying about some in place modifications, the difference between a VM and a container are relatively minimal, all told. Especially since you have to be ready to pull down the host of the containers anyway.

[cpuguy83]

The main difference is the container is focused on an application and a VM is focused on a machine.

Generally the patching problem can be solved with image scanning, of which there are tools out there to deal with this, both FLOSS and pay for.

[xj9]

the linux world is slowly (and badly) recreating technology that has existed in FreeBSD and Solaris/Illumos for decades.

[grapeli23]

You probably never heard or used - Linux VServer. Available from the later version of the 2.4 kernel.

https://en.wikipedia.org/wiki/Linux-VServer

[xj9]

> 2008

FreeBSD jails date from 2000

Solaris zones date from 2004

https://us-east.manta.joyent.com/bcantrill/public/ppwl-cantr...

https://us-east.manta.joyent.com/bcantrill/public/ppwl-cantr...

[grapeli23]

2003 http://linux-vserver.org/Overview#History

In the Linux distribution that I used (PLD), the first vserver support took place in january 2004. https://github.com/pld-linux/kernel/commit/5be58c1bcc5568676...

Maybe a little earlier because the util-vserver showed in Nov 2003. https://github.com/pld-linux/util-vserver/commit/c4036d6e748...

[taeric]

Be fair here. The software world is slowly recreating technology. Being even more fair, they do often add to it. Usually UX capabilities, but sometimes full on features. Yes, often the features require more resources than were available before. Such that previous versions couldn't have offered some of these things.

That said, it is not limited to any one set of our industry. It is likely not even limited to our industry.

[redbeard0x0a]

Software getting recreated, sometimes without the security promises (linux containers as they started vs. solaris zones) the previous solution had. To be fair, Google didn't really need the security in linux containers, they didn't run multi-tenant workloads, so it wasn't their top concern when submitting code to the linux kernel...

We like building new things, from different angles, but in the end it seems like everything cycles through the same ideas.

[taeric]

Completely agreed. It is hilarious because we all preach "don't reinvent the wheel" at the same time that our interviews are basically "can you implement this ridiculously low level wheel" on top of "come here where we are simply using the latest rims!" :)

[tomsthumb]

Which technologies were those?

I’m only vaguely familiar with FreeBSD, and not at all with Solaris.

[devonkim]

FreeBSD has had jails and Solaris has had zones (with a Linux compatible zone as well aka branded zone). Given Illumos takes a lot of its roots from Solaris, it also has LX branded zones.

[pjmlp]

And HP-UX/Tru64 before them.

We were deploying into HP Vaults in 2000.

[ecnahc515]

Yeah, except even jails has gaps, like for example; shared memory isn't jailed.