[pacala]

Containerization helps with one thing: end-to-end dependency hell management. You get the same executable artifact in prod and on every dev machine. You get to share arcane tricks required to bootstrap library X. You get to share the complete recipe of building your OS image. Hopefully, you pin versions so your build is not subject to the whims of upstream.

Kubernetes helps with one thing: taking your container and running it on a fleet of machines.

Building 18 services is an architectural choice made by the team. It has nothing to do with containerization or Kubernetes. For a single team, a monolith just works most of the time. You may consider multiple services if you have multiple [large] teams, think Search vs. Maps. Even then, consider the trade-offs carefully.

[scarface74]

I deploy code with all of the dlls in separate folders. The executables/services don't share any dlls. I kept asking the "consultants" who are trying to push us to Docker, what is the business value over raw executables + Nomad.

The build server creates one zip file that is stored as an artifact that gets decompressed and released in each environment - in a separate folder.

[bonesss]

> what is the business value over raw executables + Nomad.

It's not a given than any of the major business value generators are relevant to your shop, your domain, and your business demands. KISS is always good advice.

Low hanging fruit: Nomad (backed by HashiCorp), is a direct competitor to kubernetes (backed by google). One of those solutions is available turn-key on every major cloud provider and also the premiere Enterprise VM management solution. The other is called Nomad ;)

Raw executables pack up very nicely into containers, so if you're able to exist happily with just apps then just apps in containers won't change much (and therefore look like extra work)... For numerous domains raw executables are just a percentage of the deployment. Be it third-party apps/drivers that need to be installed, registry fixes, or whatever the Ops demands for server maintenance are a non-starter. And then things like load balancing and dynamic scaling pop up...

More importantly, for what I do, the binary validation of an immutable server in multiple zones is critical to ensuring security. Nothing can be changed, nothing shall change, and every point of customization will be scripted, or else it can't get near our data.

Cross platform and legacy scenarios are major players. More pressing, though, are the application level primatives that k8s provides in a cross-platform cross-cloud manner (which can also be federated...), so that your scaling story is adequately handled and your local apps become much more robust and cloud-native.

Bottom line: it's not a given that k8s will improve your life, here and now, apps + Nomad is viable. For the broader eco-system though the "other stuff" in k8s, and the rigidity/stability of dependency graphs in containers, are clear value drivers and highly meaningful.

[scarface74]

Yeah KISS was very important when I first started working at my current company. I was hired to setup a modern development shop with three database developers who were just learning C#, no source control, no CI/CD, basically no development "process", they ran a lot of things manually.

I was going to be introducing a lot of changes.

Every decision I made was based on keeping things as simple as possible to keep them from getting frustrated. If that weren't the case, I would have gone straight to Docker. Knowing that I might need that flexibility later but didn't want to commit right now, I chose Nomad because I knew it could both handle phase 1 and allow us to move to Docker once appropriate.

But now, that we are in AWS, there is a big push to get to the next level of cloud maturity - not just moving VMs to the cloud, but how to take advantage of a "cloud first" approach and actually take advantage of some of the features that AWS offers.

So in that vein, there is a need for Docker to go "serverless". Lambda is not an option - we have long running processes.

Even when we do go to Docker, we will probably make a transistion from Nomad straight to Amazon's Fargate.

I see a path where we move from .Net 4.6 to .Net Core and Docker with Nomad to Fargate.

The only issue with Fargate for us now is the added complexity that Fargate only supports Linux containers. I don't know how much of a lift that would be. Theoretically it shouldn't be much with a pure .Net Core solution.

[yebyen]

You are remarkably well positioned to take advantage of any solution from what you've told.

My group is skipping Kubernetes to go straight to Fargate and we are... not so we'll positioned as you happen to be.

Much to my chagrin, as a newbie to AWS who has loads of homegrown experience with Kubernetes and its predecessors (Fleet, etcd) I am wholly reliant on the AWS solutions engineers we have in-house to help me navigate this thing via CloudFormation and friends, it's too much for one person to figure out in 20 hours during a pilot/assessment study.

I am an application developer who learned Kubernetes in his free time over the past 3 years because it was free. There are thousands of us, with computers in our basements, learning these systems on our own, with no institutional support. Sure, I needed lots of help, but I didn't have to spend money on cloud instances just to learn, or be sure to remember to terminate them when the experiment was over.

By contrast, AWS has only just made Amazon Linux 2 available to run on your own machines less than two months ago. There is still no way to set up ECS or Fargate on our own metal, and probably never will be, because Amazon does not see a reason for it.

Vendor lock-in is real and it has casualties! There are real negative effects that you don't see. If you say "I would not hire someone like you because you have specific skills I won't take advantage of," you have to ask yourself is that because of something that I've done or is it something that Amazon is doing.

[pjmlp]

I look at these "solutions" and they don't seem to add much to how application servers work.

[bonesss]

If you're juggling multiple incompatible versions of application servers across multiple platforms across multiple datacenters and multiple cloud providers with multiple dev teams... you're gonna see some real value to those kinda of "solutions". It's not random that this tech is coming from cloud leaders and Entprise shops, and they don't address problems common to development, they address problems common to cloud apps and cloud heavy Enterprise shops.

I think Assembler looks like ass and it doesn't add much to how I want to program... It's still frequently used, though, because it solves problems other than the ones I have.

[scarface74]

We have a bunch of apps that run based on some type of external event - a time interval (Nomad supports cron like scheduling across an app pool), a file coming in, an external event, etc.

We submit a job via the api and it runs the job on whatever server has available resources. We specify the mininimum amount of RAM and CPU needed to run a job. If too many jobs are queued on a regular basis, we can either add more RAM or CPU to an existing instance or add another instance and install a Nomad agent.

Yes I know k8s can do the same thing but we don't have to use Docker, we can though.

[pjmlp]

My employer already had that problem solved in 2006, thanks to JEE.

An EAR packaged with everything needed by the application.

Each service, or micro-service as it is fashionable now, got their own EAR.

Deployment of UNIX based OS, JEE application server, Oracle and respective EAR packages, done.

[pacala]

That is a solid solution. As long as everyone in the ecosystem is on JVM, more power to you. If, for example, one needs to interface with some cutting edge DL modules written in Python, one needs something else. The transitive closure over "something else" is called "containerization".

PS. Maybe "EAR" also supports Python. But then I'd argue "EAR" is a "container".

[takeda]

With python you then use wheels + virtualenv, for Ruby you can use bundler. Each language has this issue solved.

Using containers is essentially:

- uh, I have problem with these dependencies, dealing with RPMs is such a nightmare, I need to generate OS specific packages and there might be conflicts with existing packages that are used by the system...

- oh I know, let just bundle our app with the entire OS!

[Apaec]

You could also use nix to handle all languages dependencies(including OS packages) and avoid the complexity of handling disparate package managers.

[closeparen]

Languages have the issue solved until your library is just bindings to a C shared library, in which case you also need the appropriate OS package.

[aldanor]

You build this shared library as a relocatable conda package and add an explicit dependency in your Python (conda) package / problem solved.

[raesene9]

I'd suggest that if containers end up looking like "the entire OS" then that misses a lot of the benefit of containers.

A container image should be "the bare minimum that allows the application to run".

[takeda]

Yes, I do agree with that.

If they are minimalistic and hold the app then this makes sense and then containers are essentially an unified packaging format that is accepted on "serverless" public cloud. This provides a value because you can then easily run your application on multiple providers where it is cheaper at given time.

I'm thinking that in the future your IDE could just compile your project into a single file that you then upload it anywhere and just run.

But the docker was promoted as something different with the union fs, nating etc. That works fine for development but it's a bit problematic operationalizing it.

[takeda]

This is what I noticed as well. Most of things containers are advocated for are already solved.

The selling point of containers is to solve certain issues (seems like package management, removing dependency on the OS etc are the most popular).

To me it looks like instead fixing the actual issues, we are taking a blanket covering all of that crap and building our beautiful solution on top of that. We have a beautiful world with unicorns on top of a dumpster fire of mixing system dependencies with our application dependencies.

Also yesterday found something amusing a coworker was complaining that putting a small app into a base container resulted with image that was almost 1GB in size, compared to ~50MB when using a minimalistic one. When asked why not just use the minimalistic one I learned that it was mandated to use the standard image for everything.

To me this is absurd since by doing that aren't we essentially making a full circle?

[Apaec]

Absolutely. I think the actual issue are the OSes directory structure(FHS for example) that impedes having libs/packages isolated and coexisting with their different versions.

Containers add a heavy abstraction on top of that. For me the simpler/better dependency management solution is nix.

[johannes12]

No, and it makes perfect sense because 1. Container size is a minor issue, docker images are layered so you only fetch the diff of what's on top anyway 2. Standardization simplifies knowledge sharing, i.e. someone else can help you

[icedchai]

As usual, "it depends." JEE isn't magic and app servers have their own issues. I think you're better off packaging Java web apps as self contained fat jars (see Dropwizard, Spring Bootstrap)

[Apaec]

For the dependency hell management part, nix is a solution that operates at a lower cost level of abstraction, it doesnt emulate the whole OS(avoiding overhead) and keeps dependencies isolated at the filesystem level(under /nix).

I think that for reproducible development environments is a much simpler solution.