Hacker News new | ask | show | jobs
by doctor_eval 869 days ago
I’m in the very unusual situation of being tasked to set up a self-sufficient, local development team for a significant national enterprise in a developing country. We don’t have AWS, Google or any other cloud service here, so getting something running locally, that they can deploy code to, is part of my job. I also want to ensure that my local team is learning about modern engineering environments. And there is a large mix of unrelated applications to build, so a monolith of some sort is out of the question; there will be a mix of applications and languages and different reliability requirements.

In a nutshell, I’m looking for a general way to provide compute and storage to future, modern, applications and engineers, while at the same time training them to manage this themselves. It’s a medium-long term thing. The scale is already there - one of our goals is to replace an application with millions of existing users.

Importantly, the company wants us to be self sufficient. So a RedHat contract to manage an OpenShift cluster won’t fly (although maybe openshift itself will?)

For the specific goals that we have, the broad features of Kubernetes fit the bill - in terms of our ability to launch a set of containers or features into a cluster, run CICD, run tests, provide storage, host long- and short lived applications, etc. But I’m worried about the complexity and durability of such a complex system in our environment - in the medium term, they need to be able to do this without me, that’s the whole point. This article hasn’t helped me feel better about k8s!

I personally avoided using k8s until the managed flavours came about, and I’m really concerned about the complexity of deploying this, but I think some kind of cluster management system is critical; I don’t want us to go back to manually installing software on individual machines (using either packaging or just plain docker). I want there to be a bunch of resources that we can consume or grow as we become more proficient.

I’ve previously used Nomad in production, which was much simpler than K8s, and I was wondering if this or something else might be a better choice? How hard is k8s to set up today? What is the risk of the kind of failures these guys hit, today?

Are there any other environments where I can manage a set of applications on a cluster of say 10 compute VMs? Any other suggestions?

Without knowing a lot about their systems, I suspect something like Oxide might be the best bet for us - but I doubt we have the budget for a machine like that. But any other thoughts or ideas would be welcome.

6 comments

(I work at Oxide)

> I doubt we have the budget for a machine like that.

Before even thinking about budget,

> for a significant national enterprise in a developing country.

I suspect we just aren't ready to sell in your country, whatever it is, for very normal "gotta get the product certified through safety regulations" kinds of reasons. We will get there eventually.

buuuuut also,

> Are there any other environments where I can manage a set of applications on a cluster of say 10 compute VMs? Any other suggestions?

Oxide would give you those VMs, but if you want orchestration with them, you'd be running kubes or whatever else, yourself, on top of it. So I don't think our current product would give you exactly what you want anyway, or at least, you'd be in the same spot you are now with regards to the orchestration layer.

Hey Steve, thank you for this comment, I did wonder if Oxide systems did container orchestration; now I know :)

Totally get it re certification etc. There is probably some kind of bilateral standards arrangement with one of the neighbouring countries but I agree with your general thrust - we’re a long way from the point where that’s actually a consideration.

You're welcome, and good luck with the project.
Have you checked out Proxmox? 16 large servers in a cluster config could possibly be powerful enough for your needs (Proxmox lets you cluster 16 servers in each cluster); if you need more, split up each part of the services into 16-server chunks.

EDIT: seems the limit is no longer 16 per cluster https://pve.proxmox.com/wiki/Cluster_Manager

Thanks, I'm checking out Proxmox now!
I've really been liking what I've been seeing with the [Ubuntu micro cloud](https://canonical.com/microcloud) product. It's basically a well coordinated effort of a deployment on lxc/lxd cluster using a (micro) ceph and ovn implementations. I think I like it because it attacks the problem at a level I can understand. (Proxmox does lxc/lxd etc also and has already been mentioned).

Again this is really more of a vmms setup so you need to orchestrate on top. (So perhaps your nomad cluster can sit on this (assuming there is some customer/org infrastructure splitting)

I personally would love to setup a new infra with Oxide systems. But for 10 VMs anything will work. If you've looked into Oxide maybe you will like SmartOS to bootstrap a small infra. Otherwise Nomad, proxmox, LXC, even VMWare is fine. If I think I need something more "serious" (understand supported/scalable with some robust cloud-like api) I would look into cloudstack from apache, which seems a lot cleaner than openstack [Removing the part about VMWare as I saw the comment about self-sufficiency]
Thanks. I expect we will eventually have a rack of servers - currently they operate their applications from a couple of full and very busy blade chassis - but I’m aiming low for the time being since we’re just getting started.

I’ll check out cloudstack. Thanks!

You can't go wrong with docker swarm. I'd say upto 100 nodes should be no problem and a hundred node cluster is no joke.

Most companies don't reach there.

In general however, post Nomad licensing - there's a gap for an open source cluster orchestrator that is not k8s and isn't Nomad.

I had a pretty bad run with docker swarm a few years ago - the network stack was flakey as hell, we had to manually restart nodes quite often, and it resulted in customer downtime on multiple occasions. So I probably wouldn’t go there, even though I liked it in theory.

I also thought it was EOL, did something change?

Well Amazon CEO himself said, there is no shortcut to experience. I am sure gaining experience in developing infrastructure solution will give you respectable return in long term. Of course Cloud vendors will be happy to sell turnkey solutions to you though.
Yeah - I do agree, over the long term, that deploying this internally would be the best outcome for them, and would give them some great skills. But this article did put the Fear in me a little.

One of the main drivers of the project is to reduce our reliance on software vendors and move to open source solutions, so a hybrid/on-prem cloud vendor is probably not on the cards for us either.