[sovietmudkipz]

Checking out the website now. Looks enticing. Would a user of accelbyte multiplayer services still be in the business of knowing about underlying VMs? I caught some copy on the website that led me to question.

As a hobbyist part of me wants the VM abstracted completely (which may not be realistic). I want to say “here’s my game server process, it needs this much cpu/mem/network per unit, and I need 100 processes” and not really care about the underlying VM(s), at least until later. The closest thing I’ve found to this is AWS fargate.

Also holy smokes if you were a part of the team that architected this solution I’d love to pick your brain.

[maccard]

There’s a couple of providers that give you that kind of abstraction. Playfab is _pretty close_ but it’s fairly slow to ramp up and down. There is/was multiplay - they’ve had some changes recently and I’m not sure what their situation is right now. There’s also stuff like Hathora (they’re great but expensive).

At a previous job, we used azure container apps - it’s what you _want_ fargate to be. AIUI, Google Cloud Run is pretty much the same deal but I’ve no experience with it. I’ve considered deploying them as lambdas in the past depending on session length too…

[gcr]

Cloud Run tries to be this but every service like this has quirks. For example, GCR doesn’t let you deploy to high-CPU/MEM instances, has lower performance due to multi-tenant hosts, etc

[maccard]

But that’s not what OP asked for. They asked for

> As a hobbyist part of me wants the VM abstracted completely (which may not be realistic). I want to say “here’s my game server process, it needs this much cpu/mem/network per unit, and I need 100 processes” and not really care about the underlying VM(s), at least until later. The closest thing I’ve found to this is AWS fargate.

You can’t have on demand usage with no noisy neighbours without managing the underlying VMs.

I used hathora [0] at my previous job, (they’ve expanded since and I’m not sure how much this applies anymore) - they had a CLI tool which took a dockerfile and a folder and built a container and you could run it anywhere globally after that. Their client SDK contained a “get lowest latency location” that you could call on startup to use. It was super neat, but quite expensive!

[0] https://hathora.dev/gaming

[dijit]

That was was actually the original intent. If we scale to bare metal providers we can get much more performance. m

By making it an “us” problem to run the infrastructure at a good cost, and be cheaper then than AWS for us to run, meaning we could take no profit on cloud vms. making us cost competitive as hell.

[sovietmudkipz]

If I understand correctly you're saying you manage hardware yourself (colocate in a data center? Dedicated hosting?) and that gives you an edge in pricing. That's pretty cool, and I think I can see how it could be less expensive to purchase hardware & maintain it rather than renting that compute from a third party. There is obviously the tradeoff of then being responsible for capacity planning for the workloads supported among other downsides and maintaining hardware lifecycle but I wouldn't be surprised to hear this downside is overstated compared to benefits reaped.

[dijit]

Now I'm at a PC and can reply properly instead of typing drunkenly from my phone.

Ok, the idea was that what we really want is "ease of use" and "cost effective".

In game development (at least the gamedev I did) we didn't really want to care about managing fleets or versions, we just wanted to publish a build and then request that same build to run in a region.

So, the way I originally designed Accelbytes managed gameservers was that we treat docker containers as the distribution platform (if it runs in Docker it'll have all the bundled dependencies after all) and then you just submit that to us. We reconcile the docker image into a baked VM image on the popular cloud providers and you pay per minute that they're actively used. The reason to do it this way is that cloud providers are really flexible with the size of their machines.

So, the next issue, cost!

If we're using extremely expensive cloud VMs, then the cloud providers can undercut us by offering managed gameservers; worse, people don't compare devex of those things (though it's important to me when I was at AB); so we need to offer things at basically a neutral cost. It has to be the same price (or, ideally cheaper) to use Accelbyte's managed gameservers over trying to do it yourself on a cloud provider.. That way we guarantee the cloud providers don't undercut us: they wouldn't cannabalise their own margins on a base product like VMs to offer them below market rate.

So, we turn to bare-metal. We can build a fleet of globally managed servers, we can even rent them to begin with. By making it our problem to get good costs (because that pays for development), we are forced to make good decisions about CapEx vs OpEx, and it becomes "our DNA" to actually run servers, something most companies don't want to think about- but cloud costs are exorbitant and you need specialists to run them (I am one of those).

The bursty nature of games, seems like it fits best in a cloud, but you'll often find that naturally games don't like to ship next to each other, and the first weeks are the "worst" weeks in terms of capacity. If you have a live service game that sustains it's own numbers: that's a rarity, and in those cases it's even easier to plan capacity.

But if you build for a single burst, and you're a neutral third-party: you have basically built for every burst, and the more people who use you, the more profit you can make on the same hardware. -- and the more we buy, the better volume discounts we get, and the better we get at running things, etc;etc;etc.

Anyway, in order to make effective use of bare-metal, I wrote a gvisor clone that had some supervisor functionality, the basic gist of it was that the supervisor could export statistics of the gameserver, such as number of connections to the designated GS port (which is a reasonable default for player count) and information if it had completed loading (I only had two ways of being able to know this, one was the Agones RPC, the other was looking for a flag on disk... I was going to implement more), as well as ensuring the process is alive and lifecycling the process on death (collect up logs, crash dumps, any persistence, send it to the central backend to be processed). It was also responsible for telling the kernel to jail the process to the resources that the game-developer had requested. (So, if they asked for 2vCPU and 12G of ram, then, that's what they get).

It was also looking at NUMA awareness and CPU affinity, so, some cores would have been wasted (Unreal Engines server for example ideally likes to have 2 CPU Cores, where 1 is doing basically everything, and the other is about 20% used- theoretically you can binpack that second thread onto a CPU core, but my experience on The Division made me think that I really hate when my computer lies to me and plenty of IaaS providers have abstractions that lie.

I wrote the supervisor in Rust and it had about a 130KiB (static) memory footprint and I set myself a personal budget of 2 ticks per interval, but I left before achieving that one.

I could go into a lot of detail about it, I was really happy when I discovered that they continued my design. It didn't make me popular when I came up with it I think, they wanted something simple, and despite being developers, developing something instead of taking something off the shelf is never simple.

Also, they were really in bed with AWS. So anything that took away from AWS dominance was looked at with suspicion.

[sovietmudkipz]

Updates: meditating on the description of your Rust supervisor helped clarify my own approach.

- Go based supervisor daemon runs as a systemd service on the host. I configure it to know about my particular game server and expected utilization target. - The supervisor is responsible for reconciling my desired expectations (a count, or % of cpu/mem/etc so far) with spinning up game servers managed by systemd (since systemd doesn’t natively support this sort of dynamism + go code is super lean). - If I want more than one type of game server I imagine I could extend this technique to spinning up more than one supervisor but I’m keeping that in my back pocket for now. - I haven’t thought up a reason to, but my Go supervisor might want to read the logs of my game servers through journald.

For my purposes I’m not making a generic solution for unknown workloads like your Rust supervisor, which probably helps reduce complexity.

My workstation uses systemd so I can see my supervisor working easily. Real heckin’ neat.

[dijit]

nice, I really think you’ll go far with that approach on a single server.

My only advice, is capture stdout from the supervisor of the child process (gameserver) instead of putting an own dependency on journald: because everyone speaks stdout and you can later enrich your local metrics with it if its structured well, and forward it centrally.

[sovietmudkipz]

I really appreciate your perspective! Pretty clear you’re someone who digs deep, doesn’t accept constraints given (eg ‘everyone’ desired an AWS based solution but that didn’t narrow your exploration), and tries to be pragmatic. At least that’s my impression.

What you’ve stated so far is interesting. I’ve reading through some of your blog content too. I sent a LinkedIn connection request (look for Andrew).

Probing question: knowing what you know do you have any opinions (strong or otherwise) on what a solo dev might want to pursue? Just curious how you respond.