[ilaksh]

What's the best provider to self-host Firecracker? I feel that AWS is not a safe or cost-effective option for a self-funded startup or small business. Although is anything cost effective anymore? Hetzner just had a massive price hike.

Part of it might just be that I am old and inflation is catching up with my understanding of prices.

But as far as AWS I still have to say no thanks. Imagine some group actually started using my hosted AI agent service for something compute and network intensive. It could turn into $2000 overnight and if I didn't account for one of the numerous types of AWS charges, I might have only collected $500 for credits purchases.

Or it could easily be ten times that. But who am I kidding. No one is going to use my agents. So it doesn't matter if it's gvisor or Firecracker or whatever.

[crabmusket]

Not Firecracker, but Incus's system containers seem like a good middle ground between Docker and VMs.

https://linuxcontainers.org/incus/docs/main/explanation/cont...

[nyrikki]

Are you looking for highly ephemeral nodes, where you are writing automation that will use the API to orchestrate it? Or do you just want small microVMs that you launch and kill?

Firecracker just has a ReSTful unix socket with a defined API and launches KVM vms with limited options.

For custom SMB I still think libvirt is a lower entry cost and may have transferable use cases to longer lived VMs, so you can just launch a qemu microvm[0] and use virsh and/or libvirt xml to set up the networking.

The ~400ms boot time of a qemu microvm vs ~120ms for firecracker may not be an issue for some loads, but qemu will also allow you a bit more density of placement than firecracker. qemu microvms will use a bit more memory individually, but they will also tend to use less real system memory with a larger number of microVMs.

It is all tradeoffs, and kata containers are yet another option that may apply depending on your use case.

You can run your own firecracker or qemu/kvm microvms on most instances that allow nested hypervisors, or on a local host. If cost containment is critical to you this is one possible way forward.

Really it just depends on if you want/need ReSTful control, or need to support short lived serverless functions, or if CLIs fit better and you many want to support full VMs.

They both are just Virtual Machine Monitors that targeted different use cases and decided on different tradeoffs.

Just be careful about hosting traditional containers and microVMs on the same system, that config is going to be problematic do to fundamental reasons that are too complex to properly address here.

[0] https://www.qemu.org/docs/master/system/i386/microvm.html

[ilaksh]

Thanks. I just looked into qemu microvms. Might be an option but I already have gvisor set up.

[coppsilgold]

The simplest worthwhile DIY sandbox you can have is to layer two tools: bwrap and gvisor.

    bwrap args -- gvisor args do args -- /path/sandboxee args

bwrap will set up the environment and then gvisor elevates it into a true sandbox.

Standalone gvisor (not the 'do' subcommand) used to be a mess with the OCI json requirement, but recently they began work on presenting their own bwrap interface (likely to pursue AI agent uses) though I wouldn't use it myself yet.

People often look down on gvisor because they think it's some kind of syscall filter, it is not. It can use one of ptrace, seccomp or even KVM to intercept ALL syscalls and service them with it's own logic (which is in Go). Basically it's a VMM and kernel in one.

[eperot]

Any reason why you wouldn't use gVisor's bwrap interface yet? We're working on it precisely to make DIY sandboxing on Linux as easy as possible in order to get Linux-sandboxing-at-home to mature beyond the current syscall-filter-and-namespaces duct tape stage, so I'm curious to know what you'd like to see.

[coppsilgold]

It just didn't seem fully baked yet, the 'do' subcommand works fine while the 'bwrap' alias has this problem: `bash: cannot set terminal process group (1): Not a tty`. When executing 'bash -li'. Also the EROFS feature of 'do' should probably be included in 'bwrap', it can be useful. Include overlay options.

Also some things you can do to make gvisor better are Wayland passthrough, vulkan support (or virtio native context). Being able to get gvisor to populate a network interface inside itself through a 'passt' (or 'containers/gvisor-tap-vsock') socket on the host would also be ergonomic. All of those are available on 'muvm' (based on libkrun) which if you have the time to set up is the next step in DIY sandboxing of graphical apps as well. See: <https://git.clan.lol/clan/munix>

[eperot]

Thanks. We're working on rootless network setup to make `runsc do --rootless` work with networking enabled when `passt` is installed right now. See issue #13337 (yes that's a cool issue number) which should unblock this.

The tty issue is known, should be fixed soon too, though contributions welcome as it sounds like it should be simple fix and we love more contributions :)

FWIW, X11 apps work well, I have a personal hacky project in which I've been running Librewolf in gVisor, with the window being reflected as a native Wayland window. It uses `Xvfb -fbdir` aimed at a bound tmpfs mount to get a shared memory region containing the window's pixel data which can be read directly from out of the sandbox, has Pulseaudio audio passthrough, and a socket server passing through mouse/keyboard events to make the window interactive. Works smoothly even for YouTube playback, and I successfully played a game of Unreal Tournament 2004 at 24fps in it, with no noticeable mouse/keyboard latency :) We're basically making baby steps to get there less hackily.

Thanks for the feedback!

[coppsilgold]

That's good to hear! Hopefully the passt approach you are pursuing will include the ability to use an existing passt socket and not just launch one for you.

Wayland is tricky because there are memory buffers being shared between the compositor and the client. crosvm (also by google) adopted 2 custom solutions to it of which one got merged into mainline.

Achieving audio passthrough is trivial as it's just a unix socket. `-host-uds=all`

[eperot]

That's the approach I initially took, but experienced some combination of noticeable stuttering and latency regardless of which buffering strategy I tried... Had to switch to a shared memory ring buffer, along with some adaptive playback speed shenanigans (sometimes imperceptibly speeding up playback when falling behind production of audio samples, sometimes imperceptibly slowing down when there's less than a few milliseconds' worth of samples left in the ringbuffer), in order to achieve actually-gapless playback.

[dbmikus]

Why do you want to self-host vs. using one of the many providers out there?

Daytona, E2B, OpenComputer, Freestyle, Blaxel, Vercel, Modal, Cloudflare, Tensorlake, Superserve, etc. etc.

Some of them work by pre-purchasing credits, so you can control the blast radius of spend.

Also, if you want a more embedded sandbox runtime as a library instead of a daemon + REST API, you can check out libkrun (and friendly layers on top of it like https://microsandbox.dev/ and https://smolmachines.com/)

[khurs]

self host = better spec machine for same price.

[rvz]

Even with the Hetzner price increase, it is still far cheaper than all of them with self-hosting.

[vidarh]

Hetzner is still cheap compared to AWS.

[magnio]

Yeah, the big 3 cloud markup is so high that most VPS providers can hike price 10x and they are still cheaper.

[vmg12]

You can't run firecracker on AWS.

[karavelov]

Even on older system types, you can provision .metal sizes and run anything on them.

[tmzt]

You can with the newer instances that suport nested VM. There was a recent story about this here https://news.ycombinator.com/item?id=48556561.

[otterley]

Yes you can.

[alexellisuk]

For self-hosting, have a look at what we're building with SlicerVM.com (disclosure: I'm the founder). Also runs just as well on Apple Silicon.

We run quite a few Slicer instances on mini PCs and Ryzen builds - also on Hetzner (and yes ouch 120 EUR / mo up to ~ 550 EUR / mo for 16core / 128GB RAM feels almost unfair)

[ilaksh]

Interesting. How does this compare to Firecracker? Also PhoenixNap looks really interesting. Do you happen to know if Linux software compatibility holds up on Ampere? 80 cores for $400 a month seems pretty good.

[Multicomp]

This reminds me of Fly.io's model off the top of my head, though its not a self-hosted firecracker as such.

[ilaksh]

I specifically complained to a fly.io staff on here about their "gotcha, b*tch" usage based pricing which they basically copied from AWS, and they stood by it and other people here backed them up. No one is giving me a pile of free money, so I can't risk that kind of thing.

[tptacek]

Exactly what did we copy from AWS here? You could get a long way in our decisionmaking process generally by just consciously avoiding what AWS does.

[ilaksh]

The short version is it seems like a big "gotcha" that there is no way to limit bandwidth or spending on that or other resources ahead of time, and that might be a deliberate business model that is more aimed at well-funded startups or large companies that are monitoring costs much less closely than an individual or small business.

It's not necessarily too hard to just not dynamically spawn a bunch of machines, but the bandwidth one is going to sneak up on people.

[tptacek]

Lots of people want limits. They might make sense for something like Sprites, where the end-users are often (but not always) individual developers. They're terrible for hosting fixed-function applications. The real gotcha is having limits, because that's the host effectively taking your app down for you.

I know talk is cheap, but I've been in the room for every one of these discussions over the last 6 years at Fly.io, and if we could have come up with a system to make limits workable, we would have done it. Charging for stuff you don't want is bad business, and we make our money from happy, growing customers (the open secret of hosting is that a huge chunk of usage is basically a loss leader search for a much smaller number of ultra-profitable customers).

These pricing models --- at least outside of AWS (I'm not cynical about them but their incentives are different from indies) --- are not meant to fuck you.

[CuriouslyC]

Cloudflare is cost effective for certain types of workloads, I've heard of businesses getting surprisingly far on the $5/mo worker plan.

[Multicomp]

At my day job, workers and sqlite-backed durable objects that quickly hibernate and quickly resume are quite nice, I prefer that to standard lambda.