For me, the cost would be the loss of ecosystem— log collection, monitoring, cores + rr recordings, being able to connect with gdb and/or sniff around the live system from a familiar userland, etc.
These concerns are reduced when you're at the thousands-of-machines scale since at that point you probably already have a somewhat hand-rolled solution for this kind of thing anyway.
We already solved those problems with containers (I know, half of HN doesn’t like containers and everyone should manage heavy VMs or bare metal machines just like our ancestors did). Notably logs and metrics are exfiltrated, detailed logging and monitoring largely obsolete debuggers for production (indeed, they aren’t even balle into the final image nor are they installable since the app user oughtn’t be root). These practices seem pretty portable to the unikernel model and they don’t require any hand-rolled workarounds or thousands of machines of scale.
Did we? I mean in prod I still frequently pop a shell in a pod, apk install some tool and reproduce issues outside of the app, sure I could trawl through gbs of istio logs or whatever but it would probably at least double incident resolution if I had no userland available on the machine with the problem…
My understanding of the best practices is that shelling into prod is a breakglass only. Developers need approval to get escalated permissions to shell into prod in the first place. Further, containers shouldn’t run as root (security) so I don’t know how you would install software anyway. Logs and metrics should similarly be queryable via some central log explorer service like CloudWatch, Splunk, Prometheus, or even kubectl+grep. You shouldn’t have to manually page through GBs of logs.
Our images are often pretty stripped down (coreutils at most, often just a Go binary and some certs), so there aren’t many debugging tools available.
This might make our time to resolution slightly higher, but it keeps our incident count quite a lot lower because we very rarely need to break glass in the first place (this means you have to establish norms for logging, instrumentation, and tests).
You are correct though that there is very much a lack of traditional user-land and that is on purpose. A lot of people expect these things to be similar to an alpine-like distro and that is just the wrong picture to have. Think more heroku/serverless - it is all about the application. So you wouldn't ssh into your application. Yes, there is a underlying kernel but it is more about freeing the developer to think more about their application and less about deploying the server itself.
This isn't true of all unikernels. Ours lets you run regular programs like perf, gdb inside the unikernel. (Of course optionally, you can go "pure unikernel" if you want)
These concerns are reduced when you're at the thousands-of-machines scale since at that point you probably already have a somewhat hand-rolled solution for this kind of thing anyway.