Hacker News new | ask | show | jobs
by iam-TJ 1096 days ago
I have something similar and have had for several years. Just needs a recent-ish kernel and decent distro!

I try to avoid virtual machines unless there is an absolute need to virtualise the hardware as well - in most cases single kernel, multiple containers is preferred.

Specifically, I use Debian with systemd-nspawn unprivileged (user namespaced) containers (managed by machinectl), defined and built using mkosi as BTRFS subvolumes (a BTRFS mounted on /var/lib/machines/ allows nspawn to create copy-on-write containers from a 'template' container).

The nice thing about nspawn is being able to create simple declarative configurations with private namespaces. With a private network namespace it has the concept of 'zones' which are containers that share the same (private) bridged network. systemd-networkd has built-in configs for both the host and containers to do automatic configuration.

I use this to create per-domain (e.g. example.com) zones with multiple function-specific containers (one for router/firewall, another for DHCP and DNS, others for single-sign-on, database, web, mail, and whatever else is required). I use veth to connect the zone selectively to the host interface public bridge interface providing external access if necessary.

This allows 'development' 'test' and 'production' domains using mkosi to define and build each repeatedly as needed. In the non-public domains the zone can be configured to use the same public IP addresses as production will use without it leaking outside the zone.

I use IPv6 only in the zones with both ULAs and global prefixes and DNS64/NAT64 in the 'router' container if the zone needs to make outbound connections (to IPv4 destinations).

I use netfilter rules applied to each zone's bridge interface to impose firewall restrictions.

Using veth pairs one can connect two containers (or zones) directly rather than using the common zone bridge interface.

I do dev work on the host itself; if you want to partition application usage I'd suggest keeping it simple by using multiple user accounts, one per 'task' so $USER configs and so on are all kept separate.

In the 'development' containers I do things like bind-mounting the base source-code directory, or project-specific directory, into the container. I do all source-code work on the host but build and test in the appropriate container.

For example I have 'linux-builder' container where I do out-of-tree kernel builds for multiple architectures. The container has the required crossbuild tools installed. Recently this has been used to build latest mainline kernel for amd64, arm64, and powerpc64 (for the Xbox 360!).