You need just the Linux kernel and Busybox in order to run Linux with a working shell. 2-4 MB all together I would say. My first Internet router ran a Linux from a 1.44MB disk 15 years ago.
You're in the right ballpark. I ported a network monitoring app to Linux and helped bring Linux up on a custom board a long time ago, and that device had 4MB for the kernel, shell/utilities and the application itself. It fit nicely.