A recent (2014) and intriguing concept for making a small computer that feels like it was made in the past is the PDP-11/Hack, which only uses a DEC J-11 microprocessor, some RAM, and a common UART.
This being the PDP-11, even if you don't have any RAM, I'm sure the registers are mapped into memory somewhere contiguous where you can still run code from them :)
I got mine for just below $40, from what I believe (though one can never be certain) a reliable seller. Still haven't decided how I will proceed, I want to use it for a real project and not just as a decoration.