Driving HDMI is way harder then you would guess; extremely fast timing to hit. A naive framebuffer alone won't fit in RAM, whilst not strictly needed it would make it a hard challenge.
Yes a very fun FPGA project; but you will end up more expensive and more power hungry then a Pi Zero W which is more or less perfect for this use-case.