If you want to build a game without using an engine, but don't want to deal with talking to the hardware on your own, SDL is there.