I would recommend "Architecture of a Database System" by Joe Hellerstein, Micheal Stonebraker, and James Hamilton [0] as a start if you want formal perspective without getting too much into theory.
As a sidenote, Michael Stonebreaker himself is nowdays building VoltDb which is a memory-db that he thinks is much more evolved than relational systems. I havn't used it but it looks interesting.
I did try OrigoDb which is an open source .NET based memory db and it worked great for my needs and the queries are really fast.