I've only had contact with DBs during my university years so please bear with me, but at least to me (mostly a systems programmer who played around with functional and logic programming languages) SQL seems, I dunno.. very crude? For instance it seems to me that PROLOG is a lot better at querying/defining relational facts.
Also every time I looked into SQL DBs I felt uncomfortable having to patch together SQL queries as strings and compile them at runtime. Why can't I define a DB schema in my compiled programming language and have it produce a typesafe query that can execute immediately? I know there's wrappers that help you define queries in a typesafe manner, but afaik these still generate query strings in the background.
Few people, when they say "SQL is amazing!", mean the language.
The language is ok minus, it's usable and isn't a problem center. String parsing isn't expensive enough for anyone to replace, and there are many benefits to the 100% language decoupling it ensures.
What we mean is "RDBMSs are amazing!", and they are. Humanity has spent a lot of resources on their design and evolution, making them into systems that efficiently solve your hard problems years before you first find out you have them.
I'd love to have the ability to write a direct query plan, for the rare occasion where the query planner does something stupid. But, ya SQL is great for the majority of situations.
AIUI, you can have a key-value store is an SQL RDBMS: Just make a big bunch of two-column -- key and value -- tables.
Also, again AIUI, you can build an RDBMS on top of a key-value store: "Just" build a whole lot of key-value pairs where the "value" is a reference to another key more often than not. (Perhaps not just for actual references to other related tables -- or what would be another table in a real RDBMS -- but for each separate key-value store that would just be another column in a real RDBMS. Then I suppose your primary ("key column") key-value pair would have a lot of rows for each key; one for each "column"?)
I suspect most real-world usage of key-value stores in one fashion or another approaches the above... And then the vaunted "simplicity" has gone out the window (and made SPLAT! on the ground far below).
But by then you're so [invested in | married to | sunk cost-fallacied with] your key-value store that changing to a real RDBMS is nigh impossible.
So better start with a real RDBMS from the beginning.
You are absolutely right! They seam easier to reason about. But relational databases actually are simpler to reason about when you compare equivalent feature sets. I've seen people argue it is just a key value store, so it is dead simple, right? But then they didn't just use key value store features. They used or implemented themselves joins, transactions, consistency constraints under parallel modifications, etc. But this is not the system for which they determined that it is simple.
But even for simple use cases durability might be a requirement. And that is not so simple to get right.
You're right too, it's hard to know from the start what you will need, and your needs will often grow to the point that you'll need a full relational database.
Also every time I looked into SQL DBs I felt uncomfortable having to patch together SQL queries as strings and compile them at runtime. Why can't I define a DB schema in my compiled programming language and have it produce a typesafe query that can execute immediately? I know there's wrappers that help you define queries in a typesafe manner, but afaik these still generate query strings in the background.