[midoridensha]

>For example, suppose you want a database-like filesystem. Either you implement it in the kernel, and now your special apps barely work on anyone’s computers.

Filesystems already are databases. They organize and catalog data, and provide various other metadata about the data stored in them (creation and modification times, permissions, etc.).

Many people have invented various other databases, some SQL, others noSQL, which all require special apps to use. Many of these have been successful.

[josephg]

The filesystem is a crappy database. Its purely hierarchical nature requires dirty compromises for things like music libraries, where you want songs to be indexed both by song name and by album. Filesystems are also lacking atomic update mechanisms (transactions). There’s no way to enforce data integrity - files are weakly typed. You need fsync on linux to know that your data has been written at all - but fsync is horribly slow. And even fsync doesn't save you from data corruption due to skewed writes.

You can use a userland database on top of linux. But that has different properties from the OS’s actual filesystem for ecosystem reasons. A Postgres instance will never be a first class citizen on Unix. I can’t “cd” into a sql table in my terminal, or use sql queries to query procfs or /etc. What would it mean to pipe into a table? Does that abstraction even make sense? An operating system built on top of a database would be different from unix because the ecosystem of userland applications would evolve in a different direction. See, for example, HaikuOS.

I’m not saying it’s a good or a bad idea. But simply firing up mongodb on a Linux server isn’t the same thing as building the whole OS with a real database at its core.

[midoridensha]

>The filesystem is a crappy database. Its purely hierarchical nature

You realize the first databases were hierarchical, not relational, right? Filesystems do well enough in this regard. They're not meant to store tons of metadata, which varies depending on your application.

>requires dirty compromises for things like music libraries, where songs should be indexed both by song name and by album.

What about songs that aren't on albums? What about live performances? What about cover songs? Who gets the credit, the performer or the composer? People's opinions about these things keep changing, which is why the original MP3 tag format was so bad, and had to be replaced by a newer format. Imagine if we were all stuck, forever, with what some clueless people in 1995 thought was good enough for MP3 tag info, for all digital music.

And why should this info be in a filesystem anyway? Most files are not digital music.

>A Postgres instance will never be a first class citizen on Unix, so I can’t “cd” into a sql table in my terminal, or use sql queries to query procfs or /etc.

Right, because then the OS would have had to be designed for that kind of thing from the start, and you'd never be able to change it afterwards. This is why we keep things minimal at the lower levels, because then you can change things easily at the higher levels later on as needs change. Postgres itself has changed a lot in the last 10 years, adding lots of capabilities; if that were baked into the OS, that wouldn't have been so easy. And what if you decide you want something different from SQL? Sorry, you're stuck with it because it's baked into the OS, so now someone else is going to complain about how our ecosystem could evolve in a different way if we adopted some other type of database paradigm.

The benefits you claim just aren't worth the cost. It's easy enough to implement a database on top of a modern OS, and then use tools and applications designed for it to interact with it.

[josephg]

> You realize the first databases were hierarchical, not relational, right?

Yes. Obviously, the first databases humanity ever made were also the worst databases humanity ever made. With the possible exception of mongodb.

The choices Linux made in 1980 made sense at the time. But 40 years is a long time! It is as you say - databases have improved a lot in that time. I don’t know if Linux built on top of a modern database would be better or worse. How could we know unless somebody tries it?

[midoridensha]

>How could we know unless somebody tries it?

Because the effort to build such a thing would be gargantuan. Think of all the work that went into the Linux kernel plus all the userspace programs and applications on top of it: you want to recreate all that effort because of a hunch?

I'm sure some people with more expertise in theoretical CS than me can tell you better why this is a bad idea, but consider we already have databases now, and different databases work better for different tasks than others. How is baking a database into your filesystem going to compete? What if you pick the wrong one? What if it sucks for certain use-cases that current systems (filesystem+DB) work better at?

Every time someone's tried getting better efficiency by baking things in at a low level, it hasn't worked out too well, because by forcing a standard that way, it prevents innovation (e.g., with your DB-as-filesystem, when everyone decides they want to work with JSON right in their DB, it can't be done with yours because it wasn't designed that way and it can't be bolted on because it'll break things, but with Postgres it's easy to add in).

[josephg]

Doing a lot of work on a hunch that it might be better is the basis of all science. And all progress in general. Building a fully verified operating system kernel was a massive amount of work - but the SeL4 team still did it because they thought it was a good idea and got funding. (And as I understand it, their OS gets a lot of use in things the baseband chips in cell phones).

Yes, making a new toy operating system is a lot of work. But we don’t need to reinvent all of the software that has been built on Linux to tell if its a promising idea. Just enough to learn and see if the juice is worth the squeeze. And maybe have a little fun along the way.

In general I think it’s really sad how little innovation there is now in the OS space, simply because of how dominant Linux is and how much work it takes to make something yourself. How many good ideas are waiting in the wings because it would take too much effort to try them out? What a pity!

[midoridensha]

>In general I think it’s really sad how little innovation there is now in the OS space, simply because of how dominant Linux is and how much work it takes to make something yourself. How many good ideas are waiting in the wings because it would take too much effort to try them out? What a pity!

People are coming up with all kinds of innovations in computing, just not so much in the OS space because it's considered a solved problem. There's tons of stuff going on at much higher levels, and has been for a long time: virtualization, containerization, microservices, etc. The low-level building blocks are "good enough" for the higher-order things people want to try out now.

We've seen this in many domains: once you have something that works well enough, it's hard to justify effort to optimize it more, when there's other problems to be solved.