The main problem is simply that people really really really don't like losing their data after they saved it to disk. A simple app that corrupts its in-memory state once a year is probably acceptable. A filesystem that corrupts its on-disk state once a year is pure garbage. You basically need to aim for zero bugs.
How hard this is, it depends on the filesystem. Something like FAT, for example, is pretty much designed for ease of implementation, with few edge cases. Modern filesystems are not like that at all, the data structures are very complicated, so they must be extremely well tested before they are good enough to use. That would probably require an fsck to check for subtle inconsistencies; in the case of APFS you can use mine, but it's still very incomplete. Apple's published fsck is not very thorough.
As an example of the kind of problems to expect, I recall a bug in the Linux HFS+ driver. If you had a drive with lots of short filenames and lots of long filenames, and you started deleting the short filenames, eventually you would lose half of your files. This kind of things happen because HFS+ has variable-length keys in the index nodes of its trees, so deleting a record may trigger a complicated cascade of node splits. APFS inherited this feature, and it was very annoying to implement.
But HFS+ is very well documented; APFS is not, and that doesn't help.
It's worse than this. You not just need to aim for zero bugs, but zero bugs despite working with hardware that can degrade with use and who's firmware often does have bugs.
And yet this didn't stop Apple from automatically converting HFS+ volumes to APFS in iOS 10.3 and macOS 10.13.0 soon after the APFS beta dropped in macOS 10.12.5 and it didn't stop Apple from requiring APFS for all volumes in macOS 10.14+. Apple must have been pretty confident that APFS was working reliably to be so bold.
Not sure why you are telling me this, I don't know anything about Apple's internal development process. I assume they did run a lot of tests. But I recall at least one serious bug early on too[0].
> C being a shitty language that does not force or even encourage programmers to handle errors
I don't see where this is coming from. Most of the world's top filesystems are written in C, and they work just fine. Maybe other languages could get better results, but it's hard to say with so little data.
> implementation knowledge about file system technology is generally stuck in the 1990s
If you are talking about me, that might be true, I'm relatively new to this and still learning. But there's definitely people out there with some serious "implementation knowledge". And tools like xfstests did not exist in the 1990s, that makes a huge difference.
I'm curious why you think file systems written in C would somehow be better than any other type of program written in C. We have plenty of data suggesting C programs have more bugs and vulnerabilities than programs written in safer languages.
If you are thinking about rust, we don't have any data about it in the context of the kernel yet, not even for device drivers. It may prevent some exploitable bugs, but those aren't a big concern for filesystems - otherwise they wouldn't be put inside the kernel at all. The reality is, we don't know if it would help; and given how conservative we all are with our filesystem implementations (for good reasons), it's possible that we never will.
I'm not thinking about anything specific, I'm simply disputing your general claim that moving away from C would not help, when it has clearly helped in every single other domain of software development. I see no real reason why file systems should be any different, but clearly you do, so I was asking you why you think file systems would be different.
And for specific examples of options with better safety records, then sure Rust would be one possibility, as would Ada, or Frama-C if you need to stick with C.
How hard this is, it depends on the filesystem. Something like FAT, for example, is pretty much designed for ease of implementation, with few edge cases. Modern filesystems are not like that at all, the data structures are very complicated, so they must be extremely well tested before they are good enough to use. That would probably require an fsck to check for subtle inconsistencies; in the case of APFS you can use mine, but it's still very incomplete. Apple's published fsck is not very thorough.
As an example of the kind of problems to expect, I recall a bug in the Linux HFS+ driver. If you had a drive with lots of short filenames and lots of long filenames, and you started deleting the short filenames, eventually you would lose half of your files. This kind of things happen because HFS+ has variable-length keys in the index nodes of its trees, so deleting a record may trigger a complicated cascade of node splits. APFS inherited this feature, and it was very annoying to implement.
But HFS+ is very well documented; APFS is not, and that doesn't help.