[pseudosavant]

This immediately made me nostalgic for BeOS's BeFS or Windows Longhorn's WinFS database filesystems, and how this kind of thing would have fit them perfect. So much cool stuff you could do with vectors for everything. Smart folders that include files for a project based on a description of the project. Show me all of my config files for appXYZ. Images of a black dog at the beach. At the OS-level for any other app to easily tap into.

I'd be surprised if cloud storage services like OneDrive don't already do some kind of vector for every file you store. But an online web service isn't the same as being built into the core of the OS.

[perone]

I share the same feeling, I think filesystems will have to reinvent themselves given the pace of how useful ML models became in the past years.

[didgetmaster]

I built a local object store that was designed to replace file systems. You can create hundreds of millions of objects (e.g. files) and attach a variety of metadata tags to each one. A tag could be a number, string, or other data type (including vector info). Searches for objects with certain tags is exceptionally fast.

I invented it because I found searching conventional file systems that support extended attributes to be unbearably slow.

[tugdual]

Got a demo ?

[didgetmaster]

Tons of demo videos on my YouTube channel. Free beta available for download on my website. Links in my profile.

[p_ing]

WinFS wasn't a file system laid down on hardware, it was just a SQL database that stored arbitrary data.

[aforwardslash]

Most modern filesystems aren't really "laid down on hardware", but on top of a volume manager; thats how both attached disks (think SAN or EBS volumes) and RAID works. Most filesystems behave exactly like an optimized database index, but lacking the extra fields; also, some databases support raw devices directly(using mapped volumes), no need for filesystems. The line between both is mostly a blur.

[didgetmaster]

I think that is one of the main reason it failed to launch. It was just too easy for the metadata stored in the separate database to become out of sync with the actual file data.

Microsoft saw the tech support nightmare this could generate, and abandoned the project.

[pseudosavant]

They just weren't able to pull it off for whatever reason. I actually ran BeOS as my daily driver for quite a while (way) back in the day. BeFS was genuinely amazing, and not something I've seen replicated elsewhere yet. There hasn't really been anything interesting done in filesystems used by users on devices in a really long time.

[aforwardslash]

BeOS was mostly a single-user system, and filesystem was optimized for that use case. Having to consider that an opened file for reading may be modified by a remote system running with a different user complicates things, so while I do agree that BeFS was quite ahead of his time, it focused on a somewhat specific use case; At the time, you wouldn't expect that you could eg. correctly backup your email client data without shutting down the application.

[aforwardslash]

It failed because it was slow and provided no obvious benefit to the average user, other than wasting disk space. There is a fair amount of tricks and optimizations in NTFS to balance speed of access and shadow copy mechanisms, in a pre-SSD era, where the average random throughput of a consumer-grade SATA disk was single digit MB/s

[p_ing]

It was abandoned due to The Cloud. There was no need for WinFS as a tech when you could store everything in The Cloud.

It was also complex, ran poorly, and would have required developers to integrate their applications.

Microsoft had long solved the problem of blobs and metadata in ESE and SharePoint's use of MS SQL for binary + metadata storage.

[WalterGR]

> it was just a SQL database that stored arbitrary data.

I mean, for some definitions of “just”, “SQL database”, and “arbitrary data.” :) It was a schematised graph database implemented on top of a slimmed-down version of SQL Server. The query language was not SQL-based.

> It was abandoned due to The Cloud.

It was discontinued circa 2007. The cloud was much less of a Thing back then. I don’t recall that factoring at all into the decision to cancel the project, though it would have been prescient.

(Disclaimer: I was on the WinFS team at Microsoft.)

[p_ing]

Skydrive released in 2007.

But fair enough, I grabbed my Beta 1 copy from \\products; it was fun to play with. I wish they’d seen it through. Microsoft had plenty of 'slimmed down' versions of SQL Server, i.e. the CRM addin for Outlook, so that isn't quite a unique feature of WinFS.

[didgetmaster]

Did Microsoft tell the WinFS team why they decided to cancel the product? If so, can you reveal what the real reason was?

[WalterGR]

I mean, it’s not like whenever a project is cancelled at Microsoft they bring team members into a room and say, “We’re totally going to lie to the public (and therefore the shareholders) but here’s the real reason...”

Maybe all the nuances aren't fully communicated publicly when a project is cancelled, but I don’t recall having a sense that what was said publicly was any different than our understanding internally. But that was almost 20 years ago.

The majority of the teams I was on during my time there were ‘internal startups’: Mira, NetGen, WinFS, MatrixDB. Like startups anywhere, projects being unceremoniously cancelled was par for the course.