[bfgoodrich]

One of the trends on Hacker News right now is the assumption that SQLite fixes everything. It's the Flex Tape of the software world.

There is zero reason a process would have more than tiny slowdowns with even millions of files in a folder. Finder has problems if you're trying to look at that folder, for obvious reasons, but it's a bit of a self-own for a backup co to claim that 200,000 files causes their solution to break. That speaks to serious algorithmic issues.

DISCLAIMER: This comment will be auto-dead because of moderation choices by dang (e.g. his pernicious need to pander to the anti-science, far-right crowd). This is a badge of honor. Never vouch for my comments.

[bob1029]

> It's the Flex Tape of the software world.

I assume you are using Flex Tape as a derogatory comparison here. That said, I do view SQLite as a kind of "fix all" in the software world.

It's cheap (free), fast, everywhere and applicable to virtually every type of problem domain. AAA game assets to B2B line of business app storage. It's the most tested and used software on earth.

It doesn't fix everything, but it certainly gives you a fighting chance to make it to the next step.

[antisthenes]

Once people realized that 90% of things aren't actually #BIGDATA, and SSDs became commonplace, it turned out SQLite is just good enough to solve those 90% things.

Things like contact lists or game assets or even web history on individual computers will never grow to multi-TB sizes, hence no reason to over engineer them.

[username135]

A solution to a problem is still a solution regardless of what it is.

Might not be the most glorious or flashy solution, but like you mention, getting to the next step is all that really matters, IMO.

[dang]

Your comments are auto-dead because we banned your account. We told you that when we banned you: https://news.ycombinator.com/item?id=25399750. We banned your account because you were egregiously breaking the site guidelines.

[majkinetor]

> There is zero reason a process would have more than tiny slowdowns with even millions of files in a folder.

What kind of BS is this? Have you ever worked with one such folder? If you did, you would know that almost every app not doing some magic slows down to the point of being unsable (or even not working as described in original post). This is true for both Linux and Windows file systems.

[didgetmaster]

Exactly. Applications that are searching for files that have a certain extension (e.g. *.jpg) must read every directory entry and compare the file extension for that pattern. There is no magic call to the FS that says "Give me all the photos" or "Give me all the .jpg files). The application (including any command processors) have to do all that work manually.

[meindnoch]

Wow. Indeed, practically all of your comments are dead. None of them seem off-topic though. How is this possible? Is there a gang of users who auto-flag all of your comments?

[yorwba]

Account was banned in 2020: https://news.ycombinator.com/item?id=25399750

[dx034]

But why would that affect new comments if they’re able to create comments?

[dang]

Banned accounts can post comments; the comments are killed by default.

[meindnoch]

Then how come I see their comments?

[dang]

You turned showdead on in your profile.

[toast0]

If you've got 1M files, your alternatives include:

a) 1M files in a single directory

b) 1k directories with 1k files in each

c) 1k aggregate files with 1k files in each

d) 1 aggregate file with 1M files in it

I think there's some filesystems that will work with option a, but any filesystem should work ok with the other options. Options c and d will make rsync much faster as you eliminate millions of syscalls.

[pjc50]

This is a weird fight to pick.

It's usually a problem for any software that enumerates a directory for any reason, because that ends up being O(n) and often at least N system calls to e.g. get file information.