Hacker News new | ask | show | jobs
Fsearch, a fast file search utility for Unix-like systems (github.com)
207 points by karlicoss 908 days ago
16 comments

This is great tool, I use it everyday, but far from it's Windows based original Everything.

Also this is anbandoned apparently, which makes me extra sad, because it lacks few crucial features like: - being able to just remove a file from the index if you delete it from the app directly (insted it shows a window how it "soon" gonna be implemented) - while i understand that indexing service is more complex job - at least caching the index would be nice, because right now when i start the app i have to wait for it to index everything again, but usually i search for files that exists for a long time, not these that was created between my fsearch uses

So yeah. Cool dead and incomplete piece of software ¯\_(ツ)_/¯ From time to time I look for better alternative, if you happen to know one - let me know.

Hi, I'm the author of this little piece of software.

> Also this is anbandoned apparently, which makes me extra sad, because it lacks few crucial features like:

PersonalIy I wouldn't call it abandoned. I'm still working on it — not as often as I'd like to, but I'm still making progress towards the next release. Though it's still months away from being released.

> - being able to just remove a file from the index if you delete it from the app directly (insted it shows a window how it "soon" gonna be implemented)

That feature is already implemented, but there are no official builds with it yet, because other parts of the software haven't been updated after the rewrite of the database engine (e.g. loading/saving the database file is broken at the moment). Once the old feature set is working again, I'll publish the first official dev builds of the 0.3 release.

> while i understand that indexing service is more complex job - at least caching the index would be nice, because right now when i start the app i have to wait for it to index everything again, but usually i search for files that exists for a long time, not these that was created between my fsearch uses

This is already supported and part of the stable releases. The index is cached and loaded upon application start, so you can search right away, even while the new index is being built. You can also disable auto index updates when the application is launched, if you prefer manual or scheduled index update instead. Or do you mean something else?

What's the best way to help you with this project?
Most definitely code and documentation contributions and to a degree donations — although I clearly prefer the former, simply because it keeps me engaged the most by talking with others about this project, getting new ideas, etc.

But I really welcome any sort of contribution. For example there's also things like improving the main interface language (English isn't my first language, so there's likely room for improvement there), helping with support questions and bug reports, artwork, ...

Last commit 2 weeks ago? Doesn't look dead. Perhaps not actively developed by original author but seems they're still acting as a maintainer and willing to take PRs.
Me too - strangely, one of the reasons I stay on Windows is Everything (https://www.voidtools.com/) - it is just so useful.
I had Everything installed but didn't use it as much as expected. I've gravitated toward FileLocator Pro instead, which uses extremely fast metadata table searching instead of requiring an index (I don't use the Agent Ransack features).

Not a Linux expert, but out of curiosity did you try Recoll when you looked at other platforms? (https://www.lesbonscomptes.com/recoll/pages/index-recoll.htm...)

What is the pattern syntax you use most often?
Just regular fuzzy finding is the most common. For example, to attach a file instead of faffing around clicking in the open dialog, I type into fluent search(a frontend to everything) and it enters it in the current context i.e the open dialog.

It has a few other qualifiers, like big: huge: etc to search large files. Same way with time etc.

Rarely when you need it you prefix with regex: and you get everything you need.

It's very useful that both everything and fluent search integrate into explorer in that if you right click on the search result you get the same context menu as you would in explorer. Drag drop works, etc,. The issue with every other tool in every other OS is the lack of this feature. You basically have a virtual fully featured directory for every search result, simply cannot beat that.

Wait, I didn't know Everything can full search text ? is it like search preview in macOS ?
Sorry, I meant that you can fuzzy find the filenames, not the contents.

For the latter, explorer top right search bar can do it surprisingly. As for a tool with an index, recoll comes to mind.

The most important thing is to sort by date modified by default. Usually, the file you want is very new.

After that I mostly just use "pic:" or "path:".

fd (fdfind on some distributions)
rg or fd?
rg if you want to find stuff in files

fd if you want to find stuff in filenames

fzf for when you want a fuzzy menu type of search on top of this.

I can't recommend fzf enough you can do some really powerful stuff with it. If you don't know it: it gives you a fuzzy search on things you pipe into it. It is powerful because it also can do things like running special commands ("preview") on the currently selected entry/line and allows for displaying the output in a separate pane.

So you could build a thing that e.g. let's you search and multiselect (enqueue) your music collection and on each entry display audio metadata using a custom script.

Or a blazingly fast PDF-content searcher that opens the PDF in the end. The possibilities are endless.

Edit: Here a short video showing my basic git log alias: https://youtu.be/9W27D8lrn-s

    gl: aliased to git log --all --pretty=oneline --pretty=format:"%Cgreen%h%Creset %s %Cred%d%Creset" --color=always | fzf --ansi --preview 'git show --pretty=medium --color=always $(echo {} | cut -d" " -f1)' | cut -d" " -f1
If you want to put this (fantastic, thank you very much for it!) command in your git toolbox, personally I did it like this:

1) create an executable script called `git-l` and put it in a place in your PATH, and make it executable 2) use `git l` to invoke it

You will avoid escaping hell and you can even expand/complicate it much more. The same extension principle works with other CLI tools like `kubectl`.

rg isn't really a file finder, it's a grepper.
rg --files
"Performance. On Windows I really like to use Everything Search Engine. It provides instant results as you type for all your files and lots of useful features (regex, filters, bookmarks, ...). On Linux I couldn't find anything that's even remotely as fast and powerful."

https://www.lesbonscomptes.com/recoll/pages/index-recoll.htm...

"Recoll finds documents based on their contents as well as their file names."

"Recoll will index an MS-Word document stored as an attachment to an e-mail message inside a Thunderbird folder archived in a Zip file (and more… ). It will also help you search for it with a friendly and powerful interface, and let you open a copy of a PDF at the right page with two clicks. There is little that will remain hidden on your disk."

Recoll serves a different purpose as it's primarily build to index and search within your personal documents. That's why it doesn't work well when you point it to the root folder, in an attempt to search within the entire system of millions of files and that's also the reason why it's not as fast, since it's doing more work (parsing complex file formats, searching within a more complex database structure and more data, ...).

FSearch is primarily built to find files on the entire system instantly (by that I mean that all results should be ready by the time you press the next character while typing), based on their name, size, time, filetype, etc. This is less work than what Recoll does and that's why it is much faster.

That's why I also use both tools.

"FSearch is primarily built to find files on the entire system instantly "

I am not sure, but I think my Bodhi "everything starter" solves this problem for me. If I look for something more specific, I use recoll.

Do any of the modern filesystem on Linux cover this very important use case of instant search anywhere like the good old NTFS does (that's how you get Everything's awesome performance)?
Between mlocate and rg I've never felt like I needed anything else.
Have you tried anything else like Everything? It's common to feel no need if you haven't experienced the awesomeness of immediate feedback that in many parts is easily refinable with a shortcut maintaning the same properties, e.g., turn case sensitivity: no need to get your query from history, edit flags and rerun it; or add another column with extra data, same thing, no need to remember the rarely used flags or run a help command to get them, edit query and rerun it
fsearch is the best locate front-end for Linux, but sadly, I've got many gripes with it... crappy drag'n'drop, no daemonization to minimize to system tray, closing the app resets the clipboard for some reason (EXTREMELY annoying when you open it to copy a file path), the list is long. Not to mention that locate itself doesn't auto-update the index with fs changes.

I miss Everything :(

Not to detract from your overall point, but the app is not resetting the clipboard. In X11 if you close any app, you lose anything you copy from it. This is because in X11 windows do message passing to emulate a clipboard. Your browser say and fsearch send messages to each other to send data in few KB chunks(yes there's an incremental transfer protocol where you have to support all kinds of irrelevant clients even if the last such client died before the turn of the century) So when one of the windows closes, it's gg. It's a pretty convoluted over-engineered idea when instead a single file ~/. clipboard would have sufficed.

For a more articulated rant from 20 years ago, see the rant file in https://github.com/porridgewithraisins/x11cp. Also shameless plug.

To fix this, use a clipboard manager. Also shameless plug https://github.com/porridgewithraisins/coffee-pesto :)

How does this search filesystems quickly on linux?
Author here. The app works in two steps:

Step one is building an index of the file system. This is simply done by walking the filesystem. The resulting index is stored in RAM and a file. On the next app start the index ia loaded from that file, which is much quicker than walking the file system.

Step two is using this in RAM index for searching. This scales really well with the number or CPU cores and on modern systems a normal case insensitive substring search should finish almost instantly with few million files.

The next release will support file system monitoring with inotify and fanotify to keep the index updated. Although this has some drawbacks.

This is simply done by walking the filesystem.

This is the part I'm wondering about. Everything scans the filesystem very fast and there is no way it is just using 'stat' on every file then diving into the directories.

Are you just using stat from C to walk the filesystem or are you doing something else?

I've used sqlite to cache filesystem results and it is also extremely fast once everything is in there, but I think a lot of approaches should work once the file attributes are cached.

On NTFS Everything reads the MFT, which is sequential on disk.

Then on subsequent starts it reads the NFTS update journal to see what changed.

> Everything scans the filesystem very fast and there is no way it is just using 'stat' on every file then diving into the directories.

The last time I checked, Everything worked by using the AV calls microsoft provides; anytime a file is written, the name (and other metadata) can be written to a log that Everything can check once every 5 seconds or so.

If I thought there was any money at all to be made from providing an Everything equivalent[1] on Linux, I'd spend the week or so to write it, but as far as I can tell there's just no market for something like this.

[1] By that I mean "similar in performance and query capabilities"; I would obviously need more time than that to hook into the common file-open dialog widgets (Gnome/KDE/etc) so that users could run their queries straight from existing file dialog widgets.

What you are talking about is file change notifications. A huge part of Everything's speed comes from reading the master file table that other people mentioned, so you would need a way to quickly read file table entries on linux.

https://learn.microsoft.com/en-us/windows/win32/devnotes/mas...

> What you are talking about is file change notifications. A huge part of Everything's speed comes from reading the master file table that other people mentioned, so you would need a way to quickly read file table entries on linux.

Not a problem. And no, I'm not talking about inotify either, and I'll additionally index the contents of (text) files as well with a negligible additional performance hit. It can be done as fast as, or faster than, `Everything`.

TBH, if I thought I could make even $100 in donations from this, I'd start it tomorrow, but absolutely no one misses ultra-fast searching when they don't have it.

Even on Windows, the number of users who go out and look for something that searches as fast as Everything is a rounding error - statistical noise. Now go and divide that fractional percentage of Everything users on Windows by 100 to get the number of Linux users who might use this.

Yes, it's simply using stat on every file/folder. There's probably some room of improvement there with clever parallelization, but it'll remain a bottleneck.

Everything is parsing a file called the MFT to build its index. This much more efficient but unfortunately this file only present on NTFS volumes, which makes it super useful on Windows systems, but not so much everywhere else.

Another benefit you get on Windows is the USN journal, which allows Everything to keep the index updated much more efficiently.

I've never used fsearch, but I use a CLI tool that replaces locate (https://plocate.sesse.net/). Do you have an idea of how the performance and index format compares with fsearch?
I'm not familiar with the internals of plocate, but I'll have a brief look at it.
Is it possible to use eBPF for this task instead of inotify?
Maybe, but I'm not sure if there's much benefit to that. The most inefficient part of the inotify or fanotify solution is that you have to walk the file system before monitoring can even start, because you first need to know which folders and files are there to begin with. And unfortunately this can't be avoided with eBPF.
I was wondering what this adds over mlocate. It seems it's a GUI only / GUI first tool.

The GitHub page recommends mlocate for a CLI version.

Hi, author here.

Likely the most significant benefit is the more powerful query language. For example you can also search by file modification date or size and use boolean operators. https://github.com/cboxdoerfer/fsearch/wiki/Search-syntax

Ah thanks for that, I can see the benefit there alright.
For example it updates after every character you type. Sometimes you don't know exactly what you are looking for or you are exploring.
My favorite search tool on windows is Agent Ransack https://www.mythicsoft.com/agentransack/

Searches not only file names but in contents as well. Also blazing fast in my experience.

Just noting this is Windows software not unix-like.
Unix-like apparently does not include macOS here

Edit: Or does it? https://ports.macports.org/port/fsearch/details/

Do you really need it on Mac? We've got mdfind
That's just a cli for Spotlight, right? I have found Spotlight to be increasingly unreliable over the years, to the point of being essentially useless now. Most recently, I discovered that absolutely nothing in my iCloud drive can be found with Spotlight, even if the file is on disk and I'm just trying to match its name. For extra fun, I found an Apple support page that suggested the Windows-in-1999-esque procedure of blocking and then unblocking the directory from indexing in the Spotlight preferences. Unsurprisingly, this didn't work.

Testing out mdfind and trying to simply find a file in a specific directory (which only contains five files):

    mdfind -onlyin . name-of-my-file.md
Nothing.

    mdfind -onlyin . -name name-of-my-file.md
Nothing.

    mdfind -onlyin /absolute/path/to/cwd -name name-of-my-file.md
Nothing.

This is pretty reflective of my experience with the Spotlight GUI. Every search turns up something, but the file I want is almost never in the results.

Spotlight progressively getting worse was one of the reasons I switched to Linux.

I think MacOS is great. Many of the features are brilliant. The UI is second to none to this day.

But holy moly, Spotlight has degraded. From searching my files, to searching my files and the internet, to consistently providing me with nothing I am looking for.

When Apple first started selling OSX it's UNIX heritage was a part of the selling point but my vibe is that they only really consider text input a feature for devs. Which thinking of Mac's history is on brand but just leaves you down many a frustrating dead end.

if you do not have the "show all file extensions" enabled, `.md` query in spotlight will not show up anything.

try without extension. (eg: just the name)

I even have scripts to locate other scripts using mdfind, it is pretty robust to be honest...

Eg:

    source $(mdfind -name spacelatte-bashlib.sh)
It also doesn’t work without the extension. It does work on my work laptop, so I expect that something is broken with my index. But this is also part of the problem. The whole thing is opaque, breaks, and offers no clear recourse for how to resolve any issues.
And FindAnyFile (https://apps.tempel.org/FindAnyFile/index.php), disclosure: happy user with no affiliation.
How does it compare to Baloosearch ? Baloo and KDE Plasma go nicely together !
Does it support find in files? I am using catfish and looking for alternatives but find in files would be a must have. Some are recommending fzf (rg, fd). What is your search workflow and what tools do you use?
> Does it support find in files?

No, not yet.

It’d be interesting if this could integrate with plocate (mlocate’s replacement) which is incredibly quick at indexing and returning results but relatively basic.
You mean like being able to read the plocate database with FSearch? I don't see much point in that, because the plocate database is missing some crucial data, which FSearch uses to make searching and sorting quicker. For example file attributes like size or modification date and the sort order by various attributes (name, path, size, ...) aren't indexed by plocate.

If plocate is faster at building the index, it probably makes more sense look at what's the reason behind that and add these improvements to FSearch.

I like that the author has not done away with the menu, like certain software projects which shall not be named but begin with GN and end with ME.
I gave up on Desktop Linux unfortunately, but this is a great Everything replacement for those who love it on Windows.
What made you to give up?
Too much research and extremely low level troubleshooting required (i.e. source code reading) to get things working. Especially around wayland, multi-monitor, multi-GPU, Nvidia, etc.

I hate what Microsoft has been doing with Windows, but Linux just isn't practical for my setup yet.

You are either extremely unlucky or chose a very strange setup.

The problems you describe were common in early 2000s, but haven't been common in Linux desktop for a decade or so.

For those reading above and thinking "I'll skip Linux, if that's the current status": it's not. Just pick Ubuntu LTS. Use it on common hardware (e.g not bleeding edge) and stick with the defaults. Don't try to make it exactly like your Mac or Windows machine but lean into how it does things. They are different . They may be uncomfortable. Then, once familiar feel free to tinker and hack.

I'm on Linux since 1996. I've hacked and tweaked everything in my younger years. Now I'm on a boring, hardly configured Ubuntu LTS. Well, my she'll and nvim are tuned beyond recognition, I guess. The rest: boring.

> For those reading above and thinking "I'll skip Linux, if that's the current status": it's not.

I politely disagree. I recently installed Fedora on my desktop PC because Microsoft decided that displaying a full-screen ad for Windows 11 that prevented my PC from booting was acceptable behaviour. Anyway, one of the first things I noticed on Fedora was that video playback was stuttery. After ages spent digging around, I discovered Fedora had disabled GPU hardware video decoding for legal reasons. Around the same time I made the mistake of trying to delete a directory with lots of files in it on an NTFS drive. The operation failed and corrupted the filesystem, and I had to spend a week or so downloading and restoring backups. Needless to say I'm back on Windows now.

I'm not new to all this - I used Slackware and Red Hat on the desktop in the early 2000s, and I use Linux on the server side daily. But on two separate laptops recently, manufactured 10 years apart, I've had all kinds of glitches and performance problems with (Arch) Linux, on a variety of desktops (Sway, KDE Plasma, Gnome) especially around video and GPU. You could say just don't buy Nvidia - but one, too late, and two, if you want to do anything interesting in AI these days they're hard to compete with.

YMMV and if you're just in the shell and in VIM all day you might not notice video glitches and performance problems (computers did text terminals in the 70s so it's not a high bar). But as a lapsed game dev I have an eye for stutter, missed frames, etc, and those were pretty constant in my configs, despite sinking probably a hundred plus hours in, time I won't get back.

I have wasted so much time trouble shooting just basic things on Linux DEs. Example: mouse wheel scroll speed. In general it’s a mishmash of various egos, various low efforts, various high efforts, etc. Not usable imo if you just want to use a computer and have the OS out of your way.
They are quite common on laptops, here is a second anecdote on the matter.

We even have an internal how-to about what works and what not, for those that want to try their path outside the Thinkpad/Windows, Macbook official IT path.

I feel you, as subscriber from Linux Journal during their print lifetime, I eventually found peace in Windows 7 and later, alongside desktop VMs, than ensuring everything on a laptop does indeed work properly.

Even Linux branded laptops keep having issues, example on the one I still have around, I never got around fixing it dropping wlan connections, so for large OS updates it needs to be plugged on the LAN.

Nvidia is indeed not well supported, on which Nvidia is mostly to blame (also see Linus Torvalds finger to Nvidia). That is too bad because the community can't do much about it.

Multi monitor support should generally be fine in most desktop environments, at least for 2 screens. More than that can indeed be quirky, dependent on the desktop environment and window manager (X or Wayland) you're using.

Multi-GPU is probably a bit niche to have good driver coverage, partly probably because of the Nvidia issue.

Looks quite nice, brownie points for being native.
E a d
For anyone looking for a Mac equivalent, there's GoToFile[0].

As far as I've seen, this is the only app for Mac that doesn't just reuse Spotlight search (which I find to be terrible). I looked for exactly this type of app for years before finding it, and when I did, it didn't seem real with the old-fashioned website and zero mentions on sites like HN. But I can assure that it works great and it's maintained. I just wish the author would promote it better so it gets more attention and isn't so hard to find.

[0] https://www.soma-zone.com/GoToFile/

+1 on the need for non-Spotlight options. Source code makes a mess of spotlight for everyday usage. I like Leap a lot for multimedia browsing with more precise search capabilities than spotlight: https://ironicsoftware.com/leap/
You can remove folders from spotlight under "Spotlight Privacy". I removed my source code folders, which vastly improved search results.
100%. That's when I discovered I needed non-spotlight search to bulk search what I just excluded from spotlight.
Soma-zone software in general is pretty neat for Mac power users. I especially like Launch Control, which lets you configure and inspect launchd (the mac service daemon) services.
How does a program like this work? Is it indexing every file on the system and monitoring all events for updates?
Foxtrot Search (expensive) uses its own index and also indexes content, not just names

There are a couple of other filename-only apps with own index, but don't remember now which of the alternatives do that (HoudahSpot?)