[hcfman]

Wuite curious. What indexing and retrieval software is this using? I couldn’t find reference to it.

Does it index phrases ?

[bdcravens]

"... who crawl the web using the Firefox extension and command line script"

https://addons.mozilla.org/en-GB/firefox/addon/mwmbl-web-cra...

https://github.com/mwmbl/crawler-script

[mdaniel]

I believe this is closer to the thing you were asking about, and the simple answer appears to be "a home grown one in python" https://github.com/mwmbl/mwmbl/blob/e544d45c374c13cdc1a5048d...