[cyanydeez]

If anyone wants down this golden path, I'd recommend forking open search server. It's quite a feat and does the crawling part well.

https://www.opensearchserver.com/