[ploggingdev]

Cool project. Could you briefly talk about

* the backend you use and how it will scale to sites with large amounts of data across servers

* can third party sites integrate your search service?

* How is it different from eg- Algolia

Good luck with the project!

[searchhn]

Thank you !

Backend is custom built written in C and assembly. Supports sharding and replication which is rack aware and data-center aware.

> can third party sites integrate your search service?

Yes of course.. that is the end goal.

Algolia is awesome.. but you end up paying a lot based on how many ways you sort / rank data. This operates with an on-the-fly ranking model and rank on any field in any direction. Also different ranking algorithms, extensibility with Lua and a lot more when I officially do a showhn

[ddorian43]

You tried any open-source ones in c/rust ? Are you doing anything differently/better (what/how) ? What are you using for replication/sharding ? Possibility to split-shards ? What are you using for server-backend-framework (ex seastar) ? Any libraries etc that you're using (i'm interested) ?

You have to write a really long blog post on why you've chosen this way.

[searchhn]

Yes.. this will take a very long blog post. This started many months ago as a project to learn 'golang' and as a way to index my everygrowing collection of music / movies / documents / subtitles / lyrics and everything on my servers.

Got hooked into it and became obsessed with speed and rewrote everything in 'C'. Replication is based on 'Raft', actually the multi raft variant proposed by the amazing folks at Cockroach (https://www.cockroachlabs.com/blog/scaling-raft/)

It does not use a backend framework. It is a simple http/https server (epoll + multi-threaded) which talks json. I use Jansson for json and utf8proc for unicode handling. Index is custom built.

I have been working on low powered distributed systems for over 10yrs, which certainly helped. Will definitely let you know when I get that blog post written :)