Very interesting project! I guess it could be even better if you didn't have to ingest the session data into a database but just build an index on top. I have an idea how to do it
We considered this, but the main thing you gain from this tradeoff is some disk space and cleaner retention semantics from not having to duplicate all of the searchable text.
But you still have to do the parsing and ingestion work to build the index in the first place, so CPU time does not go away.
And you still have to store the indexes and enough metadata to map results back to the raw session files, which bounds the benefit of not duplicating the data.
The main downside is flexibility (you would lose the ability to do arbitrary SQL queries, semantic search on top of structured corpus, etc)
But I would love to see if I can be proven wrong on this!
We've been working on remote search indexing in our project and it works pretty good. since we are building a Postgres-compatible database, everything is pure SQL. I'd say we could join forces if you're up for it.
We considered this, but the main thing you gain from this tradeoff is some disk space and cleaner retention semantics from not having to duplicate all of the searchable text.
But you still have to do the parsing and ingestion work to build the index in the first place, so CPU time does not go away.
And you still have to store the indexes and enough metadata to map results back to the raw session files, which bounds the benefit of not duplicating the data.
The main downside is flexibility (you would lose the ability to do arbitrary SQL queries, semantic search on top of structured corpus, etc)
But I would love to see if I can be proven wrong on this!