[luca-ctx]

Thanks and this is a very interesting idea!

We considered this, but the main thing you gain from this tradeoff is some disk space and cleaner retention semantics from not having to duplicate all of the searchable text.

But you still have to do the parsing and ingestion work to build the index in the first place, so CPU time does not go away.

And you still have to store the indexes and enough metadata to map results back to the raw session files, which bounds the benefit of not duplicating the data.

The main downside is flexibility (you would lose the ability to do arbitrary SQL queries, semantic search on top of structured corpus, etc)

But I would love to see if I can be proven wrong on this!

[mkornaukhov]

We've been working on remote search indexing in our project and it works pretty good. since we are building a Postgres-compatible database, everything is pure SQL. I'd say we could join forces if you're up for it.

[luca-ctx]

Very cool! Do you have any of it OSS? Or drop me an email: [e-mail]