[thenaturalist]

The optimal chunking strategy is often highly, highly dependent on the data used and questions to be answered.

The net is plastered with blog posts about optimal strategies, of which there seem to be more than 10 and new approaches popping up often.

It seems consensus that trial and error is the way to go to optimize cost and performance.

How do you plan to tackle this when providing it out of the box?

[nael_ob]

That's why we wanted to try the OSS approach where contributors can help keep up with the optimal strategy. We also plan to build an engine to test each strategy and compare retrieval perf before choosing one at runtime.