[dnautics]

You might want to consider mapping an index to an ets table-based data structure instead of an immutable object managed by a GenServer, it will give you a way to share it between processes without having to awkwardly copy a potentially huge data structure all over the place.

[heywhy]

I do have thoughts about performance too but I was following the "get it working then make improvements" route :). Thank you for the suggestions.

[kuzee]

This makes sense, and I think you've taken the correct route. I look forward to trying this in one of my projects and comparing to my current postgres-only backed search strategy. For my use case losing the index between restarts isn't a deal breaker, so hopefully I'll have some useful feedback.

[heywhy]

That's great. I will be looking forward to this.

[dnautics]

Love it. You're doing exactly the right thing.

[skrebbel]

I don't understand how this works. Is data read from ETS somehow shared more efficiently than data shared via a regular message? (which iirc is always copied)

[dnautics]

It's still copied but if you are using an ets table you're likely only copying a small subset of the data per query instead of schlepping the whole index every time.

[eproxus]

It’s still copied, but a process can quickly become a bottleneck in parallel code (every request to a process is sequential).

An ETS table can be concurrently read (and tweaked even further for that use case if desired).

[heywhy]

Like eproxus mentioned, it's still been shared through normal process messaging but improvements will be made regarding this.

[linkdd]

I'd say even using mnesia as an option for disc copies.

[dnautics]

mnesia had very difficult to debug consistency issues that can crop up. Have these been fixed?