[Veserv]

Why is it only 28 MB/core-second?

Is that production rate, inbound bandwidth, rate to persistence, rate to processed, or rate to display?

[prabhatsharma]

Compute power is required to process and store the incoming data.

It's not "only 28 MB/Sec/Core". Try doing same with Splunk/Elasticsearch - You won't go past 5 MB/Sec/Core (Typically it will be lower) on their best day.

[Veserv]

To what state?

Suppose I have 28 GB of trace data in memory on a machine and then I fire that off. What do I have after 1000 seconds?

Do I just have a file of 28 GB of raw trace?

Do I have 28 GB of raw trace in memory ready to be indexed?

Do I have a data structure in memory ready to be searched?

Do I have the full trace information rendered on my screen (or a aggregated visualization derived after processing all the data)?

If it is the first, that would be ridiculously slow. If it is one of the latter ones, then it would depend on what querying operations are fast.

28 MB/core-second makes no sense without the context of what you can do quickly after the “processing” is done.

[prabhatsharma]

Too much to give all details in an HN thread. To simplify the conversation, Data will be persisted and usable for individual searches and aggregations. I would welcome you to our slack workspace for any further questions you may have - https://short.openobserve.ai/community