[ilaksh]

Interesting because they mention many of the same issues our crypto arb had to resolve.

As far as data, there are lots of ways to handle it. Initially the old team had used kdb mainly because they literally thought that was the only possible solution. I made something called `timequerylog` that uses newline-terminated JSON files separated out by date and hour.

[villager]

>>timequerylog For storing only or for data analysis as well?

Currently our postgres have something around 1 Tb which is already not so easy to play with.

[ilaksh]

It's on npm. It has some simple query abilities. Also there is `tql-cli`. The basic idea is pretty simple, just that you use log files and query by date and time range and key and so you avoid querying or analyzing the entire dataset at once. Note that since it uses the disk with lots of files so much, the assumption is that you are on an SSD.