| Thanks for the thorough response. But firstly: > If your firm has had KDB for ages there's a good chance it's big enough to be signed up to one of the research groups who maintain a set of test-suites they will run over a vendor's latest hardware offering, letting them claim the crown for the fastest Greeks or something. If your firm is a member you may be able to access the test-suites and look at how the data in the options tests is being written and read, and there are quite a few, I think. Unfortunately my firm isn't that big ~150 in total and maybe about ~40 developers, if which there are 2 full time KDB devs who's job is mostly maintaining the ingestion and writing some quite basic functions like `as_of`. There's only two people who work on our options desk as developers, so there's a lack of resourcing for KDB. When I have these issues with KDB around performance, it's quite hard to get support within my firm from the two KDB devs (one of which is very junior). > I've never worked on options data and so can't opine on the problems it presents The thing about options data is that it's generally lower frequency but a lot more complex. If spot data is 1 dimensional, and futures data is 2 dimensional, options are 3 dimensional. You also have a lot more parameterizations which leads me to the second point :) > you may not want to load data out of KDB a day at a time. Try to do the processing in KDB Just to give you a very specific example of the processing I need to do. I have a data structure in KDB like this (sort of typescript notation): row = mapping<datetime, { a: number, b: number, mu: number: sigma: number, rho: number}>
This is vol surface. To convert that into volatility requires: f = log_moneyness - m;
total_var = a + b * (rho * f + (f * f + sigma * sigma).sqrt())
vol = total_var / time
Then in order to calculate the log_moneyness I need to calculate the forward price from an interest rate which is slightly more trivial.Now I have a base in which I can start generating data like the delta, but this also requires a lot of math. I was pulling this stuff out of KDB because I already had my code in rust that does all of this. > You said you're scared to do queries that return a lot of data, and that it often freezes. Are you sure the problem is at the KDB end? Yeah, I'm pretty sure in my case. We have some functions designed for getting data written by the KDB guys. Even for functions that return 30 something rows, like an as_of query it takes ~10s. |