[steveBK123]

Also you mention rust - https://docs.rs/kdb/latest/kdb/

The old style of kdb integration you could compile a .so and load it in to extend the language. Now people use PyKx to load in python modules. I have a guy doing this on my team to load a Python wrapped Rust lib.

It looks like you have a few options with Rust as per the link above.

Note this allows you to do the "move the code to the data" trick I mentioned.

[sonthonax]

So I’ve actually contributed to this project.

My concern with adding custom libraries into KDB, while better than writing duplicative Q code is the maintenance nightmare of keeping them up to date in KDB.

It’s still an investment but I need to be aware of of the risks and downsides.

[steveBK123]

Loading your rust code into your existing KDB data lake and periodically updating it will be a significantly smaller lift than rewriting your data lake.

It sounds like you are some sort of Quant Dev on a desk, and so it's really up to you what you want to do. If you push against the grain to do a data lake rewrite, you'll own the time/effort/outcome of a big Data Engineering project. So you better be very right and also very fast.

If you are looking for solutions within your existing data lake, I've posted up a few sources / thoughts for you to get on and do your Quant Dev work.

You sound very set on some sort of rewrite, so you should do what your heart desires. Just make sure you deliver value to your desk.

[sonthonax]

> Loading your rust code into your existing KDB data lake and periodically updating it will be a significantly smaller lift than rewriting your data lake.

This is probably going to be what I do until KDB creaks over.

> You sound very set on some sort of rewrite

I vacillate between the two things. I'm personally used to data engineering with parquet and spark, which are widely used outside of finance, and don't have expensive vendor lock in.

And then I realise that I'd have to own this stuff, and my job isn't a data engineer, and I'm a quant dev.