[snapdangle]

I'm awaiting delivery of a printed Mastering Dyalog APL book while reading this! I landed on kdb+ after seeking a reasonable alternative to the so called "best in class" Elastic Search/Kibana tooling, fell in love with K once I understood that the syntactic terseness is all for the sake of fitting the entire interpreter into L2 cache, and have now landed at the decision that learning Iverson's classic is the only way to satisfy my desire to live a life free of them dang stinking loops!

[dan-robertson]

I don’t believe that the terseness of k is necessary to fit into the I$. I think you could have reasonably longer operators and do fine.

I think it’s partly about style, partly about having a small number of operators (which compose well together), partly on using simple data structures (it doesn’t take much code to iterate an array.

I’m not even particularly convinced that fitting in the instruction cache is a trick that magically makes everything fast anyway. Most of your memory accesses in a typical data processing program (ie the kind of program one would write in k) will be to the data and hopefully these will be nearly always linear and mostly sequential accesses.

[snapdangle]

The original K design was laid out in the 1980s when the constraints were even tighter than what they are today. The utilization of very short operators means not only the interpreter easily fits into cache but also the custom function definitions you have written will as well.

When dealing with high performance computing or real time processing of high volumes of data, any fetch to RAM for loading a function call to dispatch is going to have _some_ impact in a tight loop. Add that up for all the libraries you have loaded for your application verses a ground up implementation in K... Does that whole thing live in L3 along with the VM or intepreter + dependencies underneath it? It's doubtful.

My experience was simply using their Kx's free Developer IDE and experiencing the performance differential on datasets myself. YMMV but my (admittedly limited) experience leads me to believe that there is a serious case to be made for the performance advantages of having all your computational logic living as close to your computational cores as possible.

See also the PhD by author of the OP article where he presents language where:

"The entire source code to the compiler written in this method requires only 17 lines of simple code compared to roughly 1000 lines of equivalent code in the domain-specific compiler construction framework, Nanopass, and requires no domain specific techniques, libraries, or infrastructure support."

Linked from the article, available here: https://scholarworks.iu.edu/dspace/handle/2022/24749

[dTal]

Surely none of that is an argument for terse user-facing syntax? Anything the user types can trivially be converted into K's "bytecode", ahead-of-time.

[beagle3]

No, it’s not.

The arguments for terse user facing syntax are related, though:

The ability to see a whole program (17 lines vs 1000 lines) means you need much less human “working memory” or whatever the biological equivalent of cache is, to reason about the program.

It also means you can use your visual system’s pattern matching abilities because patterns are 5-10 characters rather than 5-10 pages as they often are in C++.

Totally different hardware, but it’s still about the L1 and registers.....

[dan-robertson]

I agree with these points. Also keeping a language small allows it all to fit in your brain (though of course there are all those idioms in apl) which isn’t really true of larger languages

[snapdangle]

True but if I understand correctly (and I'd be happy to be corrected by someone with more K experience), the smaller size of input data should also have an impact on real world parser performance (including the memory usage involved therein).

[dan-robertson]

Any sane interpreter would parse things beforehand and so you shouldn’t expect a significant fraction of performance to be parsing dependent for anything except a trivially sized dataset. Two reasons to care a lot about parsing performance:

1. Silly reasons mean that you need to block while you parse and reducing this blocking time is important (eg JavaScript. One trick is to optimistically assume that the script won’t do a document.write and try to process the html after it. Another would be to eg only parse enough of a function definition to find out where it ends and only parse it more (or block on parsing it) if it gets called)

2. You get a large amount of source code that needs to be quickly parsed all at once to be interactive (eg JavaScript and booted web pages)

I think in the case of k you only parse a small amount at a time in interactive use and any non interactive use would be for a big enough job (yet still generally small enough program) that parsing time would not be significant.

If parsing efficiency really mattered, surely the language would use something easier to parse (eg something rpn based to avoid parens, using single characters for each operator and just disallowing variable names from containing those characters).

In summary: I think parsing speed is basically irrelevant to whether k is fast or not, so long as the parser is reasonably fast, it’s efficiency shouldn’t be a constraint on the language.