[mahyarm]

It almost sounds like kdb needs an alternative syntax that is more human readable.

[beagle3]

It has one: q. But once you get over the syntax, you realize that you also need to grok different semantics that you are used to.

Some q is readable english - e.g., an expression like

   sum price where size>3

is (to the uninitiated) more readable than the equivalent k

   +/price@&size>3

but that only works for simple stuff. The (idiomatic!) computation of maximum-subarray-sum[0]

   |/0(0|+)\

becomes

   max over 0 (0 max +) scan

which is not more readable. And you can drive the point ad absurdum by making it even more verbose:

   max over zero (zero max plus) scan

The syntax seems like it is what stops you from understanding it because it is the first thing you meet. But it's the semantics that you need to grok, and the syntax just matches them.

[0] http://en.wikipedia.org/wiki/Maximum_subarray_problem

[sqrt17]

I would find something like

over(max, scan(0, max(+, 0))

slightly more readable, even if that still gives me a higher-order-headache.

Because, y'know, it's confusing to have symbol salad with a weird mixture of infix and postfix operators, even more so if you have type raising (or the moral equivalent thereof) thrown in for good measure.

[bastawhiz]

The q in your latter example is the same complexity in my book as a dense Python list comprehension. Could be worse.

[beagle3]

The equivalent python is:

    mss = lambda x: max(scan(lambda a,b: max(0,a+b),0,x))

assuming a definition "scan" (which is like "reduce", except it gives you all intermediate values), an example of which is:

    def scan(f,x0,x):
      r = [x0]
      for x1 in x:
        x0 = f(x0, x1)
        r.append(x0)
      return r

Note that the K is idiomatic whereas the python is (arguably) not. Of course, it could be worse; the advantage is that, much like math, the 9-char K version is pattern-matched by your eyes once you are familiar with it, whereas no other version presented here (or in almost any other language) can utilize that feature of your brain.

[cbsmith]

Oh goodness... don't do scan like that. Way easier (and more efficient) to use a generator:

    def scan(f, iterator, initial=0):
        yield initial
        yield from scan(f, iterator, f(initial, next(iterator)))

Even with python2, you could make a non-recursive version that'd be still shorter than your scan and faster. Alternatively, you could pair a coroutine with that reduce function....

But always be suspect of your code if you are iterating and appending to a list. Likely there is a much better way.

[beagle3]

First, it is not equivalent - next() cannot apply to range() output, for example - you will need to do some iter() games and watch out for iteration order side effects if your values are iterators vs. lists.

Second, it is ~10% faster, but that speed difference disappears completely if you eliminate the namespace lookup (that is, add e.g. "o = r.append" before the loop, and call o() instead of r.append() inside the loop). It potentially uses less memory - but not the way you did it (unless Python 3 gained TCO when I wasn't looking. Did it?) - your formulation does not load the call stack, but it does create len(iterator) generators that - until the innermost StopIteration - all need to live somewhere on the heap. recursive solutions without TCO are rarely good enough to replace iteration.

Even if you did it right, it's more efficient, but not significantly so timewise, and slightly easier to use iterators in general, yes. It is mostly space-efficient in general.

I think it is more idiomatic, though - and also Python2 compatible - to just replace references to 'r' with yield in my code, than using the recursive definition you gave above - which is more idiomatic in functional languages, but less in Python (and harder to debug in any language than the iterative version)

[cbsmith]

> First, it is not equivalent - next() cannot apply to range() output, for example - you will need to do some iter() games and watch out for iteration order side effects if your values are iterators vs. lists.

It uses generator/iteration semantics instead of list semantics. If you wrap the whole thing with a decorator like function that does:

    def scan_wrapper(f, x0, x):
        return list(scan(f, iter(x), x0)

You get the exact same semantics. For most cases (including the one you cited), the alternate semantics are actually better, more flexible, and avoid requiring a list to be built in the first place.

No problem with iteration order side effects either unless your f() somehow invalidates your iterable... and you still have some potential exposure there in your original implementation.

> Second, it is ~10% faster... > It potentially uses less memory...

Yeah, I think you are understating it to say the least. Not only are you using less memory, but you are saving having to rejuggle/resize the list all the time.

> (unless Python 3 gained TCO when I wasn't looking. Did it?)

I guess in a way it sort of did for the case of yield from: 'The iterator is run to exhaustion, during which time it yields and receives values directly to or from the caller of the generator containing the yield from expression (the "delegating generator").'

So, even without full on TCO (which is still possible... I'm not sure if they did it with yield from), you at least have direct pass through from the generator to the caller. Because of iterator semantics, that should mean that each of the generators gets created on an as needed basis and the previous generator should get destroyed right thereafter. It is possible though that it isn't quite doing it right, in which case I'll concede that I'm still allocating an N deep generator stack, but that is still likely to be more memory efficient because it isn't having to reallocate/resize/copy increasingly larger lists throughout the execution.

> recursive solutions without TCO are rarely good enough to replace iteration.

As I mentioned, you can do the recursive solution as well, and it has the advantage of working with old Python. Still simpler and still far more efficient (here it is with extra wrapping to keep the semantics the same):

    def scan(f, x0, x):
        def scan_helper():
            yield x0
            for x1 in x:
                x0 = f(x0, x)
                yield x0
        return list(scan_helper())

> It is mostly space-efficient in general.

You say that like when doing statistical analysis space-efficiency isn't a concern...

> I think it is more idiomatic, though - and also Python2 compatible - to just replace references to 'r' with yield in my code, than using the recursive definition you gave above - which is more idiomatic in functional languages, but less in Python (and harder to debug in any language than the iterative version)

I was actually mostly getting at using yield instead of list append. I was just trying to express it as tersely as possible, which unsurprisingly became Python 3 and a functional style mechanism.

While I agree that often there is a struggle to understand functional programming, I think in this case it is very idiomatic Python (particularly since they defined "yield from" specifically for cases like this), and the code is very simple, readable, and easier to verify for correctness.