[a_bonobo]

One interesting side-thing with list comprehensions in Python 2 vs. Python 3:

Python 2.7:

>list_of_numbers = [1,2,3]

>[x/2 for x in list_of_numbers]

>print(x)

3

Python 3:

>list_of_numbers = [1,2,3]

>[x/2 for x in list_of_numbers]

>print(x)

NameError: name 'x' is not defined

They "leak" their variables in Python 2, if you're someone who reuses variables this can lead to an enormous headache!

[Cyph0n]

Wow, I never noticed that! Who the hell thought that was a good idea?

[raymondh]

Barry Warsaw proprosed List Comprehensions 16 years ago in PEP 202.

The original concept was that it provided a more concise way to write loops so that:

   t = [expr(x, y) for x in s1 for y in s2]

was just a short way to write:

   _t = []
   for x in s1:
       for y in s2
           _t.append(expr(x, y))
   t = _t
   del _t

The original implementation reflected that design. I later added the LIST_APPEND opcode to give the list comprehensions a speed advantage over the unrolled code.

The question of whether to expose the loop induction variable didn't get much discussion until I proposed Generator Expressions in PEP 279 and decided to give them the behavior of hiding the loop induction variables so that the behavior would match that of a normal unrolled generator function.

When set and dict comprehensions (displays) came along afterwards, they were given the latter behavior because there was precedent, because there was a mechanism to implement that precedent, and to provide a short-cut for then common practice of creating dicts and sets with generator comprehensions:

    s = {expr(x) for x in t}

was a short-form for:

    s = set(expr(x) for x in t)

The advent of Python 3 gave us an opportunity to make a four forms (listcomps, genexps, set and dict displays) consistent about hiding the loop induction variable.

The current state in Python 3 has the advantage of being consistent between all four variants and matches how mathematicians treat bound and free variables.

There are some disadvantages as well. List comprehensions can no longer be cleanly explained as being equivalent to the unrolled version. The disassembly is harder to explore be cause you need to drill into the internal code object. Tracing the execution with PDB is no fun because you go up and down the stack. It is more difficult to explain scoping -- formerly, all you had was locals/globals/builtins, but now we have locals/nonlocals/globals/builtins plus variables bound in list comps, genexps, set/dict displays plus exception instances that are only visible inside the except-block.

[Cyph0n]

Thanks for the detailed reply, Raymond! I never expected a core developer who worked on implementing list comprehensions to respond haha!

That last point about traceability is something I never thought about, but I think the trade-off is worth it.

Keep up the awesome work man. For anyone interested, you can follow him here: https://twitter.com/raymondh

[bearfrieze]

Thanks for the excellent explanation and your great work on Python.

[maxerickson]

It was largely an implementation detail. This mentions it some:

http://python-history.blogspot.com/2010/06/from-list-compreh...

The PEP is silent on the issue:

https://www.python.org/dev/peps/pep-0202/

I guess mail list spelunking would probably reveal some conversations about how to handle it.

[bpicolo]

Pretty sure it was a bug that they kept for backwards compat, not intentional, though I may be wrong. Py3 let them shed it off though.

[masklinn]

> Pretty sure it was a bug that they kept for backwards compat, not intentional

Yep, as demonstrated by only list comprehensions exposing this behaviour (on python 2) whereas none of generator comprehensions, dictionary comprehensions or set comprehensions do:

    >>> [x for x  in range(5)]
    [0, 1, 2, 3, 4]
    >>> x
    4
    >>> {y for y  in range(5)}
    set([0, 1, 2, 3, 4])
    >>> y
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'y' is not defined
    >>> {y:y for y  in range(5)}
    {0: 0, 1: 1, 2: 2, 3: 3, 4: 4}
    >>> y
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'y' is not defined
    >>> list(y for y  in range(5))
    [0, 1, 2, 3, 4]
    >>> y
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'y' is not defined
    >>>

[bpicolo]

To be fair, generators are lazily evaluated so I would sure hope they wouldn't expose scope! That would show up in truly unique places haha

[chishaku]

> One interesting side-thing with list comprehensions

Just to clarify, this is the behavior of py2 for loops in general, whether in a comprehension or not.

[prlin]

Except dictionary comprehensions don't leak their variables.

[sophacles]

Probably because dict comprehensions were introduced as a 3.0 feature, and put into 2.7 since that version is designed to be a bridge to 3.0.

(They were actually proposed a long time ago, it was decided not to add them, but the decision changed to add them for 3.0)

[bearfrieze]

Nice bonus info. I didn't know about this (haven't spent much time with Python 2.x).