[CoffeeDregs]

Perhaps the video makes more clear the rationale. E.g.

  Possible solution: "Standardizing gevent solves all its problems". 
  One of the responses: "I like to write clean code from scratch".
  Another: "I really like clean interfaces".

So I'd prefer that the BDFL work with the gevent folks to get it cleaned up and integrated while adjusting it to expose a "clean interface".

Perhaps the whole thing will make more sense once Guido provides more detail, but I'm underwhelmed and confused.

[ekimekim]

I find that unlikely: Guido doesn't like gevent.

Though yes, it's a nice thought.

[fzzzy]

Guido has been resisting the stackless stack slicing assembly technique since I first learned about Python and Stackless Python in 1999. That's obviously never going to change.

[rdtsc]

That reminds me of one of those famous Roman Emperors that all is well and good as well as they make rational decisions, then eventually they turn senile or mad, and everyone realizes how dictatorship is not that much fun sometimes.

[fzzzy]

From a certain perspective it is a rational decision. Because the CPython API relies so heavily on the C stack, either some platform-specific assembly is required to slice up the C stack to implement green threads, or the entire CPython API would have to be redesigned to not keep the Python stack state on the C stack.

Way back in the day [1] the proposal for merging Stackless into mainline Python involved removing Python's stack state from the C stack. However there are complications with calling from C extensions back into Python that ultimately killed this approach.

After this Stackless evolved to be a much less modified fork of the Python codebase with a bit of platform specific assembly that performed "stack slicing". Basically when a coro starts, the contents of the stack pointer register are recorded, and when a coro wishes to switch, the slice of the stack from the recorded stack pointer value to the current stack pointer value is copied off onto the heap. The stack pointer is then adjusted back down to the saved value and another task can run in that same stack space, or a stack slice that was stored on the heap previously can be copied back onto the stack and the stack pointer adjusted so that the task resumes where it left off.

Then around 2005 the Stackless stack slicing assembly was ported into a CPython extension as part of py.lib. This was known as greenlet. Unfortunately all the original codespeak.net py.lib pages are 404 now, but here's a blog post from around that time that talks about it [2].

Finally the relevant parts of greenlet were extracted from py.lib into a standalone greenlet module, and eventlet, gevent, et cetera grew up around this packaging of the Stackless stack slicing code.

So you see, using the Stackless strategy in mainline python would have either required breaking a bunch of existing C extensions and placing limitations on how C extensions could call back into Python, or custom low level stack slicing assembly that has to be maintained for each processor architecture. CPython does not contain any assembly, only portable C, so using greenlet in core would mean that CPython itself would become less portable.

Generators, on the other hand, get around the issue of CPython's dependence on the C stack by unwinding both the C and Python stack on yield. The C and Python stack state is lost, but a program counter state is kept so that the next time the generator is called, execution resumes in the middle of the function instead of the beginning.

There are problems with this approach; the previous stack state is lost, so stack traces have less information in them; the entire call stack must be unwound back up to the main loop instead of a deeply nested call being able to switch without the callers being aware that the switch is happening; and special syntax (yield or yield from) must be explicitly used to call out a switch.

But at least generators don't require breaking changes to the CPython API or non-portable stack slicing assembly. So maybe now you can see why Guido prefers it.

Myself, I decided that the advantages of transparent stack switching and interoperability outweighed the disadvantages of relying on non-portable stack slicing assembly. However Guido just sees things in a different light, and I understand his perspective.

  [1] http://www.python.org/dev/peps/pep-0219/
  [2] http://agiletesting.blogspot.com/2005/07/py-lib-gems-greenlets-and-pyxml.html

[rdtsc]

Thank you for the providing all the background! I didn't know all the historical context.

> or custom low level stack slicing assembly that has to be maintained for each processor architecture.

Yeah I would personally also say that the advantages of writing the equivalent assembly for a handful of architectures outweigh re-writing / re-inventing high level library code dealing with concurrent IO.

I think Guido and a few others have decided to focus on some things rather than others and the Python ecosystem will suffer in the long run based on these decision. The ability to write beautiful concurrent IO code without a whole new async framework, is better, even if means breaking some C extensions, writing assembly, or not supporting exotic CPUs.

Programming world is not going to get less concurrent over time. Concurrency will spread more and more. There is already some momentum with gevent and eventlet and it is something that Node.js doesn't have -- and I see Guido turning away and making something worse.