[cmccabe]

GREAT SOLUTIONS IN ENGINEERING

Problem: bicycle seats are hard. They hurt.

Analysis: there must be something wrong with your pants.

"Solution": dorky pants.

from http://dilbert.com/strips/comic/1994-07-18/

Problem: Python has poor support for threads, and concurrency in general.

Analysis: you must need more middleware.

"Solution": incomprehensible callback-based frameworks.

[readme]

If by incomprehensible, you mean you need to read more than 10 pages of documentation to understand it, then sure.

[pyalot2]

There are two problems with callback based frameworks.

#1 In python specifically, we do not have proper closure and no anonymous blocks, this makes operating it tedious as the definition of the logic for a callback is always somewhere else from where the callback is setup.

#2 In any language supporting proper closures (javascript for instance, hello node.js) excessive, nested use of callback based frameworks leads to the phenomenon of the pyramid of death, whereas you keep nesting closures and you have to keep them all in one place because each depends on its outer scope.

I intently dislike twisted and node.js for these very reasons. I've written my own little framework based entirely on greenlets in python, which is delightful and easy to use and entirely avoids both dislocality of action and the pyramid of death.

[lvh]

In what sense does Python not have "proper closure"? Do you mean mutable closures?

Also, does @inlineCallbacks address your issue? The only fundamental difference I see (other than implementation nastiness, which I guess both have some issue with :)) is that with inlineCallbacks your yields are explicit, and with greenlets they're not.

[lmm]

What do you mean by greenlets?

Twisted feels to me like a grand experiment that's groping towards something that we know is there, but not quite catching it yet. I guess I/O monads feel like slightly further along the path - but they're still not the true solution. What does your system look like?

[lvh]

Greenlets are one-shot coroutines (with no implicit scheduling). They generally work with a C extension that slices the stack.

In IO frameworks that are greenlet based, the greenlet is resumed when it has stuff waiting for it.

The obvious difference to the programmer is that a greenlet won't have yield statements in them, whereas a generator would.

[cmccabe]

Callbacks are bad for the same reason non-local gotos are bad. They destroy locality and make the code hard to read.

[lvh]

"Hard to read" is subjective; many Twisted afficionados, myself included, strongly disagree.

What impact do you feel the destruction of locality has, specifically?

Do you feel @inlineCallbacks and Corotwine answer these issues?

[cmccabe]

Destroying locality makes the program very difficult to read for anyone other than the person who wrote it. I've been there before and done that. I've written Z80 assembly code where functions had more than one entry point as well as exit point. Sometimes I didn't bother to keep a stack at all. Spaghetti code. I was proud of it at the time. Not so much now.

My understanding of Corotwine and greenlet is that they're basically poor man's threads. Basically, if the language doesn't give you usable threads, you can go ahead and implement your own. I've done that one too-- I once wrote an M:N threading library in userspace. Basically making a small number of kernel threads look like a large number of threads to the programmer.

The problem with rolling your own threads is that the normal tooling that comes with the language won't know how to handle what you built. There is no syntax for your add-on, and the semantics may be murky at best. Stack traces become useless, and error messages become guru meditations. Interfacing with third party libraries becomes a challenge, unless they're designed to be purely single threaded.

Additionally-- how to put this delicately? Something as fundamental as threads should not be built as an add-on to a programming language. Things built on top of a language tend to be very fragile and buggy compared to things built in to a language. The C programming language was designed before threads were even a thing in UNIX, so lacking threads was understandable. We built our own threading packages because we had to. But you shouldn't need to do that for a modern language.

[cmccabe]

It looks like Guido van Rossum has posted a proposal for adding asynchronous I/O support to the core Python libraries.

http://thread.gmane.org/gmane.comp.python.devel/136422

This will certainly improve the situation with interfacing different Python libraries together if it gets adopted. I still don't think asynchronous APIs are as easy to use as blocking ones, but I guess I've said enough about that already :)