Hacker News new | ask | show | jobs
by ben_jones 3713 days ago
Does anyone enjoy doing async work in python? I've done a few hobby projects and honestly I was yearning for javascript + async lib after awhile. As great as python is maybe we should yield async programming to the languages designed for it?
6 comments

I think async and aiohttp are game changes for Python 3.5. After working with Twisted callbacks for over a decade, it's a pleasure to write async code that does not use the callback approach (granted Twisted is a mature environment with lots to offer).

I've switched to Python3.5 and aiohttp for all new web service applications. The coding style is clean, enjoyable to write, and easy to debug.

Plus, I've never once been stymied for speed. I know there's applications out there where people expect to to be handling zillions of connections -- but the bulk of my use cases think 100 transactions per second is a huge through-put, and aiohttp handles that with ease.

Have you used eventlet or gevent? I thought they were game changers. Gevent has been working very well for me for quite a while, without callback hell.
I found it easier than doing it in javascript because I could insert `import pdb;pdb.set_trace()` into the code and get an interactive debugger.

Supposedly you can do this in javascript by running node with a particular flag, then connecting to a port on localhost, and opening the chrome debugger. However, the multiple times I've tried throughout 2014-2016 has show that to be incredibly finnicky. It is especially frustrating when trying to insert a debugger into an automated test.

I guess I don't know how JS was any more "designed for" async than python was.
From https://developer.mozilla.org/en-US/docs/Web/JavaScript/Even...:

JavaScript has a concurrency model based on an "event loop". This model is quite different than the model in other languages like C or Java.

...

A very interesting property of the event loop model is that JavaScript, unlike a lot of other languages, never blocks. Handling I/O is typically performed via events and callbacks, so when the application is waiting for an IndexedDB query to return or an XHR request to return, it can still process other things like user input.

>>JavaScript, unlike a lot of other languages, never blocks

I think that's a little strong. It's more like "The group controlling Javascript has mostly tried to discourage introduction of things that block".

You can, for example, do a blocking XMLHttpRequest. It's deprecated, but possible. https://jsfiddle.net/923d5sda/

You can do blocking everything.

A 10.000 repetitions for look while block the whole interpreter for its duration.

Any JSON parsing does the same.

Processing strings.

Doing math work.

...

Guess I should have said "blocking I/O"? I thought it was a given that a single threaded language wouldn't magically inject some kind of concurrency around tight loops or CPU intensive tasks. Node.js people don't really think that sort of thing is "non blocking", do they?
>Guess I should have said "blocking I/O"? I thought it was a given that a single threaded language wouldn't magically inject some kind of concurrency around tight loops or CPU intensive tasks.

Well, Erlang (and Elixir) does just that -- it's preemptive, and implicitly yields under the covers even in loops.

>Node.js people don't really think that sort of thing is "non blocking", do they?

Judging from forum threads and blog posts, a lot of them do, especially web programmers not familiar with blocking and non-blocking that only know that "Node is webscale".

I think that quoted statements are not very good.

The other languages don't have a builtin concurrency model. For C, Java and others event loop libraries and applications that are built on top of them can be found (nginx, netty, ...). As well as there are libraries that build on top of synchronous IO.

The eventloop was also not tied to Javascript in the older standards. Only the introduction of Promises and other stuff required the existence of an eventloop in order to define when continuations should run.

"The event loop model never blocks" is also only true as long as you (and all the libraries that you use) do not block it. There is no automatic "does not block" guarantee.

>A very interesting property of the event loop model is that JavaScript, unlike a lot of other languages, never blocks.

JavaScript actually always blocks. It's only external function calls that somebody took care to write in an evented style that don't block -- but anything written in pure Javascript (from for loops to text manipulation) blocks.

JS single-thread async without preemptiveness is not something to write home about...

This is a genuine question: in what ways is Python's async implementation lacking? Could it have been baked in a better way?

In what ways do languages that were supposedly designed for async programming different than Python?

Python is definitely lacking an elegant interface for async programming.

I think that Python 3.5 now has a very elegant interface for async programming. I prefer Tornado to the standard library's asyncio, but the new keywords are nice for both packages (disclaimer: I'm the maintainer of Tornado).

The downsides have nothing to do with the design of the language. The problem is that introducing a new concurrency model late in a language's life splits the ecosystem. Most existing packages are synchronous, so if you want to build asynchronous systems you must avoid packages like requests, django, or sqlalchemy and find (or develop) asynchronous equivalents for the functionality you need.

Javascript has an advantage here not because the design of the language is especially well-suited for asynchronous programming, but because it never went through a synchronous/multi-threading phase. Every javascript package is designed for asynchronous use.

Yeah, my biggest complaint personally is that combining multithreading and async is a massive pain in the ass. Now, realistically, you aren't usually going to want to do that, except if you have multiple event loops, or are bridging between external synchronous code and internal async code. Otherwise, I really enjoy async python -- of course, I'm also the kind of person who has written my own event loops using synchronous code before, so maybe I'm just crazy like that.
>> The problem is that introducing a new concurrency model late in a language's life splits the ecosystem.

Really you could say that about a number of features/changes in python 3, not just the new async syntax. Python 3 itself was an ecosystem-splitting instrument.

simple reason - there is NO framework that is built ground up for nodejs style async programming.

Obviously there is Twisted and Tornado - but gevent or asyncio are actually the paradigms that people are using now. If there were a Flask like framework that was ground up built to leverage async (rather than bolting it on) and included all the batteries for web development.. then python would have a serious edge over node.

Tornado works in pretty much the same way as asyncio.
Javascript was hardly designed for async work -- it is just that it wasn't designed for anything else.
aiohttp has completely replaced flask for my "small web apps/web apis" needs. For my personal performance needs, just running `python myapp.py` is enough, no need for gunicorn or other "complicated" setups.