I've been seeing a lot of Python vs Go stuff lately and I think a fair amount of the folks involved in these are not aware of general Python web architecture patterns.
Of course something compiled directly is going to be a bit faster, but development time is important too. Python has more libraries and is (for many people) probably faster to write.
Serving multiple requests is best utilized using a preforking webserver in front of Python, whether Apache, nginx, etc. This allows multiple requests in without any async voodoo code. Twisted for example is not the right answer in this case, because it doesn't get you multiple processes and messes up the way you write code (async event driven code is more time consuming to write/debug).
On the backend, your webserver does not start longrunning backend processes, but you can launch them using things like celery, which is a process manager that allows you to start jobs and so forth. Celery can run on any number of machines, and your backend can scale independently of your frontend if you wish.
Historically, some very computational parts of Python were often written with C bindings. While I haven't done so, things like Cython may also be promising for extensions. There's also things like ctypes for quickly just taking advantage of native libraries in a Python function.
Personally, given, I like how Go has things like channels, but I would never adopt a programming language for just one specific feature when I lose out on other features that are valuable to me, for instance, an object model.
(I'm also really curious to see how the typing options in Python 3 play out)
Anyway, I mostly wanted to point out as most people are doing web services that you should be fronting Python with some sort of web server that allows preforking, and then the concurrency issue, in my experience, becomes not a thing.
Many backend libraries can easily take advantage of libs like microprocessing, which are not the most 100% friendly in their more complex IPC-type cases, but are pretty workable.
I've been seeing a lot of Python vs Go stuff lately and I think a fair amount of the folks involved in these are not aware of general Python web architecture patterns.
I absolutely agree, but I also think that deploying python on the web still has too much of a learning curve. Even the standard nginx > gunicorn > wsgi model is kind of a pain. Couple that with celery, and init systems, and you're basically down a sysadmin rabbit hole.
> Anyway, I mostly wanted to point out as most people are doing web services that you should be fronting Python with some sort of web server that allows preforking, and then the concurrency issue, in my experience, becomes not a thing.
Spot on. Concurrency vs parallelism, and clean distinction of responsibility (web server vs backend threads).
> Many backend libraries can easily take advantage of libs like microprocessing, which are not the most 100% friendly in their more complex IPC-type cases, but are pretty workable.
This is painful. In Javascript I can be careless (well to a great extend for people like me likes magic) using promises. Python can achieve this too but with a great effort of learning either coroutines, gevents, or asyncio. Though I have to admit that Javascript has its own problem facing parallelism.
I have done things with gevents, spawning greenlets and respond to user immediately. The thing is, backend should always be stateless, so worker models like celery and rabbitmq pub/sub and etc are more popular.
> Personally, given, I like how Go has things like channels, but I would never adopt a programming language for just one specific feature when I lose out on other features that are valuable to me, for instance, an object model.
That really depends on your requirements. If you need multithreading (not multiprocessing), you cannot use Python.
That really depends on your requirements. If you need multithreading (not multiprocessing), you cannot use Python.
About to show my ignorance, but when is multithreading useful when multiprocessing is not? Assuming it is a use case that is suited for a high level dynamic language to begin with.
Multithreading lets you share memory. Multiprocessing requires you to serialize your data and copy it to the other processes. That can be very expensive.
This only matters for performance-critical applications.
Let's say you have a REST request come in. In order to fulfill that request you need to make 10 REST requests of your own to various back-end systems. If you can make some of those requests asynchronously you'll greatly reduce your response time. While I suppose you could mangle your way through it multi-process, it seems like a bastardization of the model.
If you're just sending HTTP requests, regular threading in Python works fine - waiting for a socket response doesn't block the execution of other threads.
Python threads are real threads, and things like blocking socket IO does not block Python execution in other threads.
While I suppose you could mangle your way through it multi-process, it seems like a bastardization of the model.
That kind of sounds like you're nitpicking over an implementation detail. Which brings me back to my original confusion. If I have an interface that allows me to articulate concurrent tasks, I don't see the difference between threads, processes, or separate machines.
Again, this is assuming we're not in a resource constrained environment, and we're not doing something so processor heavy that it really should be compiled.
The GIL only restricts utilization of multiple cores. This makes parallelism harder but not concurrency. Python fully supports threads, but you won't find two threads in the same process running at the same time.
Small clarification - you will if they are down in C code. i.e. you can have multiple threads in Python blocking on IO just fine. Or three threads concurrently running C code which releases the GIL until it needs to be re-acquired. Quite a few standard libraries which require significant processing power do exactly this.
> Personally, given, I like how Go has things like channels, but I would never adopt a programming language for just one specific feature when I lose out on other features that are valuable to me, for instance, an object model.
I'm no sure go creators would agree with this. The go object model is different enough from other object models that it would not qualify as one for a lot of people.
These are great ideas. I told Valentin to drop by. Because DAS is an aggregator with an expert-system style query language, there is sharing among the Python services for caching. It's the caching that makes async code a good option. Preforking might work against this without significant increase in complexity in order to communicate with a single local cache. Remember that this is a web service for the Large Hadron Collider. Nothing about that is small.
Basically their Python version ("3 thread pools, 175 threads") is synchronous and single (OS)-threaded, while the Go rewrite uses goroutines and multiple OS threads. The fact that their Python version takes "minutes to startup" indicates that a rewrite was necessary anyways.
Go is a good tool for the job, Python threads are not. asyncio or one of the event-based IO frameworks should work much better.
As for the problem of sharing data between processes (slide 5): it appears that this service is read only? If that's true, what do you need to share? Every process can have it's own connection pool. You don't even need multiprocessing, just use SO_REUSEPORT and start your application multiple times.
You could probably get decent performance for a similar application written in another language (then Python) using 175 threads. 175 threads is not that big of deal, the OS can manage it pretty well. It's only when you start talking about thousands of individual connections and thousands of threads that you need to worry. Python sucks at that at low number of threads (GIL).
175 threads use a lot of memory and cause a lot of context switching. I would never write an application so that it needs 175 OS threads, because if it needs that many, how many am I going to need down the road? It's an ominous sign for scalability in my view, even if it works for a while.
[Edit] I'm a assuming a CPU with 8 cores, not some 64 core monster.
175 threads really don't use that much ram. I know userspace stacks are large by default but most apps don't use them and they are never materialized. So even if you're using 1MB of stack space for each one that's only 175MB. You can easily fit that on whatever is the smallest AWS instance.
Additionally there are many legitimate cases for for a lot of threads like disk IO. If you find your self having to push a lot of bytes to/from a high iops drive like an SSD / NVM drive. Unless you're doing large sequential transfers that you can do in one large call, you will needs submit many concurrent request to saturate the drive (via threads). Disk IO is not network IO.
To be honest, I don't really have a good intuition or hard data on where the context switching overhead (or other limits) starts to bite, because I have always avoided architectures that go into the hundereds or thousands of threads.
Maybe you are right and it's one of those urban myths that we sometimes carry over from times long past based on assumptions that are no longer true.
I would love to have more hard information on that one, because I think that the currently fashionable async/event based way of doing a lot of things makes programs much harder to understand and write.
Good rule of thumb for modern kernel and server class hardware : 100's of (native) threads is ok. 1000's will probably be ok. 10's of 1000 is where you will start to see trouble and 100's of 1000 will most likely cause you to pull out your hair. So the comment above about 175 threads being too many is incorrect.
Typically rewrites like this focus on core functionality; I truly down the project is a 1:1 equivalent. There may be factorings, as well (functionality included as part of Go).
I rewrote a python webservice in go once and found go need A MORE boilerplate code because of json structures need to be strictly defined for better or worst.
In python, it is at lease 10x less code for json parsing. json.dumps(), json.loads() is basically what I needed. The exception handle fill in all the undefined easily.
Also, go used a lot more memory because the GC is not under my control. In python, I can tell the GC to collect and one can see the memory shrink immediately. In go, that was not the case for me. A program that build GB of search index database in go end up using 4x the amount of memory as compare to python. Golang at that time (2+ years ago) lack the gc debugging infrastructure for me to resolve the problem.
That said, I'm not really surprised about the performance details. My experience was that Go made it pretty easy to "light up all the cores" on a machine. I say this as a person who spent a lot of time releasing the GIL for multithreaded C++ code hiding behind python front ends.
That's a design choice, but even if you always use getters and setters in Java and never in Python, the program would have to consist exclusively of getters and setters to get you to 2:1 (or 3:1 if you count closing brace lines).
What makes Java code look bloated is a culture of overdesign, not so much the language itself.
Not really. It's just a report written by someone who has an existing code infrastructure and is experimenting with alternative approaches so wrote some basic scripts for benchmarking.
While it may be true, when it is posted without the proper context and an unbiased way to assess the outcome, this presentation will be used as "proof" that Go is better than Python. (which it might be, but not everywhere)
Possibly. But you'd expect most developers to be smart and impartial enough to read these statistics and take them as a case study rather than hold them as gospel.
Quite frankly, I'm getting a little sick of the arguments that happen on HN whenever a Go-related article comes up (and particularly so with Python vs Go). You get people who seem to hate Go who go all out to criticise Go and/or statically typed languages. Who argue that pro-Go articles are biased; and so forth. And then you get the Go fanboys (of which seem to be less vocal lately) who declare that Go is our lord and saviour and we should be rewriting core OS internals in it. It's just nuts. Sensible people would see that Go is better at some things and worse at others. And that articles like this are just an interesting case study and might not apply to their own personal software problems but not in any way biased beyond the fact that the article is tailored specifically to their own software problems.
I don't hate Go, as much as I detest the breathless fans of the language. I am a "former-ish" python programmer who now uses D for present projects with an eye towards Rust for building libraries and low-level system stuff. I also use Nim in place of Python for a few personal scripting projects.
All these languages are statically typed and in my understanding of Programming Languages, way better than Go.
Sorry, I didn't mean that as a dig against you specifically. Just a generalised comment as I've just noticed a backlash on HN wrt Go.
Nim's often interested me. I'd really love to try it but sadly it's lack of mainstream support means it would be hard to justify using Nim in any professional capacity, which vastly limits it's usefulness to me. And sadly I don't have enough free time to learn languages for fun these days.
I haven't done any real work in Go yet but this sounds like one of the (many) use cases it's well suited to.
Unfortunately this overview is light on any meaningful details. As a general rule a rewrite of any project will result in fewer lines of code, however, in general, a rewrite of any project is a terrible idea.
Given that this seems to be a situation in which you have a lot of blocking waiting for concurrent requests, why not try something like gevent?
It's good for people to try different approaches and technologies. I'm glad they managed to have success with Go, that's good for everyone. It would have been interesting for the reader to see some of the gory details of hacking around with the existing codebase to see some of the ideas that may (not) have worked.
It's a Go rewrite of an existing, and probably old, Python application. You are asking them, who already did a rewrite in Go and kindly provided their assessment of the process, to also to a Python rewrite using more modern approaches.
Feel free to rewrite their old Python app in Python for free. They may thank you and even use your port.
I think they were just comparing their existing Python deployment infrastructure to a generic Go set up (there are also ways to optimise Go that wasn't explored in that article). It wasn't meant as a "look how much Python sucks compared to Go" type article like a few seem to have taken it. More just disclosing the results of some internal testing they've been doing.
On that note, I would suggest that if you think they could see big gains with little code refactoring simply by switching Python frameworks or even to a different Python runtime, then maybe you should contact them. I'm sure the author would be open to ways to increase their throughput with less developer overhead.
Do you have an example in Python of doing fan-out/fan-in? I've done it in Go before, and didn't find it particularly nice to look at (although it did work, and worked well).... so I'm curious what a Python example would look like.
I'm not saying you shouldn't use dynamic languages at all (in fact, I'm developing in one right now), but you should keep in mind that you are paying a computational price for that dynamism every time a line of your code is executed.
Static languages don't take that much longer to write than dynamic languages. But on the flip side: a more performant software stack (regardless of language paradigm) does reduce your sysadmin time due to them having to maintain a smaller server cluster, as well as reducing your hardware / cloud costs. Generally speaking, of course. But this is quite a generalised discussion as is.
I once worked on a project that was written in Groovy (a hipster version of Java, so to speak). At some point I have converted it from dynamic to static, by adding the @CompileStatic directive and adding some type declarations. It became MUCH more productive and maintainable. If I would start a new project on the Java platform right now, I would be certain to choose Groovy with @CompileStatic instead of Java. It has all the good things of Java without all the bad things.
I've found Groovy's only good for dynamically typed code...
* the stuff you want to write quickly but don't mind it running slowly, i.e. throwaway code
* the small stuff you know won't become a larger system one day, e.g. 30-line Gradle build scripts and code testing Java classes
* when you know you won't be upgrading to Java 8, which Groovy hasn't kept up with syntactically
Groovy's static compilation was tacked on for version 2.0 and doesn't work, except for sprinkling the occasional @CompileStatic around your code in a trial and error fashion. You didn't actually say you HAD started a new statically typed project in Groovy on the Java platform. If you do, use Java 8 which makes much of what Groovy brought to Java 7 redundant, or another language written from the ground up for static compilation, e.g. Scala or Kotlin.
In fact, I've found even for code testing Java classes, using Clojure is more productive than Groovy once you get over the syntax hurdle because macros can eliminate verbosity in repetitious test scripts in a way functions can't.
In the 1990s when Python et al were being created, static typing really sucked, plus you were almost always working in a really complicated environment like Windows or something where the details of the OS were sticking out of the types of way too many of the function you wrote.
So it was easy to switch to Python and come to the conclusion that it was all the static typing's fault, because that was the most obvious difference between the languages of the time and Python/Perl/etc.
But it's 20 years later now, and the statically-typed languages have not been standing still. They're a lot easier to work in now, even for prototyping in my experience. Go, which I mention since it's on topic, is reasonably fluid with its interfaces and structural typing. It isn't quite Python's level of fluid, but I also find it doesn't take that long before the static typing wins start to balance out anyhow when it tells me about problems before they've metastasized because I didn't know about them and kept going.
The whole functional programming world with its type inference is fairly fluid to work in, and when it gives errors, I find they tend to be real errors, even in your prototyping code, that really do need to be addressed before you can compile. (If you get good at it, prototyping is actually really easy in Haskell, but there is an up-front investment.) Optional/incremental typing, despite my general distaste for them [1], also can be easy to work with. And even some of the old school statically-typed languages are easier than they used to be, because you can find some abstraction layer that isolates you from the sort of thing that made the 1990s a really crazy time to program in.
So, it helps to use a language that doesn't date back to the time of Python's origination.
The only thing inconvenience that truly static typing brings to prototype code in my experience is that it really wants you to push a given change all the way through the code base, when you may just want to experiment with it on one code path. Generally there's a way to just beat the compiler down, though. And when you've learned the art of using types to mean things, and when you've learned to read what type errors mean and that they are not just a content-free shouting of "NO!", the ability to harness what the errors mean and react to them quickly in your design, rather than bake them into your code accidentally, can end up being a net gain, even in mere hours of work.
[1]: Mostly a consequence of the fact that, as I'm describing in the rest of this post, static typing is a lot easier than it used to be, which means I consider incremental typing to mostly be solving a problem that no longer exists. YMMV.
> But it's 20 years later now, and the statically-typed languages have not been standing still. They're a lot easier to work in now, even for prototyping in my experience. Go, which I mention since it's on topic, is reasonably fluid with its interfaces and structural typing.
What type system feature does Golang have that was invented post 1990?
Interfaces certainly were not created post 1990—Java had them, to name one obvious example—and structural typing is very old as well (and I don't think the sort of pseudo-structural typing that Go has for interfaces is particularly important for usability anyway).
It's not that painful to write in large quantities, at least no more so than other static languages. And (given the IDE support), it's significantly easier to refactor code in Java than a dynamic language.
This has never been true for me at all. On the contrary. So either people differ a lot in the way their brains work or it's because dynamic languages are often used for different tasks than static languages. Maybe both. I'm not sure.
Can anyone comment on what the CMS DAS web service is? I'm having a hard time understanding what it is supposed to do. I'm sure the audience knew or maybe it's obvious and I'm just missing something.
It's essentially a way to look up meta data about the different data files produced by the CMS detector. There are petabytes of data produced by the detector and these are stored in countless data file. In order to determine which datasets are available and right for your particular analysis you would use the DAS system to look for them and find out where they're located. This is a complicated task b/c the petabytes of date are distributed across the CMS computing grid that spans many dozens of institutes across the globe.
Of course something compiled directly is going to be a bit faster, but development time is important too. Python has more libraries and is (for many people) probably faster to write.
Serving multiple requests is best utilized using a preforking webserver in front of Python, whether Apache, nginx, etc. This allows multiple requests in without any async voodoo code. Twisted for example is not the right answer in this case, because it doesn't get you multiple processes and messes up the way you write code (async event driven code is more time consuming to write/debug).
On the backend, your webserver does not start longrunning backend processes, but you can launch them using things like celery, which is a process manager that allows you to start jobs and so forth. Celery can run on any number of machines, and your backend can scale independently of your frontend if you wish.
Historically, some very computational parts of Python were often written with C bindings. While I haven't done so, things like Cython may also be promising for extensions. There's also things like ctypes for quickly just taking advantage of native libraries in a Python function.
Personally, given, I like how Go has things like channels, but I would never adopt a programming language for just one specific feature when I lose out on other features that are valuable to me, for instance, an object model.
(I'm also really curious to see how the typing options in Python 3 play out)
Anyway, I mostly wanted to point out as most people are doing web services that you should be fronting Python with some sort of web server that allows preforking, and then the concurrency issue, in my experience, becomes not a thing.
Many backend libraries can easily take advantage of libs like microprocessing, which are not the most 100% friendly in their more complex IPC-type cases, but are pretty workable.