[isbjorn16]

I always love coming into these Python threads. It's extremely cathartic to know there are enough people who hate the language as much as I do, even if we are stuck with it for any number of good reasons.

All of the reasons we're stuck with it related to datascience and scientific computing are the exact reasons why we hate it. It's exceptionally fast to write code that does a very specific task without sparing a single solitary thought for any sort of software development best practices. It's incredibly approachable, which means it can be used by any keen scientific or mathematical minds who spend more time reading and writing whitepapers than they do writing code. The code, as an entity in and of itself, clearly shows that. But the _output_ of that code also shows that, which is why it isn't going to go away or be swapped out for anything else.

This is like the ideal breeding ground for further scientific or academic pursuits, and since everyone is standing on the shoulders of numpy/scipy/sklearn/pytorch-esquian giants, you're stuck in the scenario of having to replicate all of these libraries in any other language. I can't honestly say it's worth the effort to replace the entire stack, and it's also not going to help when these same researchers continue to use python going forward and start using some other library that you didn't port yet.

And so, they will continue pumping out new papers and prototypes-masquerading-as-quality-libraries, and we'll continue roping them in and duct taping fix after fix after fix on this kaleidoscope of horrors and we'll continue self medicating to dull the daily frustration of python while also accepting that it's the price we're going to need to pay if we want to work with these brilliant minds with all their neat new ideas pushing forth such a breadth of new understanding from the data _we already have_. Or at least that's what we say when we try to ignore the growing bald spot and bursts of incandescent rage every time we get a "quick script that needs a little polish" that clocks in at 2000 loc and literally doesn't have a single function definition in it and variable names that identify up to 14 different types depending on some global state and...

You know, I'm going to stop typing this. It's christmas eve and I wasn't planning on being this angry today

Edit: Some people weren't liking this, it may be because they think I'm being sarcastic when I talk about these brilliant people. I want to just clarify - I am not being sarcastic. They ARE brilliant. They're far more intelligent than I, and they're really pushing the boundaries of human understanding in mathematics and sciences. I acknowledge they're bring a whole hell more good to the table than they are bad, and there's a damn good reason why I am going to be doing this thing I'm doing in perpetuity, or until I can't handle it anymore.

But that doesn't mean their coding practices aren't often _abysmal_, and I won't apologize for my frustrations in this and the language that just _lets_ them do it.

[p1esk]

I'm a deep learning researcher. I have no idea what you're talking about. When I crank out some Pytorch code of questionable quality it's not because of Python. It's because I either don't know better, or don't care. Switching to another language won't change this.

[isbjorn16]

Some languages force you into better practices. The extra overhead that you may not want to deal with - and that learning curve - are the exact sorts of things you want to do if you want to write code that is of less questionable quality.

Good Python looks a lot like what you would get from a compiled language, and every researcher I work with looks at it and goes "holy shit this is so much extra work I don't wanna do it"

And like I said, that's fair. The code is not the point, it's just a tool to get the data - that's the point.

I equate it to the one time use jigs woodworkers use. Some can get pretty fancy and be awesome, but most are just slapped together and will probably get chucked in the bin. I don't mean to shame anyone - there's a reason for it, and it serves it's purpose.

It's when someone hands you that shitty-ass jig, from a language that lets you do some pretty heinous things by design, the frustration builds like crazy! I've seen some pretty gnarly Java and C# in my day too, and all I know is the worst Java still is an order of magnitude easier to handle than the worst Python I see on the regular, it's wild.

[p1esk]

Can you please show me some examples of what good Python code looks like, and point out the overhead? Next month I will be working on a major redesign of a fairly complicated simulations code (Pytorch) with the goal of making it more flexible, and incorporating some new features. This code will be used by many others, so I want to follow good software engineering practices.

[isbjorn16]

I don't off hand, but I didn't want to leave you hanging too long without saying anything either.

The main two things that get me all bent out of shape is having a purposefully disjoint types passed in as the same variable with a bunch of logic around handling either of them, often in very different ways, and never, ever checking types of values or ranges.

If you start throwing in type hinting and make use of mypy, it keeps your own code pretty coherent. If you do need to have disjoint types coming in, spend a lot of time thinking about how you want it to work. It may be reasonably cheap to force everything into a single type from the many possible types coming in, which should simplify things a lot. If that won't work, consider wrapping any of these types in a composite object that unifies the _how_ of accessing the data inside the type into a single low-cost abstraction. Whatever you do, don't let the logic about how to operate over your abstract data input bleed into the logic of how you're building off of it.

This is one of those things that Python makes hard, not because it purposefully stops you, but because it makes it _so very easy_ to spew implementation details through every single function call. It's easy, and programming is hard, and people have deadlines and it's something you can easily convince yourself you don't need to do, and suddenly you're writing bad code and nothing and nobody is there to stop you from it.

I'm a big fan of keeping my type hints in my code and not using typeshed; I WANT people to see what types I'm agreeing to support. You don't have to agonize over documentation or look 14 levels deep in my code to see what I'm handling. The goal here should be for someone to read my function signature and go "oh, okay, I know exactly what I need to provide for use".

Another vexing thing that comes about ALL the time in dsci/ml code is single letter identifiers. A lot of this is because the paper says this variable is `p` or maybe even `epsilon` so... that's what the variables get named. I've even seen `f()`, `g()`, and `h()` in the wild, and of course there wasn't a lick of documentation around it.

Unless your audience is only ever the people who wrote the paper itself, or those who studied it vigorously (more so than just reading it), these are terrible choices.

This advice is not python specific - it's language agnostic, but I notice it most in Python solely because I run into it the most dealing with non-devs taking a stab at writing their first libraries (or, rather, a bit of polish on their initial prototype). But in general, write your code not so that it's easy to write, but so that it's easy to read, easy to reason about, and doesn't require a chain of whitepapers to understand. The goal is so that someone reading this later doesn't have to literally be you, at this exact moment in time, to understand what is going on. You want your code to live long past your current attention on it, so write it in a way that is easy for another maintainer to pick up and run with.

In the end, other languages make some of this table stakes - not the naming, obviously, but the types? Range checking? Handling error conditions explicitly? Python gives you all the rope in the world necessary to bind yourself into a knot with, so it's all up to the developer to do the right thing. When I lament python, it's not because it's inherently bad, but because allowing people to play fast and loose with the rules means you're going to find a lot of people who just don't give a shit.

[legerdemain]

Right on! It's not the job of ML specialists to waste their time polishing "clean," "performant," or "correct" code.

[indymike]

You know, what you are describing sounds exactly like most production code I've inherited over the years. It's what imperative programming devolves into. Start at the top, run to the bottom. You see a lot less of this with compiled languages, and I suspect that has a lot to do with the additional complexity of actually compiling the code filtering out people who don't want to be bothered with details. Enjoy what's left of Christmas Eve.