FWIW, I wrote isort, but am seriously considering migrating my projects to use Ruff. Long term I think the design is just better over the variety of tools we use within the Python ecosystem today. The fact we have a plethora of projects that are meant to run per a commit with each one reparsing the AST independently, and often using a different approach to do so, just feels untenable long term to me.
Yes, as of last month. I’m not sure if it works with YAPF; it’s designed to work with Black and doesn’t currently have many of isort’s configuration options. Worth a try!
Yes. The code started as being indented by tabs, so changing it now is a mess. Also, not to start a flamewar, but I've always preferred tabs to spaces in any language, I find them more reliable, easier to use...
Not going to argue about your personal preference, but in Python spaces for indentation isn't just a personal preference - you'll run into a fair amount of issues with that, this one being just the tip of the iceberg.
Apart from isort not knowing how to deal with tabs, and changing the defaults "spaces" to "tabs" in the tools (along with line length too, because for some reason the 80 character limit is set in stone even though it's absolutely outdated) I haven't had many issues. And this has been running for years already.
If your import statements are indented, they must be in control statements (try/except or conditional imports, I fail to see why you would put those in a loop) thus they will be difficult to reorder if you import another module (with different name or stdlib status) on import error.
> If your project builds a Docker container, also create a .dockerignore file to specify files and directories that should be excluded from the container.
I would nitpick this. You build images not containers and since files are not copied by default there is more nuance here that the .dockerignore file makes builds faster by not including them in the build context.
That does ultimately prevent COPY directives from using them but it is these sorts of brief, slightly inaccurate summaries that mislead folks as they build understanding.
Shouldn't the speeding up of the build make the program less boring?
From my understanding, the program gets more boring as the time it takes an application to build increases.
> slightly inaccurate
Not entirely, I'm not sure the author even wanted to stress on this in the article. People won't learn docker from a python article about the same.
Not sure if I like the recommendation to not let Black change your code and just give out errors.
I absolutely let Black change code and see the value in Black that it does that so the devs do not have to spend time on manually formatting code.
Black shouldn't break anything (and hasn't broken anything for me in the years I used it) but in the unlikely case it does it, there's still pytests/unittests after that that should catch problems...
As I understood it, it was to not let black do the formatting during CI builds. In local dev you’d let it reformat.
Even while it won’t break anything you want CI to be your safety net, flagging a local setup as being wrong is more valuable than magically autocorrecting it.
> CI/CD has no business changing your code; it builds stuff using it, exactly as if commit such-and-such.
That going too far unless you define code to be a subset of the files checked into the repository and simply define any file that's touched in an automated manner to be not code
There are a lot of useful automations that can be part of the CI/CD pipeline, such as increasing a version number, generating a changelog, creating new deployment configuration etc
They don't have to be part of it and it's possible to work around it/don't commit... But that comes with it's own challenges and issues
It's not about not trusting automated generation of certain artifacts. Stamping version numbers may be a fair idea. (A changelog entry needs human review and approval, in my book.)
It's mostly about the flow of data and control: source files, beside some known auto-generated files / single lines, are the source, and whatever is generated is downstream from them, not altering them. It's like a React app: data flows through props in only one direction, you don't patch DOM in event handlers, or something.
Its completely fine if you wish to treat your repository like that, but that doesn't make this treatment the only viable and correct way.
There are a lot of people which include their helm package as part of their project repository and even more that generate the changelog from standardized git messages like conventional commits and still wish to persist them in a changelog.md inside of the repository to make completely banal examples spelled out to the letter. It works well and lets you scale these things pretty well inside of a corporation with a lot of teams.
Its completely fine to keep all that out of your automated pipeline, triggering only after these things have occured... but that's not the only viable choice a developer can make and you're very much talking from of an extremely limited perspective if thats your point of view.
This is the way. We have format-on-save in our editor, works like a charm. Sometimes the CI still catches sometimes, but generally its very low friction.
My current project is my first project in a while which does not use black.
I liked black, though I was never satisfied with the fact that there was no way to normalize quotes to be single quotes: '. Shift keys are hard on your hands, so avoiding " makes a lot of sense to me. But there's the -S option that simply doesn't normalize quotes so it has never been a real issue.
However, this new project has a lot of typer functions with fairly long parameter lists (which correspond to command line arguments so they can't be broken up).
black reformats these into these weird blocks of uneven code that are very hard to read, particularly if you have comments.
Everyone is a fan of black; no one liked the result. :-/
I have a key in my editor to blacken individual files, but we don't have it as part of our CI. Perhaps next project again.
> I absolutely let Black change code and see the value in Black that it does that so the devs do not have to spend time on manually formatting code
100% this. I also let Black auto-format code in the CI and commit these formats.
A lot of developers, intentionally or not, don't have commit hooks properly setup. If Black doesn't change the code in CI they need to spend another cycle manually fixing the issues that Black could have just fixed for them.
You're saying that there's a risk that Black could break your code when formatting? Well, so could developers and I'd trust a machine to be less error-prone.
I don't understand why people are against this so much. Black does a sanity check and compares the AST before and after to make sure there aren't any meaningful changes (unless you are running it with --fast). So there is almost no risk that it will break your code.
There is nothing more frustrating than coming back from a coffee break only to find out that you have to rerun your CI check because of a trivial formatting issue.
If you use feature branches, it is quite annoying that can't push two consecutive commits because CI changed something in your branch and now you have to resolve conflicts.
We have it set up to only run after you've opened a PR. You shouldn't run into the issue often that way because if you're opening a PR then all your big immediate changes are already committed so you won't be pushing another commit until someone reviews which will be after black has already finished.
Sometimes I still sneak in minor changes after I opened a PR. Sometimes I open PR early because CI has integration/e2e tests that are hard to run locally. Sometimes I want feedback on certain parts early on and PR is the easiest way to show something.
There can be workflows in which pushing through CI works fine, but as a general advice it's not great because there are many edge cases.
> Not sure if I like the recommendation to not let Black change your code and just give out errors.
Let black format code before it is checked in. Code should not be reformatted for CI or production, and bad formatting should either ALWAYS throw errors (no known defects allowed) or NEVER throw errors (if it passes tests & runs ship it). Consistency is the key.
Even since the start of python typing, it was recommended to use a more generic type like Iterable instead of List. The author claims that List is too specific -- this seems like a straw man argument against typing that doesn't acknowledge python's own advice.
Also, mypy has gotten really good in recent years and I can vouch that on projects that have typing I catch bugs much much sooner. Previously I would only catch bugs when unit testing, now they are much more commonly type errors.
The other thing typing does is allow for refactoring code. If anything, high code quality relates to the ability to refactor code confidently and typing helps this. Therefore I would put it at the top of the list above all the tooling presented (exception I agree with ci/cd)
Iterable is an import away, while list is already at my fingers.
There's zero harm in using list in private interfaces: I know I'm the only one passing the value, I know it is always a list.
As an argument type, Iterable is compatible with list, so it's benefits are minimal (with rare exceptions).
Lists are easier to inspect in a debugging session.
Iterable can be useful as return type, because it limits the interface.
Iterable is useful if you are actually making use of generators because of memory implications, but in this case you already know to use it, because your interfaces are incompatible with lists.
I can count on fingers of my hands when using Iterable instead of list actually made a difference.
> As an argument type, Iterable is compatible with list, so it’s benefits are minimal (with rare exceptions).
Iterable is not compatible with list, but list is compatible with iterable. As the more general type, Iterable is better as an argument type unless you have a reason to force consumers to use lists. Even in private interfaces, I tend to prefer it, because I often end up wanting to pass something constructed on the fly, and creating an extra list for that rather than using a genexp just seems wasteful.
What I meant is argument marked as Iterable is compatible with list being passed.
> Iterable is better as an argument type unless you have a reason to force consumers to use lists
See, I feel the exact opposite: I use Iterable only if I have a reason to force consumers to use Iterable.
When you're marking argument as Iterable, how confident do you feel that you will never query collection size or access it by index?
I understand the desire to limit the interface and YAGNI, but since lists are more familiar and ubiquitous, using Iterable feels more complicated and unnecessarily pedantic.
> See, I feel the exact opposite: I use Iterable only if I have a reason to force consumers to use Iterable.
A broad argument type doesn’t force consumers not to use a narrower type. (It forces the implementer of the function to not rely on additional features of the narrower type, but if I am writing the function, I can be certain whether or not that is acceptable.)
Meanwhile, using a narrower type than needed for an argument does impose additional, unnecessary constraints on the consumer.
> When you're marking argument as Iterable, how confident do you feel that you will never query collection size or access it by index?
Absolute certainty, since I know what the function does and what I need to do it.
> I understand the desire to limit the interface and YAGNI, but since lists are more familiar and ubiquitous, using Iterable feels more complicated and unnecessarily pedantic.
Since all lists are Iterables but not all Iterables are lists, Iterables are necessarily more ubiquitous than lists.
> Since all lists are Iterables but not all Iterables are lists, Iterables are necessarily more ubiquitous than lists.
Yeah, that's what I meant by being pedantic :)
Here's a question: you receive a JSON payload that contains a list. You will then pass this list to two functions, one of them only iterates, another one uses list interface (let's say checks length among other things). Should you mark the argument as a list, or as an Iterable in the first function?
Solely from the code perspective, it's definitely an Iterable. But in my mental model it still remains a list. I don't like it when code deviates from my mental model. Forcibly treating it as an Iterable only makes it more complicated, while not giving anything in return.
Sure, you could say that callee should not have expectations of the caller, but what if those functions are already coupled? They are in the same module, and argument names clearly denote a collection. The fact that in certain scenarios it is "technically Iterable" serves nothing but pedantic value.
> The other thing typing does is allow for refactoring code.
No. What allows you confident refactoring code are automated tests. I honestly can't understand why people are so obsessed about types, especially in languages like Python or Javascript.
It's not just about types. It's about having interfaces I should expand on. And, I'm assuming there are automated tests, otherwise and typing is additive. I should clarify that it's also having defined interfaces using the type system to do it.
By depending on interfaces/abstractions instead of specific cases you can refactor the interface and not break clients. It's very difficult to do this unless you have types.
This is something that Go is really good at and encourages but can be done with python/js on top of their type systems.
> I honestly can't understand why people are so obsessed about types,
Types in Python feel like an added layer of confidence that my code is structured the way I expect it to be. PyCharm frequently catches incorrect argument types and other mistakes I've made while coding that would likely result in more time spent debugging. If you don't use any tools that leverage types you won't see any benefit.
> I honestly can't understand why people are so obsessed about types
It's a very powerful sanity check that lets me write correct code faster, avoiding stupid bugs that the unit tests will also, eventually, find.
And, to me, reading the code is much much nicer. Types provide additional context to what's going on, at first glance, so I don't have to try to guess what something is, based on its name:
> I don't have to try to guess what something is, based on its name
It's probably just a bad example, but in case it isn't:
Sounds like you ended up at the same place. You went from guessing what is some_api.get_results(), based on it's name, to guessing what is SomeAPIResult, also based on it's name.
If some_api is your library, then you could have just added type hints to get_results() and let type inference do it's job.
If it's a third party library, then using your custom SomeAPIResult means that code is becoming alien to other engineers that worked with that library in the past. It might be worth it, but it's definitely controversial. You probably should've done it with stubs anyway.
I disagree. It’s not a guess, it is precisely what it is, where the variable name is free to betray me. A sane IDE/linter will tell me if my local assumption is incorrect, where a variable called result_SomeAPIResult relies on an assumed, possibly ancient, state of reality.
I got good use of the run-time type checking of typeguard [0] when I recently invoked it via its pytest plugin [2]. For all code visited in the test suite, you get a failing test whenever an actual type differs from an annotated type.
> Even since the start of python typing, it was recommended to use a more generic type like Iterable instead of List. The author claims that List is too specific
These statements contradict themselves? List is too specific, and Sequence[item] is preferred. Sometimes you are dealing with a tuple, or a generator, and so it makes more sense to annotate that it is a generic iterable versus a concrete list.
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. Yet the Python type-annotation ecosystem was strongly oriented around nominal typing (i.e., caring that something is exactly a list) from the beginning.
I'm saying that this quote is a straw man and that contrary to what is claimed in the quote, instead, the ecosystem would go with/recommend Iterable[Item] or Sequence[Item] and not List[Item] if applicable.
I think we both agree, not sure which part of my comment you think is contradictory.
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it.
This is an odd complaint. typing.Sequence[T] has been there since the first iteration of typing (3.5), for exactly that use case, along with many related collection types.
You should never be using static typing with a scripting language like Python or Ruby.
Dynamically typed code is 1/3rd the size of statically typed code, that means that one developer who is using dynamic typing is equivalent to 3 developers using statically typed code via MyPy.
Since the code is 1/3rd of the size it contains 1/3rd of the bugs.
This is confirmed by all the studies that have been done on the topic.
If you use a static type checking with Python, you have increased your development time by 3 and your bug count by 3.
Static typing's advantage is that the code runs a lot faster but that's only true if the language itself is statically typed. So with Python you have just screwed up.
Ah no I meant a proper peer reviewed source. The claim that untyped code has fewer bugs is completely bonkers, so I was quite sure that no such source existed.
Why do you think microsoft, google and facebook are all in the business of typechecking python? If typechecking would actually introduce bugs, it'd be better not doing it right?
Using github for statistics is flawed. There are millions of 10 line js libraries. Yes it's easy to not make type mistakes in 10 lines. I suppose that type errors increase more than linearly with size.
> The claim that untyped code has fewer bugs is completely bonkers
Not really. It is, however, quite expensive to measure, because dynamic typing really shines at the evolution of software, that is being able to respond fast to changing requirements. Legos vs play-doh: https://weblog.jamisbuck.org/2008/11/9/legos-play-doh-and-pr...
> Why do you think microsoft, google and facebook are all in the business of typechecking
A billion flies can't be wrong? Companies with unlimited amount of money are not the right place to search for good practices. Both Facebook and Google became flush with cash way before modern type obsession. Sure, once you are a multi-billion dollar company slowing down can be a good thing. But you need to get there first.
> If typechecking would actually introduce bugs, it'd be better not doing it right?
If sugar caused us to die sooner, we'd be better to eating too much if it, right? And yet, here we are.
"The claim that untyped code has fewer bugs is completely bonkers"
There are plenty of academic sources that will tell you that the number of bugs in a program is directly proportional to the number of lines in the program and static typing has no effect on this.
Additionally, statically typed code involves large amounts of boilerplate code in the form of abstract base classes, interfaces, generics, templating, etc. It's a very verbose code style.
It's your turn, find an academic source to backup your claim that static typing reduces the number of bugs. Cause it just isn't true.
Microsoft, google and Facebook have a lot of programmers coming from languages with static typing and want to make Python more familiar.
It's a far distance away from anything resembling good practice.
Actual Python houses typically don't use static typing.
Please see Raymond Hettinger's keynote on efficiently handling bugs[0]. He makes the case that static type checking is a boon for Python except for in specific programs that make extensive use of covariant and/or contravariant types.
> You should never be using static typing with a scripting language like Python or Ruby.
You should use it where it makes sense, and not where it doesn’t. I haven’t used any of Ruby’s type checkers, but Python makes this easy enough; make what has a reason to be dynamic dynamic, and have static safety rails everywhere else.
(This is true with many “statically typed” languages that have dynamic escape hatches, too, not just traditionally “scripting” languages.)
> Coverage measurements are too easy to “game” — you can get to 100% coverage without meaningfully testing all or even most of your code
Still it's a good low bar for testing. It's easy and rises code quality. I have very good results with coverage driving colleagues to write tests. And on code review we can discuss how to make tests more useful and robust and how to decrease number of mocks, etc.
Hard disagree: 100% coverage is not a "good low bar" and does not increase code quality.
Depending on the language and the particular project, my sweet spot for test coverage is between 30-70%, testing the tricky bits.
I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.
You can have bad unittests which make the system worse and you would be better of without them.
You can also have useless unittests with 100% coverage, which is pretty much the same as bad tests because more code means more bugs and more work. Unittests are also code after all.
The only thing you can say about a very low coverage is that you probably don't have good tests. That's not a very useful metric, since you likely already know that.
The metric 'coverage' is almost useless. Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number. I found it is very useful though if you can generate detailed reports on each line of code or better yet, each branch in the code, indicating whether that line or branch is tested. Eyeball all the lines which don't have tests and ask yourself: would it be useful to add a test exercising this codepath? How do I make sure it works and what cases can I think of that could go wrong? This doesn't automatically lead to good tests, but it helps you spot where you should focus your testing efforts.
Code coverage is a good tool to help think of test cases, as a metric for the total codebase it is nearly useless.
> I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.
It's a red flag to blame high coverage for fragile tests. Use narrow public component interfaces to reach code parts and you simultaneously gain robust tests which can be used during refactoring and you can be guided by coverage to generate test cases. Bob Martin has a great article: https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrav...
One useful technique for checking whether the tests are actually meaningful is mutation testing - mutmut is a great Python implementation: https://mutmut.readthedocs.io
Absolutely not. This leads to testing being invasive and driving the design of your software, usually at the cost of something else (like readability). Testing is a tool, you can't let it turn into a goal.
Could you elaborate with python-oriented examples? I tend to agree for static typing languages like Java — to fully test you have to go a total DI path. It leads to bloat and additional layers. I don't see anything similar for python because you have to do nothing to bring your code into a test environment.
> Testing is a tool, you can't let it turn into a goal.
Yep, and I use testing as a tool to be sure we ship quality code. It's 2x important for our case, we don't have control on hosts where our product is run and 100% coverage was a salvation. We even start to ship new versions without any manual QA.
My examples are just personal anecdotes, you can dismiss them by saying "our team won't make the same mistake".
If your goal is 100% coverage then it will turn testing into ritual and only give you the illusion of quality. Instead of testing inputs and edge cases, you will focus on testing lines of code.
I don't understand. The title of the post is: "Boring Python: code quality". Further down: "Today I want to talk about what's generally called "code quality" - tools to help...". I'm sorry but "code quality" is not "tooling". The post should be titled: "Python tooling". Code quality: What abstractions are you using in your code?, How easy is to make a change?, How easy is to understand your code base?, What patterns are you using and why?, Are you abusing class inheritance?, How many side effects are present out there and how does that affect your program?, Are you taking advantage of the Python language facilities and idioms?, Is it easy to write unit tests for?, etc. To sum up: "tooling" != "code quality".
> This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible.
It doesn't really matter if it is fun, sad, entertaining or boring Python. The post wrongly claims that putting all these tools in a project will lead to "code quality". It says that at the very beginning as I quoted it. This is harmful, especially for a junior developer or someone that doesn't have much or none experience coding. It will make the naive reader believe that having those tools in place quality code is being produced.
While it doesn't produce quality, it can help a developer write better code; I've learned tons about JS internals just by sticking to whatever eslint popped up with. Before that I learned tons about Java's internals and best practices by fixing issues that these automated "best practice" tools came up with.
It's far from perfect, but it helps if you don't know any better. And most people don't know any better.
I'm currently spinning up a new project (React Native, Typescript) and I'm spending a lot of effort in locking down the project - eslint, unit tests & coverage, CI, strict typescript rules, etc - because this did not happen with the previous iteration of this project, leading to tens of thousands of LOC worth of unit- and end-to-end integration tests to become worthless and unusable. Sure, that was a lack of developer discipline as well, but why rely on other people when you can do it through technology as well? You can't control everyone.
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it.
Terrible advice not to use type hints and this reason makes no sense. There's already pretty good support for Sequence and Iterable and so on, and if you run into a place where you really can't write down the types (e.g. kwargs, which a lot of Python programmers abuse), then you can use Any.
Blows my mind how allergic Python programmers are to static typing despite the huge and obvious benefits.
It's true that Python's static typing does suck balls compared to most languages, but they're still a gazillion times better than nothing, and most of the reason they suck so much is that so many Python developers don't use them!
> I recommend using two tools together: Black and isort.
Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
isort's completely random… For example the latest version I tried decided to alphabetically sort all the imports, regardless if they are part of standard library or 3rd party. This is a big change of behaviour from what it was doing before.
All those big changes introduce commits that make git bisect generally slower. Which might be awful if you also have some C code to recompile at every step of bisecting.
Set black up in the pre-commit with a specific version. When you make a commit it will black the files being committed using the specific version of black. As it's a subset, it's fast. As it's a specific version, it's not going back and forth.
It doesn't… people use a million different distributions. Forcing everyone to use a single version of black means that people will just not bother with your project.
The authors of black just don't understand that it'd be ok to introduce new rules to format new syntax, but it isn't ok to just change how previous things work.
This is mostly nonsense and FUD. We have virtualenvironments, requirements files, setup.py with extra_requires that can all be used to manage versions without relying on the particular packages installed on an OS. Most people contributing to open source would be familiar with at least some of these methods and if they are not it’s a good opportunity for learning.
And if they are not, then maintainers can pull, run black over the diff, and commit.
CI prevents poorly formatted code from entering main.
The actual changes between black versions of late have been minor at best. You’re making a mountain out of a molehill.
Having a tool that dictates formatting is a lot less oppressive to new developers than 100 comments nitpicking style choices.
since developing in python should be done in a virtual env to start with, I fail to see how this will be any problem.
The pre-commit documented version of black will be installed in the venv of the project, problem solved.
pre-commit is very useful, in my opinion. When organising code from a lot of Python developers at least, getting the boring stuff like formatting, import ordering, linting, mypy etc. sorted is a time saver.
Two developers on the same python project should also use the same version... with poetry it is straightforward to keep track of dev dependencies. Reorder python imports is an alternative for isort: https://github.com/asottile/reorder_python_imports
> Two developers on the same python project should also use the same version
Why? It is expected for the thing to run on different python versions and different setups… what's the point of forcing developers to a uniformity that will not exist?
It's actually better to NOT have this uniformity, so issues can get fixed before the end users complain about them.
> So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
Any team of developers who aren't using the exact same environment are going to run into conflicts.
At the very least, there must be a CI job that runs quality gates in a single environment in a PR and refuses to merge until the code is correct. The simplest way is to just fail the build if the job results in modified code, which leaves it to the dev to "get things right". Or you could have the job do the rewriting for simplicity. Just assuming the devs did things the right way before shipping their code is literally problems waiting to happen.
To avoid CI being a bottleneck, the devs should be developing using the same environment as the CI qualify gates (or just running them locally before pushing) with the same environment. The two simple ways to do this are a Docker image or a VM. People who hate that ("kids today and their Docker! get off my lawn!!") could theoretically use pyenv or poetry to install exact versions of all the Python stuff, but different system deps would still lead to problems.
Would you please stop breaking the site guidelines? You've been doing it repeatedly, unfortunately. We want thoughtful, curious conversation here—not flamebait, unsubstantive comments, and swipes.
A lot of modern open source projects include a lock file or some other mechanism that ensures that all contributors use the same versions of certain key tools. Obviously there are still going to be some differences in the environment, but for things like formatting, linting, etc, it's generally fairly easy to lock down a specific version.
In Python, the easiest way to achieve this is using Poetry, which creates a lock file so that all developers are using a consistent set of versions. In other languages, this is generally the default configuration of the standard package manager.
The popular Rust tool "ripgrep" uses a lock file for development (you can see it in the GitHub repo), and yet is in the official repositories for homebrew, various Windows package managers, Arch, Gentoo, Fedora, some versions of openSUSE, Guix, recent versions of Debian (and therefore Ubuntu), FreeBSD, OpenBSD, NetBSD, and Haiku.
With all due respect, I don't think you're correct.
Distros can keep their own lock file that is based on their own release branch's versions. If it doesn't build, the pkg maintainer will either file a bug report or make a patch, or neither.
> All those big changes introduce commits that make git bisect generally slower.
Bisection search is log2(n) so doubling the number of commits should only add one more bisection step, yes?
> Which might be awful if you also have some C code to recompile at every step of bisecting.
That reminds me, I've got to try out ccache (https://ccache.dev/ ) for my project. My full compile is one minute, but the three files that take longest to compile rarely change.
Perhaps the poster meant that the contents of the commits themselves make bisection slower? By touching a lot of files unnecessarily, incremental build systems have to do more work than otherwise.
Could be? I've not heard of that problem, but I don't know everything.
I'm not used to Python code (which is all that black touches) as being notably slow to build, nor am I used to incremental build systems for Python byte compilation.
And I expect in a project with 2 developers which is big enough for things to be slow, then most of the files will be unchanged semantically speaking, only swapping back and forth between two syntactically re-blackened representations, so wouldn't an caching build system be able to cache both forms?
(NB: I said "caching build system" because an incremental build system which expects time linear order, wouldn't be that helpful in bisection, which jumps back-and-forth through the commits.)
Every file that changed because rules changed, which shouldn't be frequent, I don't remember black changed radically since its creation, do you have an example of some widespread syntax change ?
1) you don't have one black commit for every non-black commit, do you? Because the general best practice is to do like kuu suggested and have a specific black version as part of the development environment, with a pre-commit hook to ensure no random formatting gets introduced.
2) assuming 500 commits in your bisection, that's, what, about 9 compilations you'll need to do, so it will take you 9 hours to run. So even with a black commit after every human commit, that yes, 1 hour more, but it's also only 11% longer.
Even with only 10 non-black commits and 10 black commits, your average bisect time will only increase from 3.6 hours to 4.6 hours, or 30% longer.
I'm curious to know what project you are on with 1 hour build times and the regular need for hours-long bisection search, but where there isn't a common Python dev environment with a specific black version. Are you using ccache and/or distributed builds? If not, why isn't there a clear economic justification for improving your build environment? I mean, I assume developers need to build and test before commit, which means each commit is already taking an hour. Isn't that a waste of expensive developer time?
And, I assume it's not the black formatting changes which result in hour-long builds. If they do, could you explain how?
As I said in other comments, if you try to force contributors to reproduce exactly your local setup, you will be left with no contributors. Which is why you set up a CI to run the tests… because people will most likely not.
As for build times, it was an extreme example. But even an extra step taking 5 extra minutes is very annoying to me…
> if you try to force contributors to reproduce exactly your local setup, you will be left with no contributors. ...
That's not been my experience. To the contrary, having a requirements.txt means your contributors are more likely to have a working environment, as when your package depends on package X but the contributor has a 5-year-old buggy version of X, doesn't realize it, and it causes your program to do the wrong thing.
In any case, your argument only makes sense if no one on the project uses black or other code formatter. Even if you alone use it, odds are good that most of your collaborator's commits will need to be reformatted.
> .. an extra step taking 5 extra minutes ...
How do black reformatting changes cause an extra 5 minutes? What Python code base with only a couple of contributors and no need for a requirements.txt takes 5+ minutes to byte-compile and package the Python code, and why?
Adding 5 minutes to you build means your bisections are taking at least an hour, so it seems like focusing on black changes is the wrong place to look.
> isort's completely random… For example the latest version I tried decided to alphabetically sort all the imports, regardless if they are part of standard library or 3rd party. This is a big change of behaviour from what it was doing before.
This is not isort! isort has never done that. And it has a formatting guarantee across the major versions that it actively tests against projects online that use it on every single commit to the repository: https://pycqa.github.io/isort/docs/major_releases/release_po...
Hi! I said this with more certainty than I should have. Software can always have bugs! For reference, I wrote isort, and my response came from the perspective that I have certainly worked very hard to ensure it doesn't have any behavior that is random or non-deterministic. From your description, it sounds like someone may have turned on force-alphabetical-sort (if this is in a single project). See: https://pycqa.github.io/isort/docs/configuration/options.htm.... You can do `isort . --show-config
`, to introspect the config options isort finds and where it finds them from within a project directory. The other thing I could think of, is coming from isort 4 -> 5, I wouldn't think it would fully ignore import groupings, but maybe it doesn't find something it used to find automagically from the environment for determining a first_party import. If that's the case this guide may be helpful: https://pycqa.github.io/isort/docs/upgrade_guides/5.0.0.html. If none of this helps, I'd be happy to help you diagnose what your seeing.
> So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
You should never develop using the system Python interpreter. I recommend pyenv [0] to manage the installed interpreters, with a virtual environment for the actual dependencies.
> Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
What's the alternative? YAPF is even worse - it will flip flop between styles even on the same version! Its output is much less attractive, and there are even some files we had to whitelist because it never finishes formatting them (Black worked fine on the same files).
Not using a formatter at all is clearly worse than either option.
Because some people are really bad at formatting code manually and constantly nitpicking them about it is both tedious and antagonistic. Its much better for a faceless tool to just remove formatting from the equation entirely.
I think the sane part of the software engineering world has realised that auto-formatting is just the right way to do it, and the people that disagree just haven't figured out that they're wrong yet.
Maybe you meant "why is Black specifically better than no autoformatting, given that it isn't perfectly stable across versions?" in which case the answer is:
a) In practice it is very stable. Minor changes are easily worth the benefits.
> the people that disagree just haven't figured out that they're wrong yet.
This is unnecessarily confrontational. Please read my other comments where I consider the extra effort that automatic formatting causes for code reviews.
> In practice it is very stable.
It has never happened to me to upgrade black and have it not change opinion about black formatted code.
> Minor changes are easily worth the benefits.
It doesn't matter how minor they are. A 1 bit difference is still going to fail my CI.
> You can pin the version!!
I usually do, but working with old releases that must be maintained, mean that I can't cherry pick bug fixes from one branch to the other, because black fails my CI.
I love this template as well, and wholeheartedly recommend it. There are a couple things you probably don't need (click and nox, for instance, seem only useful if you're really building a couple specific things) but the gestalt of it is really strong. The [article series](https://medium.com/@cjolowicz/hypermodern-python-d44485d9d76...) that spawned the template is worth reading in full.
I would go so far as to say that the hypermodern template, nomenclature aside, is strictly better than the recommendations that the OP put forward both here and in the previous essay on dependency management. Poetry and ruff, for instance, are both very good tools — and I can understand _not_ recommending them for one reason or another but to not even mention them strikes me as worrisome.
I don't work on large python projects, mostly just small scripts that need to work well (integrating with a 3rd party rest api is a good example). I don't do CI or unittests but I use git. This is because it takes time and honestly no one outside of myself would care for small stuff like that. But I do run autopep8 and pylint it (I ignore stuff like line being too long,broad exception handling or lack of docs).
My concern is a) It needs to be reliable (don't wanna spend a ton of time chasing bugs later on) b) How can I write the actual code better? I see what pro devs write and they use smarter language features or better organization of the code itself that makes it faster and reliable, I wish I could learn that explicitly somewhere.
I mean, just the 2.7->3.0 jump was big for me because since I don't code regularly that meant googling errors a lot basically. Even now, I dread new python versions because some dependency would start using those features and that means I have to use venv to get that small script to work and then figure out how to troubleshoot bugs in that other lib's code with the new feature so I can do a PR for them.
I love python but this is exactly why I prioritize languages that don't churn out new drastic features quickly. Those are just not suitable for people whose day job is not coding and migrating to new versions, supporting code bases, messing with build systems, unit tests, qa,ci,etc... coding is a tool for me, not the centerpiece of all I do. But python is still great despite all that.
> I love python but this is exactly why I prioritize languages that don't churn out new drastic features quickly.
What do you mean by "drastic" features "quickly"? Python releases new version once a year these days, and upgrading our Django-based source code with 150 dependencies from 3.4 to 3.11 literally meant switching out the python version in our CI configuration and README.rst every once in a while, no code changes were necessary for any of those jumps...
Our developer README also contains a guide how to set-up and use pyenv and it's virtualenv plugin which makes installing new python versions and managing virtualenvs easy, just pyenv install, pyenv virtualenv, pyenv local, and your shell automatically uses the correct virtualenv whenever you're anywhere inside your project folder...
jumping to python3 was big, but you had plenty of time to prepare for that and plenty of good utilities to make the jump easier (2to3, six, ...). python2.7 itself was released 18 months after python3.0, and by the time python2.7's support ended, python3.8 was already out...
Firsr, to this date, stuff I absolutley need that is in 2.7 i have to either try to fix or venv or somehow get it to work is one of my biggest headaches (Not my code).
Second, yes, all you have to do is switch out the python version to upgrade but let's say you start using f-strings that means all of your users (doesn't apply to django since it is server software) have to upgrade to the right python version including all the deps. But what if your project is a library? That means all other libraries need to use the same or greater python version but what if your distro doesn't yet support the very latesr python version? It's such a nightmare.
New versions should come out no more often than every 3-4 years imho and even then every effort should be made to have those features backward compatible like have a tool that will degrade scripts to be usable on a previous language version.
> Firsr, to this date, stuff I absolutley need that is in 2.7 i have to either try to fix or venv or somehow get it to work is one of my biggest headaches (Not my code).
2.7 was supported for 10 years and it's support ended 2 years ago. There's been ample time to upgrade the code or look for an alternative. If I "absolutely needed" to use a piece of code that I didn't write, is for an unsupported platform and is itself unsupported, I'd absolutely find the time for it. As a developer if I use a library that hasn't been touched for 3 years it's a red flag and I start to look for alternative libraries or forking the code.
> That means all other libraries need to use the same or greater python version but what if your distro doesn't yet support the very latesr python version? It's such a nightmare.
if your distro doesn't support the latest python version you're probably on a very old distro. For example python3.11 installs fine on all supported versions of Ubuntu (18.04+) and Debian (10+) and both Windows (8.1+) and macOS (10.15+). And python3.9 installs fine even on centos7 (released in 2014) and still supports the vast majority of python libraries.
If you're on an OS nearing or past its end of support, you can't reasonably expect all the latest software to work on it. And it's usually fine to just use an older version of python / libraries until you're ready to update.
> New versions should come out no more often than every 3-4 years imho
If new versions came out every 3-4 years, that would mean they would have more drastic changes, because the smaller changes would accumulate over that duration. The longer the "new features" are out, the longer users have to upgrade their system and the longer developers can take getting used to them.
But in the end, it doesn't really matter how often a new version comes out but rather how long the old versions should be supported, right? And I think it's up to the library authors to decide how long to support older versions, not the authors of the programming language.
> ...I'd absolutely find the time for it. As a developer if I use a library that...
Like I said, my day job isn't being a dev which means time for that is rare.
Languages are not user software, they should never be deprecated. I get deprecating standard libraries bur under no context is it ok to deprecate a whole language. Yoh can freeze development and only perform security updates to the interpreter but there is no need to deprecate a language. It is a betrayal of the trusr users put into python when they invested time on it and this is exactly what I mean by avoiding rapidly changing languages. They don't care one bit that people are using their language, they treat like any other software that gets supported and discarded. C89 is still supported! People write new stuff with it. You know why? Because there is nothing to support, just parsing and compiling of a langauge. No new features need to be developed and bugs should be accepted instead of fixed. The interpreter should be available for download and use on any platform for as long as even one guy is using the language.
> you're probably on a very old distro.
Maybe, I usually go for debian but I have run into this issue and it becomes a dependency nightmare on anything that needs the old python version (in the package manager dependency resolver not in python).
> If new versions came out every 3-4 years, that would mean they would have more drastic changes
That's fine, because there would be less versions. All changes are drastic changes from the perspective of someone that is having to google random python errors to figure out what broke and how to fix it. At least it won't be a constant nightmare fixing problems made by the langauge itself in addition to the 3rd party code and your own code. The frequency of how many bugs you have causes by those 3 cause categories should in that increasing frequency. I should not have lnaguage version bugs more often than bugs in my own code.
> But in the end, it doesn't really matter how often a new version comes out but rather how long the old versions should be supported, right? And I think it's up to the library authors to decide how long to support older versions
It does matter because most devs that code for a living like to tinker with new flashy versions so each version of a library they release is that much more prone to requiring newer python versions. The less frequent python releases, the more they will use the current python features before introducing version breaks.
I really think it is a developer culture problem at the end of the day where because you are writing foss code, you don't care about the experience of those who depend on your code.
> Even now, I dread new python versions because some dependency would start using those features
If a dependency breaks compatibility with earlier Python versions because the author wants to use a fancy new feature is not really the fault of Python, is it? Library authors should target the earliest supported Python version they can.
Being backwards compatible (at which Python has been doing a good job since the 2->3 fiasco) is one thing, but trying to be forwards compatible is something else.
Are you suggesting that Python developers should only ship bug fixes so that Python 3.0 can still run code written for Python 3.11?
This isn't a general problem, but I have been seriously burnt by a change from a minor version upgrade.
In 3.8 someone decided that they didn't like the way people were excepting the Exception for cancelled asycnio tasks. So they changed the cancelled task exception to inherit from base exception instead of exception. This meant a bunch of well used libraries immediately had a load of subtle bugs that in normal operation just didn't happen. I can't remember the exact details but I think when the bug did happen the task queue would just continue to grow until we ran out of memory.
This change wasn't a bug fix, more an optimization or an attempt to get people to code a certain way.
I'm all in favour of bug fixes, but Devs shouldn't have to worry about minor upgrades breaking everything.
It doesn’t mean it’s Python’s fault, but it fosters a culture where Python developers who regularly follow the language and are some of the ecosystem’s biggest authors are enticed to trying out the fancy new features (even if the old way still works) because “this is cleaner, this is how I want to do things from now on”.
You should not pin the public requirements that get uploaded with a library (listed in setup.py, setup.cfg, or pyproject.toml), since that will restrict your downstream users, leading to version conflicts and persistent security vulnerabilities.
But it’s totally reasonable to pin the private requirements that you develop it against (listed in requirements.txt, poetry.lock, or similar), updating them every so often during the course of development, so that contributors can use a consistent set of tools.
> For your dependency/versioning issue, use a virtualenv per-project and pin your dependency versions in requirements.txt
requirements.txt is not uploaded to PyPI and has no effect on your package’s dependencies when a user installs it (leaf package or no). It’s only used for developing the package itself, typically in a unique virtual environment.
Agree in principle, but I'm giving advice to someone who programs on occasion and is primarily concerned with their programs breaking due to dependency version upgrades when they come back to them after a little while.
My advice would be to use stuff that is in distributions… it hopefully (not necessarily) is maintained by less noob people who don't break API all the time.
Not agreeing/disagreeing with the message, but the style of writing here is quite nice. It's focused, reasoned, and doesn't make too many assumptions about your tools and environment--and I appreciate that acknowledgment.
One thing that is underestimated is keep the tools version in sync between your app dev dependencies and pre-commit. This also includes plugins for specific tools (for instance flake8). A solution would be to define the hooks in pre-commit to run the tools inside your venv.
About typings: I agree the eco-system is not mature enough, especially for some frameworks such as Django, but the effort is still valuable and in many cases the static analysis provided by mypy is more useful than not using it at all. So I would suggest to try do your best to make it work.
I disagree with this assessment on running a static type checker, although I will admit, every update of python over the past 3 years seems to add more and more typing changes which tends to force global typing updates (looking at you Numpy for python 3.12!)
When python converges on consistent typing across its extended numpy and pandas ecosystem, I believe we will be able to move towards a fully JIT'd language.
> I believe we will be able to move towards a fully JIT'd language.
Unless they actually go ahead with the deferred evaluation of types (PEP 563), make all types strings at runtime and make it impossible to know which type they actually are. :)
What's the current state of the art of managing multiple virtual environments, running tests and running your application?
On Ubuntu and Windows I use Poetry [0], and it works, although it has (had?) some quirks during the installation on Windows.
I liked its portability and lockfile format though.
A few years ago I used conda [1], which was nice because it came batteries included especially for Deep Learning stuff.
I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi.
And then there are also Docker Images, which I use if I want to give an application to somebody that "just works".
I use pip-tools to build a requirements.txt file from a requirements.in file. It does basically the same as poetry, but more manually. For me that's good because one of the application has a lot of requirements, and it needs to be deployed on systems with different Python versions, and the requirements need to be packaged along with the application because the servers have very limited internet access. So as long as Poetry doesn't add good support for multiple python versions and/or easy packaging of all dependencies, it isn't worth it for me to do the migration.
I'm liking PDM for a while now. Quicker than Poetry and built according to the Python package spec in mind and not as an afterthought. While it was originally meant to work with PEP 582, it works with virtual environments too (now default).
>I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi.
Agreed. I like docker images for smallish portable scripts. At home I can develop on my Mac and port it to a Raspberry PI or another x86 Windows/Linux box.
Planning on running a docker swarm with a few Pi’s to see how it works.
I wish VSCode would figure out that ExampleModel.objects.first() returns ExampleModel or None or ExampleModel.objects.filter() returns an iterable of ExampleModel. Has anybody gotten this working, automatically or manually annotating?
You can annotate the manager and get some typing help in the editor. And there’s django-stubs which helps a little when running mypy. It’s not as good as pycharm though.
https://github.com/charliermarsh/ruff
It’s literally 100 times faster, with comparable coverage to Flake8 plus dozens of plugins, automatic fixes, and very active development.