[CJefferson]

The switch to the 'python' command running python 3, as a non-arch user, put me off Arch forever. It just broke everything, which had long assumed 'python' would run python 2.

Not installing python 2, and just python 3, with the name python3, would have been fine.

Arch put us in a situation where it was basically impossible to run python 2 with a #! line, as some distros hadn't yet introduced a 'python2' symlink yet.

[StreakyCobra]

So to avoid re-opening the debate of Python 2 vs Python 3, let's take this point of view: The Python 3 language may, or may not, have been an improvement, and the transition from 2 to 3 may, or may not, have been done correctly. Everyone has its own opinion on this. But at the end this problem concerns the Python community.

From the Arch developers point of view, the Python developers decided to release a new version of their language. They were going to support python 2 for a while, but the direction that the language was taking for the future was Python 3, so one day or another Python 3 would become the standard, so doing the switch was the good decision.

As said by @tbranyen the work of adapting shebangs have been done on the Arch side anyway, so I don't see a problem for people developing their projects in Python. Also Arch users are mostly power users (you need to understand quite a bit about linux to install it) so they are usually able to handle python 2 vs 3 errors quite well, especially as it is known than Arch use Python 3 by default.

[bbatha]

Except that python says that python 3 should be an executable named `python3` and `python` should refer to `python2` [0]

0: https://www.python.org/dev/peps/pep-0394/

[jvreeland]

It says right in the pep

"...however, end users should be aware that python refers to python3 on at least Arch Linux (that change is what prompted the creation of this PEP), so python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3..."

There was no guidance before the Arch devs made their choice. There's no reason to change back.

Especially since the goal is that python3 will be default eventually.

"...in preparation for an eventual change in the default version of Python, Python 2 only scripts should either be updated to be source compatible with Python 3 or else to use python2 in the shebang line."

[srssays]

There's a good reason to change back. There is now convention in place, and Arch's approach is obsolete.

[lelandbatey]

It seems to me that Arch established their convention first, so why should Arch "change back" when they where first; why are others not "changing back" to Arch's convention, given that Arch's came first?

Well, the answer is that this is a silly discussion. Arch is doing what Arch is doing because Arch is intentionally forward oriented, sometimes to the detriment of backwards compatibility with the old school (and yes, a distro not having a symlink for python2 is at this point nearly 5 years into being "old school").

[srssays]

A particular convention has been adopted by nearly all distros, the Python project and the Python community. That convention is better than Arch's, because it is backwards-compatible with the large body of Python 2 code that exists.

Arch's change angered many in the Python community, and broke a lot of code. That Arch refuse to adopt the established convention, because it would involve admitting a mistake, shows a distinct lack of professionalism.

[StreakyCobra]

Fair point. But Archlinux made the switch in 2010 [1], and this PEP was created in 2011.

[1] https://www.archlinux.org/news/python-is-now-python-3/

[blipmusic]

Using the official, graphical installer for Python 3 on Windows with no previous version installed creates an executable 'python' that refers to Python 3.

(Normally not a Windows user so I have no idea why this is the case or if by design.)

[groovy2shoes]

Windows doesn't support shebang lines, so the exact name/location of the interpreter doesn't matter as much as it does on 'nix.

[maxerickson]

Recent versions of python ship with a launcher shim for windows that does support the shebang line (to simplify using multiple interpreters):

https://www.python.org/dev/peps/pep-0397/

As you say though, it doesn't rely on the names of the exes.

[CJefferson]

I work on two programs which either not distributed by Arch, or our users choose to download directly (I don't know).

They both contained Python scripts, and I had to waste my time figuring out why they had broken when I had e-mails from Arch users complaining they didn't work -- and then keep replying to future Arch users with the fix.

I don't know how many hours and days of other developer's time was wasted by Arch with this change, but it certainly wasted a good day of my time over the last few years.

[Latty]

I'm an Arch user, and it literally broke nothing for me. Packages were updated, life went on.

You seem to be complaining because other distros hadn't done the right thing. I don't see how that's Arch's fault. Are we to hold back for the lowest common denominator? Do we need every distro to join together and agree to a switchover date?

There was official guidance from Python on the switchover, this wasn't some maverick decision. It was just moving forward.

Not everyone wants to be tied to ancient stuff for backwards compatibility - if you do, it's your job to deal with that.

[rlpb]

My Python 2 scripts on Github start with "#!/usr/bin/python", as do many others. All of these broke for you as an Arch user.

Because Arch did what they did, Python now recommends that Python 2 scripts start "#!/usr/bin/python2" (or the env equivalent).

> Are we to hold back for the lowest common denominator? Do we need every distro to join together and agree to a switchover date?

It's not just other distros; it's the rest of the world, including all of the scripts that people run but distros don't necessarily ship.

The right way to migrate would be to ship and use both /usr/bin/python2 and /usr/bin/python3. Leave /usr/bin/python as a symlink to python2 for a while. Eventually, drop the symlink, but do not replace it with python3. Allow users to opt-in to a compat symlink if they wish. Let stuff catch up. When the expectation that "python" is python2 has faded, then introduce a symlink from python to python3 by default, letting users opt-in earlier if they wish.

> There was official guidance from Python on the switchover, this wasn't some maverick decision.

No, there wasn't, and yes, it was some maverick decision. It was done without consultation with upstream.

(I am a non-Arch distribution developer)

[Latty]

This just feels like culture clash to me - the whole change to Python 3 was about accepting the cost of breaking changes to avoid dragging around useless cruft and bad decisions forever.

The idea that we should then never use `python` to mean `python3` is just so backwards. Sure, in your corporate environment where backwards compatability is suepr important, do that.

In Arch, people have a distro that moves fast and breaks things, and we are fine with that. I'll accept the cost of occasionally having to change a shebang line (although as I have almost never had to go outside the AUR - someone else in the Arch community has almost always dealt with this for me).

> No, there wasn't, and yes, it was some maverick decision. It was done without consultation with upstream.

Fair enough, I got my timelines mixed up - but to be fair, it was then ratified as the correct way to do things because the upstream project agreed with the core idea.

> The right way to migrate would be to ship and use both /usr/bin/python2 and /usr/bin/python3. Leave /usr/bin/python as a symlink to python2 for a while. Eventually, drop the symlink, but do not replace it with python3. Allow users to opt-in to a compat symlink if they wish. Let stuff catch up. When the expectation that "python" is python2 has faded, then introduce a symlink from python to python3 by default, letting users opt-in earlier if they wish.

I don't really see why I should have to do this manually? If you don't want that behaviour, use another distro. This just feels like you feel like you should have some say in how other people set up their systems.

If you really hate it so much, don't support Arch - that'd be fair. Trying to shame the distro for doing it ignores the fact it's the core idea of the distro to do stuff like that, that's the point.

[rlpb]

> The idea that we should then never use `python` to mean `python3` is just so backwards.

I did not present any such idea. I presented a sane migration path to where `python` means `python3`.

> ...it was then ratified as the correct way to do things because the upstream project agreed with the core idea.

No. It was because the upstream project had their hand forced and are pragmatic about these things.

> I don't really see why I should have to do this manually?

In my proposal for a sane migration path to the ideal? You wouldn't have to do it manually. The distribution would do it. As a user you'd be able to override it to speed up the migration for yourself if you chose; that's all.

[Latty]

I chose to run a bleeding edge distro to be on the bleeding edge. Requiring me to make the changes to be on the bleeding edge manually makes no sense.

Expecting that is as weird as expecting super stable LTS releases to run the latest-and-greatest of everything.

[rlpb]

> Requiring me to make the changes to be on the bleeding edge manually makes no sense.

Which part of "You wouldn't have to do it manually" do you not understand?

[AsyncAwait]

> My Python 2 scripts on Github start with "#!/usr/bin/python", as do many others. All of these broke for you as an Arch user.

So you made certain assumptions. Have you made your scripts use python2, which is safer anyway, everything would've been fine, don't blame Arch for your incorrect assumptions.

[rlpb]

No. Before Arch made their change, that /usr/bin/python is Python 2 was the only possible assumption that could be made. No distribution shipped /usr/bin/python2 at that time. There was no need, because /usr/bin/python was always Python 2.

It was only because of what Arch unilaterally did that forced the community to start providing a /usr/bin/python2. Before that, /usr/bin/python2 did not commonly exist at all.

[AsyncAwait]

> It was only because of what Arch unilaterally did

Arch is a bleeding-edge, latest software kind of distribution. They did not made the decision 'unilaterally'. They made the decision to ship the latest software for their own distribution, as they always do. It may have proven that a lot of software is extremely poor/inflexible, (i.e. rests on weak assumptions), but I am glad Arch moved forward as that showed their resolve to move technology forward even in the face of a lot of pressure from outside groups.

I just think that instead of blaming Arch, try to make your scripts more robust, i.e. loop on all 'python*' binaries in /usr/bin and use the first one whose --version gives you '2.x' instead of relying on /usr/bin/python being python 2 when the latest version is 3 and Arch is known to ship latest software.

[rlpb]

> They made the decision to ship the latest software for their own distribution, as they always do.

As did all the other distributions. Shipping Python 3 is not the issue here. Breaking compatibility with all existing Python 2 scripts is what they did. The two are not mutually exclusive.

> I just think that instead of blaming Arch, try to make your scripts more robust, i.e. loop on all 'python<glob>' binaries in /usr/bin...

You want me to do that in a shebang line? No. That'd be crazy.

> ...when the latest version is 3 and Arch is known to ship latest software.

See https://en.wikipedia.org/wiki/Application_binary_interface. What Arch did was break the ABI of what /usr/bin/python means, breaking all scripts that relied on that ABI. Shipping the latest software is orthogonal to this. The rest of the world ships the latest software too, but without breaking ABI.

[RUG3Y]

The decision is pretty simple: if a user doesn't like the way Arch does things, they're free to use any of a hundred other distros. I'm with you, I think this is fine.

[simplyluke]

> Are we to hold back for the lowest common denominator?

Well, the fact that over a decade after the release of python 3 we're _still_ arguing about it shows that lot of people think we should.

[CJefferson]

I thought the official advice was that 'python' should remain 'python2'?

from https://www.python.org/dev/peps/pep-0394/ (not yet replaced/update) "for the time being, all distributions should ensure that python refers to the same target as python2 ."

[StreakyCobra]

Archlinux made the switch in 2010 [1], and this PEP was created in 2011. [1] https://www.archlinux.org/news/python-is-now-python-3/

[michrassena]

What I wonder is why one would assume the python command would run python 2, instead of running python 1, assuming python 1 was a thing. By your logic, shouldn't python2, run python3 run python 3, etc. What I'm getting at, is we get used to things being a certain way, but often we're just following a convention that was set in place before we got onboard. Why do we still type "bash" instead of "bash4"?

[CJefferson]

The reason you would assume it is that, for a long time on every Linux distribution, there was a single way to run python2, and it was called 'python'. I'm not sure what you are getting at, this was just the way it was. You couldn't run python 2 any other way in some linux distros.

The reason we don't need a 'bash4' is that, by and large, bash tried very hard to make sure they didn't break any bash3 scripts, so there is no need for old bash3 scripts to know they are now running in bash4, unlike python3.

[michrassena]

I get the point, python 2 scripts aren't fully compatible with python 3. I've written a little python and know that to be the case.

However, I can predict with near certaintly, if python 4 doesn't break python 3 scripts, people will complain if "python3" doesn't invoke python 4, because they've always used "python3" to run the new python and "python" to run python 2.

[gravypod]

> Why do we still type "bash" instead of "bash4"

Because Bash was made by, and is maintained by, people who understand how backwards compatability in their program is important.

[hobarrera]

Just like python. That's why they bumped the major when release backwards-incompatible changes, which is the right thing to do.

[AsyncAwait]

Why not be explicit with 'python2' and be done with it?

[Zancarius]

I agree, but I'd guess the chief complaint is that doing so requires the `python2` symlink to be available. There's at least one claim up thread that it doesn't exist on all systems (create it yourself?). I didn't catch the distribution used, but I DID check an older Ubuntu image I have (I run Arch) and there's definitely a python2 link pointing to python2.7 (`python` also points to python2.7 and there's a `python3` symlink that does what you'd expect). So outside some LTS build, it seems surprising to me that it wouldn't exist in 2017 on a modern distro. I still can't shake the feeling that the thread started around a relatively insignificant complaint (dumping Arch because they use Python3 by default? Come on...).

Personally, I'd rather just use `/usr/bin/env python2` to invoke the appropriate interpreter (if you don't need to pass arguments to it), and if the symlink doesn't exist, the onus is on the user to create it. This is ESPECIALLY true for older distros.

Plus, for a sufficiently complicated application, why would someone want to pollute the global site-packages anyway? I'd rather manage most things inside a virtualenv or similar (pick your Python version), because I've seen far too many install_requires/requirements locking to a specific library version.

[CJefferson]

In 2017, indeed everyone has added python2, but I assure you it wasn't that way when Arch originally did the python3 -> python rename.

While this is a small issue, this original thread used it as an example of how Arch is good and pragmatic. To me it seemed like the opposite -- it broke lots of code and packages for no good reason. Why should users of older distros have to add a symlink, when before Arch everyone could be sure that '/usr/bin/python' if present would be python2, and '/usr/bin/python3', if present, would be python3?

Now I agree that using virtualenv or similar is a good idea, but this actually caused the most problems for little 20 line python scripts, because often the users of those didn't even realise they were using python.

[Zancarius]

> I assure you it wasn't that way when Arch originally did the python3 -> python rename.

I disagree. I don't believe it was as painful as some make it out to be. The official packages were updated within a relatively short period of time, and while the AUR packages took longer they're also not officially supported. There was also a news announcement [1]. Honestly, I think of a few other transitions over my years of using Arch as being far, far more painful. The Python2 -> Python3 defaults change isn't one of them.

Although I would agree that Gentoo's approach was somewhat better via eselect, which I believe predated Arch's migration.

> Why should users of older distros have to add a symlink

You're right. That should be the responsibility of the package maintainer. Conversely, why should I, as a developer, have to continue assuming `/usr/bin/python` points to any particular version of Python when most distributions have migrated away from this anachronism? sed does magic, and you really should be using the appropriate symlink for your desired version anyway. Python 3 isn't new. I have a hard time seeing this as problematic because 1) things change and 2) the solution is easy. I'd be happy to entertain a use case where this actually has presented material difficulties, however.

> but this actually caused the most problems for little 20 line python scripts, because often the users of those didn't even realise they were using python.

Well, I do agree: It's a problem for the end user, but the solutions aren't difficult (if someone's using Linux, they probably know enough to fix it). So, either change the hashbang, add a symlink, and--failing that--complain to the package maintainer if it's been packaged up by the distro and the upstream software itself has changed (it should).

My recommendation for developers would be to use the appropriate /usr/bin/python* symlink. Nearly all distros have them in place now (including Ubuntu from a couple years ago). Perhaps assuming /usr/bin/python will always point to a fixed version is a matter of misplaced assumptions.

[1] https://www.archlinux.org/news/python-is-now-python-3/

[AsyncAwait]

> for no good reason.

There was good reason, to move to the latest version of a particular software, since that's kind of the whole philosophy of Arch, in fact the reason I originally switched to it in 2011 was because I had enough of me not being able to use the latest software on Ubuntu.

> Why should users of older distros have to add a symlink, when before Arch everyone could be sure that '/usr/bin/python' if present would be python2

If you made no effort to detect the actual version, it's wrong to assume that. It could've just as well been python 1.

[Zancarius]

> in fact the reason I originally switched to it in 2011 was because I had enough of me not being able to use the latest software on Ubuntu.

It's funny. My reason for switching to Arch was almost the opposite. I grew tired of having to rebuild the latest software in Gentoo every time I updated and ran into someone on Slashdot who encouraged me to give Arch a try. Sure, there's more prebuilt packages, mirrors, etc for Gentoo these days, but then you're in the same boat as everyone else (waiting for the package to be built).

[RUG3Y]

You're right. - If Arch is targeted for power users, what's preventing them from configuring symlinks etc? - Why aren't they using virtualenvs? I never install Python packages outside of a virtualenv.

[Zancarius]

> I never install Python packages outside of a virtualenv.

Global site-packages is all fun and games until someone loses an eye (or a dependency).

I had a fun learning experience with a large Python package once while trying to mix and match dependency versions, simultaneously trying to avoid clobbering the ones already installed by pacman. Without a virtualenv, it quickly becomes impossible to guarantee you won't break something in the process of installing something else. I never made that mistake again.

[hobarrera]

You really shouldn't install global packages circumventing the package manager (this applies to python and everything else really). You're just asking for eventual breakage, or leaking obsolete files.

[tbranyen]

It really wasn't that bad and the AUR + official channels had patched packages with the correct shebang line. The worst offender for a while was the node-waf builder, had to patch it manually to get any packages to build, otherwise you needed a command line arg every time. This was like 5 years ago tho... These days I have never hit an issue w/ a properly installed python package.

[throwaway7767]

Those scripts should probably be using something like the following, if they really are python2-only: #!/usr/bin/env python2

I get your frustration though, it's a change that breaks a whole lot of existing scripts.

[maxerickson]

That doesn't address the part where python2 didn't exist in a lot of places at the time, so scripts using python2 to work on Arch would have to be adjusted to work elsewhere. There was no universal solution.

[hobarrera]

The thing is, python guidelines (PEPs) stated that shebangs should specify python2 or python3 explicitly during the transition period. Using "#!/usr/bin/env python" for a project that was python2-only was not following guidelines, so the breakage was really the dev's fault.

Also, the python2 symlink should have already been there. Bad packaging on some distros resulted in bad practices packaging by some devs. Don't blame that on Arch.

[rlpb]

You're getting the ordering of events wrong. Arch's actions weren't in response to the PEP. The PEP was written in response to Arch's actions.

> Also, the python2 symlink should have already been there.

Nope. There was no such convention or specification at the time. There was, however, convention and specification (in the form of "what Python upstream build systems do") on what /usr/bin/python means, which is "Python 2".

[hobarrera]

> There was no such convention or specification at the time.

There was no formal specification, but there was a convention that many distributions were already following (and quite a few others were not).

[Fiahil]

Well, even Ansible folks started to switch to python 3 (https://docs.ansible.com/ansible/python_3_support.html). Pushing people off python 2 is not "absolutely evil".

[CJefferson]

I have no problem with Arch having a "only install python3 by default" rule. They can even not distribute python2 for all I care.

But, don't make /usr/bin/python run python3, that just confuses programs which have made the (reasonable based on past experience, and official python advice) assumption that it /usr/bin/python will run python2.

[simplyluke]

So forever into the future $ python should run python2? That's not pragmatic at all.

[marvy]

On Ubuntu, "sqlite" starts sqlite2.x; if you want sqlite3.x, you must type "sqlite3". The last version of sqlite2 came out in 2005. As far as I know, hardly anyone still uses sqlite2 for anything anymore. (I could be very wrong though.) Nevertheless, people must still type that extra character "3". The sky has not yet fallen :)

It seems to me that the same approach could work for Python.

[michaelmrose]

Why not? Can you elaborate?

[greggyb]

Pythons 4, 5, 6..., N will feasibly exist in the future. Why should Python 2 forever retain the executable name 'python'?

[michaelmrose]

Eventually logically python 2 will be little used enough to be dropped and there may not be a python4 any time in the foreseeable future so perhaps you are just borrowing trouble.

[marvy]

If Py4 is compatible with Py3, why bump the version number? If not, you better be careful which one you get!

[boznz]

Lets hope Python 4 comes out soon so you guys can unite against the real enemy :-)

[djsumdog]

Ubuntu does this now with Xenial.

[pseudalopex]

Ubuntu uses Python 3 where possible, but /usr/bin/python is 2.7.

[Asooka]

In fairness, this was back when we all thought Py3 would take over Py2 "any day now"™

[avian]

"back then" the decision made even less sense, since a lot of efforts to make Python 3.x code backwards compatible did not happen yet. Like for instance the u'' literals introduced in 3.3.

[seiferteric]

Why didn't python just opt to have the python 2 and 3 interpreter in the same binary? It could have detected if a file was python3 with a magic comment or something (overridable with a -2/-3 option), or otherwise assume python2. Or even simpler, have a wrapper script just called 'python' that checks if the file you are calling is python2 or 3 and calls the appropriate binary.

[Zancarius]

"Explicit is better than implicit." is the Python mantra. Although it's not often taken as literally as some would like, I think it's appropriate in this case. After all, magic detection sounds good on the surface, but it's liable to be problematic and bug ridden.

You don't need magical file-specific juju or a wrapper script on most systems these days to select the Python version. Just call python2 or python3 as appropriate; the symlinks usually exist depending on packager, distro, etc. The problem up-thread appears to be with calling the default `python` on $PATH. Sometimes it points to Python 2.x, sometimes it points to Python 3.x. Sometimes it might even change (Gentoo's eselect, IIRC). I don't see the problem being explicit with which version is needed in the script's hashbang. Python 3 isn't exactly new, and I remember when Gentoo and Arch both migrated to versioned symlinks. It wasn't that painful.