[BiteCode_dev]

Most of the time, you don't need all that, since Python has zipapps. You defined deps, you zip it, you ship to any same os with the same python version. It embeds everything, and just run.

We even how have a nice tool to automate the bundling for you:

https://pypi.org/project/shiv/

Of course you still have to figure out how to get a Python installed on the final machine, that's the price to pay to be an interpreted language.

We don't have yet a story to ship a beautiful exe/dmg/deb/rpm that embeds the zipapp and libpython in an easy way.

[michaelcampbell]

> you ship to any same os with the same python version

This is a non-trivial thing to handwave away.

[BiteCode_dev]

I agree, but it's still way easier than the original story, which is the one you also have with PHP, Ruby, JS, etc.

Using an interpretted language always leads to this.

I know no popular interpretted language with a seamless experience to ship a standalone exe.

In fact, Python is probably the one with the best story here, since it has nuitka (https://nuitka.net/), which allows to compile Python code into a fully standalone exe.

But then you need to install a compiler, headers, etc. And no cross compilation of course. Not to mention on Linux, you have to ensure you target the lowest version of libc you can.

You are still very far from Go or Rust, and I'm hoping one day that RustPython will succeed because that would mean an amazing deployment story.

Meanwhile, you trade the ease of deployment of compiled languages for the ease of development of interpretted ones.

I think it's a fare trade for most people: you dev the program much more often that you deploy it.

That doesn't mean we shouldn't work, as a community, to improve the deployment story. It's a serious hindrance.

That's the raison d'être of the Briefcase project (https://beeware.org/project/projects/tools/briefcase/). It's still in progress, but the last prez I saw on it was quite impressive already.

[pansa2]

> I'm hoping one day that RustPython will succeed because that would mean an amazing deployment story.

Isn't RustPython just an alternative interpreter to CPython, implemented in Rust instead of C?

How would RustPython offer better deployment than CPython?

[BiteCode_dev]

Rust has a fantastic deployment story: compiling a rust program is super easy, and you can cross compile. Using cargo and rustc is a breath of fresh air compared to any similar experience with C compiling.

So if one day RustPython gets compatible enought with CPython that you can use it as a drop in replacement, you can start creating a tool that compiles any Python VM for any target, and bring along your program with it. Making a standalone version of it would become much easier.

Right now, doing so either requires you to bring in a pre-compile version of cpython for your target (which is what briefcase does) or compile the thing yourself with gcc + headers + deps(which is what nuitka does).

It's not easy.

[throwaway894345]

> So if one day RustPython gets compatible enought with CPython that you can use it as a drop in replacement

I don't think this will ever happen unless the community converges on a standard C-extension interface. Presently Python leans so hard on C-extensions, but there is no standard interface--if you're writing a C-extension library, you just depend on whatever obscure corner of CPython that suits your purpose. If you're writing an alternative Python interpreter, you have to implement the entire surface area of CPython, which generally means you must implement CPython exactly and you are severely restricted on the improvements you can make. At that point, why even bother?

Fortunately, I think there are emerging candidate interfaces, but the community needs to either update C-extension packages to use those interfaces or support packages (and maintainers) who already do. https://github.com/pyhandle/hpy.

[BiteCode_dev]

There are probably only a dozen of c popular extensions that needs to support HPY reach the tipping point of mass adoption: numpy, scipy, pycuda, tensorflow, matplotlib, uvloop, etc. and some db drivers.

The rest is not popular enought to be a blocker. You will hear them scream a lot, but they will be like 0.00001% of the user base, and we can just tell them to stay on CPython with its limitations. They don't lose anything, just not gain anything either.

Those C extensions authors are directly in communication with Python core devs, when they are not core devs themselves, so if HPY is adopted, we can expect a total adoption under 5 years.

Numpy authors already said it would take 1 year to adopt it.

Give the huge number of benefits of HPY, I deeply hope it will be a success.

[panarky]

> a breath of fresh hair

Indeed.

[BiteCode_dev]

:) Fixed

[Hamuko]

youtube-dl for example is distributed as a zipapp and it seems to be distributed just fine. It only requires you to have Python installed on your system, which isn't too burdensome of a requirement on macOS/Linux. On Windows they do actually distribute a Python interpreter.

[mmm_grayons]

Youtube-dl is a great tool, but the audience (primarily technical people) of any CLI app won't be representative of shipping an app for general use.

[airstrike]

GP's point was not about "general use" apps, but apps for other developers / system maintainers

[ses1984]

What's the story if my zipapp depends on tensorflow and/or cuda?

I don't know exactly how it goes but I'm pretty sure it's a horror story.

[BiteCode_dev]

As usual with extensions, you are not using Python anymore, but a compiled language. To get 100% certainty, you'd need to compile the whole thing.

That being said, a lot of extensions are pre-compiled and provided as wheel, which is the case for tensorflow (I don't know for CUDA, I can't test on a laptop without a GPU).

Let's see what this means:

    $ py -m venv test
    $ test\Scripts\activate
    $ pip install tensorflow
    $ code hello_tensor.py
    # import tensorflow as tf

    # def main():

    #     with tf.compat.v1.Session() as sess:
    #         a = tf.constant(3.0)
    #         b = tf.constant(4.0)
    #         c = a+b
    #         print(sess.run(c))

Now with shiv:

    $ copy hello_tensor.py test\Lib\site-packages
    $ shiv -e hello_tensor.main --site-packages test\Lib\site-packages\ -o hello_tensor.pyz
    $ python hello_tensor.pyz
    ...
    2020-05-28 17:31:46.580704: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
    7.0

So it works fine, but remember:

- it will only run on the system this particular wheel has been designed to run on. In my case cp38-win_amd64.

- it will come bundled with tensorflow, which is a behemot, meaning your hello world pyz will around 500 Mo.

- it needs to unzip, so the first run will be REALLY slow

For something like this, I would advice a more generic deployment tool, like fabric 2 if it's remote, or a make-like tool such as doit if it's local only.

Make your deployment script, and zipapp that.

[throwaway894345]

Zipapps are an order of magnitude improvement in the Python world, but there are still lots of other major pain points like dependency management and performance which still leaves Python several orders of magnitude behind its competition. Hopefully these things change going forward.

[pansa2]

It looks like zipapps built with “shiv” need to extract the contents of the zip file to disk before they can run? Does it delete the extracted files on exit?

If so, the extraction is going to make startup very slow. If not, that’s just messy. Either way, it’s not ideal.

[BiteCode_dev]

It's doing the work only once.

Yes, it's not ideal.

But it beats shipping your entire dev env to the server.

I find it a good compromise. The extraction is done is $HOME/.shiv/{zipappname}_{zipapphash} so it's not a horrible mess. But if your project is big, you do have to clean up the old install because it can eat a significant amount of space.

For you regular cmd tool though, it's a blip.

[jasonpeacock]

> you still have to figure out how to get a Python installed on the final machine, that's the price to pay to be an interpreted language

Also for Java. The installs usually include their own favorite version of Java alongside the actual application.

[zo1]

Installing Java is a bit of a difficult task these days. Even worse if you want to get the JDK which is behind a register-wall.

[pritambarhate]

Not if you are installing OpenJDK or Amazon Corretto. Add the PPA and install with apt-get. Like every other Ubuntu/Debian package.

[alexhutcheson]

jlink[1] solves this for JDK 9 and later.

[1] https://docs.oracle.com/javase/9/tools/jlink.htm