[clavigne]

why? I commonly do things like screen 10,000 molecules for some property by calling a Fortran program 10k time. If each run takes 60s of compute, the 10s boot from Julia (over the nfs disk) would be pretty significant.

Using Julia in this case basically means having to rewrite all of the other programs into it and getting rid of eg snakemake or gnu parallel or any number of other very common HPC workflows.

In fact I'd venture that doing a sequence of things many times is just as common an HPC workload than single jobs running for very long times.

[oscardssmith]

One reason to consider Julia for workflows like this is that it's been designed to improve reproducibility. Julia makes it really easy to distribute package state (through manifest.toml files), that make it easy to get exact copies of the package versions that the original author used.

[snicker7]

Guix/Nix does this better across many programming languages. Unlike language-specific package managers, Guix is fully recursively reproducible ("bootstrappable"). And Nix isn't terribly far behind, either.

Julia itself is not even reproducible: https://github.com/JuliaLang/julia/issues/25900 https://github.com/JuliaLang/julia/issues/34753

[oscardssmith]

Note that those issues are about deterministic performance, not results. Also guix and nix appear to be Unix only, which severely limit their use for Windows users. One of the places Julia really excels with reproducibility is inter OS, yes there are probably some bugs, but the ability to manage versions cross OS this easily is something I'm not aware of a good alternative for.

[snicker7]

> Note that those issues are about deterministic performance, not results.

Not at all. The issue is that the default sysimage distributed with Julia cannot be reproduced across machines, e.g. there is no proof that it hasn't been tampered with. This is an open issue in Guix and Nix, though it affects all platforms.

> One of the places Julia really excels with reproducibility is inter OS

There really is no such thing as reproducibility in Windows, at least not in the sense outlined in bootstrappable.org . If you care about reproducibility, you've chosen the wrong platform.

[oscardssmith]

Sorry for the confusion. I meant reproducibility in the sense that I can have a git repo to share code between a laptop running windows and a cluster running Linux, and if I run the program on both I'll be running the same program (same libraries, same versions, equivalent binary blobs, etc). I don't mean that it can give you a cryptographic proof that your code hasn't been tampered with maliciously, just that you can debug your code locally, then deploy it to whatever you want and be pretty sure that it will still work even if you deploy to a different OS an/or cpu architecture.

[hansvm]

Is there any particular reason you wouldn't want to wrap that workload in a function/module/... and call it in a loop? Is the problem that your commonly used tools are exposed only as command line interfaces and not as libraries?

[clavigne]

Yeah that's what I mean. This is the case for both commercial software (eg Q-Chem) and free software.

This is not to say that the boot time is always a problem, there are plenty of applications in which it doesn't matter at all (as you point out). However that's not always the case (by a long shot) which is why tools like GNU Parallel and snakemake exist.

[shele]

Just keep it running as a daemon? https://github.com/dmolina/DaemonMode.jl